2022-06-27T06:38:15.394534065Z SYSTEM: Preparing env...
2022-06-27T06:38:15.936080766Z SYSTEM: Running...
2022-06-27T06:38:16.723513366Z 2022-06-27 14:38:16.723151: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:38:18.316735193Z WARNING:tensorflow:From /home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
2022-06-27T06:38:18.316789541Z Instructions for updating:
2022-06-27T06:38:18.316798297Z non-resource variables are not supported in the long term
2022-06-27T06:38:18.323267476Z Imported model (for Places365, 128x128 images)
2022-06-27T06:38:21.064529748Z WARNING:tensorflow:From /home/jovyan/work/src/model.py:20: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.
2022-06-27T06:38:21.064570688Z Instructions for updating:
2022-06-27T06:38:21.064600199Z Use `tf.keras.layers.Conv2D` instead.
2022-06-27T06:38:21.076688641Z WARNING:tensorflow:From /home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
2022-06-27T06:38:21.076716602Z Instructions for updating:
2022-06-27T06:38:21.076724171Z Please use `layer.__call__` method instead.
2022-06-27T06:38:21.221472244Z WARNING:tensorflow:From /home/jovyan/work/src/model.py:79: conv2d_transpose (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.
2022-06-27T06:38:21.221498205Z Instructions for updating:
2022-06-27T06:38:21.221506439Z Use `tf.keras.layers.Conv2DTranspose` instead.
2022-06-27T06:38:21.312483203Z 2022-06-27 14:38:21.311776: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2022-06-27T06:38:21.356502814Z 2022-06-27 14:38:21.355592: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.361476016Z 2022-06-27 14:38:21.356582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
2022-06-27T06:38:21.36149573Z pciBusID: 0000:00:08.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
2022-06-27T06:38:21.36151434Z coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 7.75GiB deviceMemoryBandwidth: 681.88GiB/s
2022-06-27T06:38:21.361522652Z 2022-06-27 14:38:21.356626: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:38:21.36152877Z 2022-06-27 14:38:21.359432: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-06-27T06:38:21.373639957Z 2022-06-27 14:38:21.362305: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-06-27T06:38:21.373676623Z 2022-06-27 14:38:21.362747: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-06-27T06:38:21.373685101Z 2022-06-27 14:38:21.365718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-06-27T06:38:21.373690705Z 2022-06-27 14:38:21.367188: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-06-27T06:38:21.387364092Z 2022-06-27 14:38:21.374139: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-06-27T06:38:21.387383972Z 2022-06-27 14:38:21.374593: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.387390216Z 2022-06-27 14:38:21.376159: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.387406699Z 2022-06-27 14:38:21.377495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-06-27T06:38:21.387412376Z 2022-06-27 14:38:21.378494: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
2022-06-27T06:38:21.387418019Z To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-06-27T06:38:21.431497702Z 2022-06-27 14:38:21.431103: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2499990000 Hz
2022-06-27T06:38:21.432398121Z 2022-06-27 14:38:21.432015: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5e57cd0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-06-27T06:38:21.432415269Z 2022-06-27 14:38:21.432102: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2022-06-27T06:38:21.657683854Z 2022-06-27 14:38:21.654246: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.657726491Z 2022-06-27 14:38:21.655356: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4efe4c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-06-27T06:38:21.657733833Z 2022-06-27 14:38:21.655398: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2022-06-27T06:38:21.657739026Z 2022-06-27 14:38:21.655783: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.657757214Z 2022-06-27 14:38:21.656944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
2022-06-27T06:38:21.657780321Z pciBusID: 0000:00:08.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
2022-06-27T06:38:21.6577862Z coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 7.75GiB deviceMemoryBandwidth: 681.88GiB/s
2022-06-27T06:38:21.657790652Z 2022-06-27 14:38:21.656994: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:38:21.65779509Z 2022-06-27 14:38:21.657030: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-06-27T06:38:21.65779984Z 2022-06-27 14:38:21.657057: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-06-27T06:38:21.657804356Z 2022-06-27 14:38:21.657082: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-06-27T06:38:21.657808837Z 2022-06-27 14:38:21.657104: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-06-27T06:38:21.657813355Z 2022-06-27 14:38:21.657126: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-06-27T06:38:21.65781946Z 2022-06-27 14:38:21.657149: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-06-27T06:38:21.657823716Z 2022-06-27 14:38:21.657381: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.660267085Z 2022-06-27 14:38:21.660041: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:21.668181623Z 2022-06-27 14:38:21.661956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-06-27T06:38:21.668216811Z 2022-06-27 14:38:21.662074: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:38:22.353724159Z 2022-06-27 14:38:22.351212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-06-27T06:38:22.35376374Z 2022-06-27 14:38:22.351279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2022-06-27T06:38:22.353768071Z 2022-06-27 14:38:22.351336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2022-06-27T06:38:22.364752412Z 2022-06-27 14:38:22.361278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:22.364805738Z 2022-06-27 14:38:22.362655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:38:22.364814235Z 2022-06-27 14:38:22.363649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7116 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:08.0, compute capability: 6.0)
2022-06-27T06:38:22.375506339Z Traceback (most recent call last):
2022-06-27T06:38:22.375556853Z File "src/test.py", line 33, in <module>
2022-06-27T06:38:22.386370827Z saver.restore(sess, model_PATH)
2022-06-27T06:38:22.386415106Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1291, in restore
2022-06-27T06:38:22.386423007Z checkpoint_prefix)
2022-06-27T06:38:22.386428006Z ValueError: The passed save_path is not a valid checkpoint: /home/jovyan/work/src/output/models/
2022-06-27T06:38:23.059942491Z SYSTEM: Finishing...