2022-06-27T06:36:09.110278599Z SYSTEM: Preparing env...
2022-06-27T06:36:09.755697526Z SYSTEM: Running...
2022-06-27T06:36:10.582438824Z 2022-06-27 14:36:10.578599: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:36:12.27114199Z WARNING:tensorflow:From /home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
2022-06-27T06:36:12.27118886Z Instructions for updating:
2022-06-27T06:36:12.271196741Z non-resource variables are not supported in the long term
2022-06-27T06:36:12.27588787Z Imported model (for Places365, 128x128 images)
2022-06-27T06:36:15.127571182Z WARNING:tensorflow:From /home/jovyan/work/src/model.py:20: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.
2022-06-27T06:36:15.127625625Z Instructions for updating:
2022-06-27T06:36:15.127635424Z Use `tf.keras.layers.Conv2D` instead.
2022-06-27T06:36:15.127641681Z WARNING:tensorflow:From /home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
2022-06-27T06:36:15.127654313Z Instructions for updating:
2022-06-27T06:36:15.127659052Z Please use `layer.__call__` method instead.
2022-06-27T06:36:15.256535403Z WARNING:tensorflow:From /home/jovyan/work/src/model.py:79: conv2d_transpose (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.
2022-06-27T06:36:15.256581182Z Instructions for updating:
2022-06-27T06:36:15.256588253Z Use `tf.keras.layers.Conv2DTranspose` instead.
2022-06-27T06:36:15.327579181Z 2022-06-27 14:36:15.326484: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2022-06-27T06:36:15.413601745Z 2022-06-27 14:36:15.410378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.413646851Z 2022-06-27 14:36:15.411277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
2022-06-27T06:36:15.413654172Z pciBusID: 0000:00:08.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
2022-06-27T06:36:15.413657624Z coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 7.75GiB deviceMemoryBandwidth: 681.88GiB/s
2022-06-27T06:36:15.413660889Z 2022-06-27 14:36:15.411345: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:36:15.424689012Z 2022-06-27 14:36:15.423389: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-06-27T06:36:15.432079412Z 2022-06-27 14:36:15.431364: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-06-27T06:36:15.496414752Z 2022-06-27 14:36:15.493736: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-06-27T06:36:15.593235058Z 2022-06-27 14:36:15.591720: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-06-27T06:36:15.599321627Z 2022-06-27 14:36:15.596773: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-06-27T06:36:15.805424915Z 2022-06-27 14:36:15.801512: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-06-27T06:36:15.805479188Z 2022-06-27 14:36:15.801981: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.805486999Z 2022-06-27 14:36:15.803112: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.805492776Z 2022-06-27 14:36:15.804002: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-06-27T06:36:15.805497988Z 2022-06-27 14:36:15.804847: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
2022-06-27T06:36:15.805503237Z To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-06-27T06:36:15.81417475Z 2022-06-27 14:36:15.813899: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2499990000 Hz
2022-06-27T06:36:15.814946288Z 2022-06-27 14:36:15.814691: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5547d70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-06-27T06:36:15.814965987Z 2022-06-27 14:36:15.814721: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2022-06-27T06:36:15.978500134Z 2022-06-27 14:36:15.972620: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.978540247Z 2022-06-27 14:36:15.973668: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4916460 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-06-27T06:36:15.978547933Z 2022-06-27 14:36:15.973703: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2022-06-27T06:36:15.978553148Z 2022-06-27 14:36:15.974073: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.978558435Z 2022-06-27 14:36:15.974960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
2022-06-27T06:36:15.978579396Z pciBusID: 0000:00:08.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
2022-06-27T06:36:15.978583413Z coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 7.75GiB deviceMemoryBandwidth: 681.88GiB/s
2022-06-27T06:36:15.978586337Z 2022-06-27 14:36:15.974996: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:36:15.978589107Z 2022-06-27 14:36:15.975027: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-06-27T06:36:15.97859187Z 2022-06-27 14:36:15.975052: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-06-27T06:36:15.978594744Z 2022-06-27 14:36:15.975075: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-06-27T06:36:15.978597775Z 2022-06-27 14:36:15.975098: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-06-27T06:36:15.978600631Z 2022-06-27 14:36:15.975121: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-06-27T06:36:15.978604493Z 2022-06-27 14:36:15.975144: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-06-27T06:36:15.978607328Z 2022-06-27 14:36:15.975351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.978610359Z 2022-06-27 14:36:15.976383: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:15.978613279Z 2022-06-27 14:36:15.977323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-06-27T06:36:15.978616007Z 2022-06-27 14:36:15.977373: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-06-27T06:36:16.61051414Z 2022-06-27 14:36:16.609849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-06-27T06:36:16.610556469Z 2022-06-27 14:36:16.609912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2022-06-27T06:36:16.61056136Z 2022-06-27 14:36:16.609925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2022-06-27T06:36:16.621638578Z 2022-06-27 14:36:16.618766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:16.621703314Z 2022-06-27 14:36:16.619824: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-27T06:36:16.621711487Z 2022-06-27 14:36:16.620774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7116 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:08.0, compute capability: 6.0)
2022-06-27T06:36:16.659863984Z 2022-06-27 14:36:16.659585: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.661458563Z 2022-06-27 14:36:16.661189: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.66147741Z 2022-06-27 14:36:16.661256: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at save_restore_tensor.cc:182 : Data loss: Unable to open table file /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.858577056Z Traceback (most recent call last):
2022-06-27T06:36:16.858614223Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
2022-06-27T06:36:16.860723274Z return fn(*args)
2022-06-27T06:36:16.860770309Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
2022-06-27T06:36:16.860778361Z target_list, run_metadata)
2022-06-27T06:36:16.860783251Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
2022-06-27T06:36:16.860788557Z run_metadata)
2022-06-27T06:36:16.860793314Z tensorflow.python.framework.errors_impl.DataLossError: 2 root error(s) found.
2022-06-27T06:36:16.860798308Z (0) Data loss: Unable to open table file /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.860804505Z [[{{node save/RestoreV2}}]]
2022-06-27T06:36:16.860809493Z (1) Data loss: Unable to open table file /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.860815269Z [[{{node save/RestoreV2}}]]
2022-06-27T06:36:16.860819647Z [[save/RestoreV2/_17]]
2022-06-27T06:36:16.860824283Z 0 successful operations.
2022-06-27T06:36:16.860829198Z 0 derived errors ignored.
2022-06-27T06:36:16.860833772Z
2022-06-27T06:36:16.860838066Z During handling of the above exception, another exception occurred:
2022-06-27T06:36:16.86084295Z
2022-06-27T06:36:16.860847246Z Traceback (most recent call last):
2022-06-27T06:36:16.860865911Z File "src/test.py", line 33, in <module>
2022-06-27T06:36:16.867479466Z saver.restore(sess, model_PATH)
2022-06-27T06:36:16.867497042Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1299, in restore
2022-06-27T06:36:16.867501318Z {self.saver_def.filename_tensor_name: save_path})
2022-06-27T06:36:16.867504328Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
2022-06-27T06:36:16.867507425Z run_metadata_ptr)
2022-06-27T06:36:16.867510248Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run
2022-06-27T06:36:16.867513326Z feed_dict_tensor, options, run_metadata)
2022-06-27T06:36:16.867516126Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
2022-06-27T06:36:16.867519148Z run_metadata)
2022-06-27T06:36:16.867521656Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
2022-06-27T06:36:16.867524457Z raise type(e)(node_def, op, message)
2022-06-27T06:36:16.867527053Z tensorflow.python.framework.errors_impl.DataLossError: 2 root error(s) found.
2022-06-27T06:36:16.867529858Z (0) Data loss: Unable to open table file /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.867532909Z [[node save/RestoreV2 (defined at src/test.py:30) ]]
2022-06-27T06:36:16.867535905Z (1) Data loss: Unable to open table file /home/jovyan/work/src/output/models: Failed precondition: /home/jovyan/work/src/output/models; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?
2022-06-27T06:36:16.86753885Z [[node save/RestoreV2 (defined at src/test.py:30) ]]
2022-06-27T06:36:16.867541735Z [[save/RestoreV2/_17]]
2022-06-27T06:36:16.86754431Z 0 successful operations.
2022-06-27T06:36:16.867547008Z 0 derived errors ignored.
2022-06-27T06:36:16.86754958Z
2022-06-27T06:36:16.8675522Z Original stack trace for 'save/RestoreV2':
2022-06-27T06:36:16.867554898Z File "src/test.py", line 30, in <module>
2022-06-27T06:36:16.86755794Z saver = tf.train.Saver()
2022-06-27T06:36:16.867560564Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 836, in __init__
2022-06-27T06:36:16.867563425Z self.build()
2022-06-27T06:36:16.867565938Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 848, in build
2022-06-27T06:36:16.867580705Z self._build(self._filename, build_save=True, build_restore=True)
2022-06-27T06:36:16.867583844Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 886, in _build
2022-06-27T06:36:16.867586717Z build_restore=build_restore)
2022-06-27T06:36:16.86758929Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 516, in _build_internal
2022-06-27T06:36:16.867592137Z restore_sequentially, reshape)
2022-06-27T06:36:16.867594694Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 336, in _AddRestoreOps
2022-06-27T06:36:16.867597486Z restore_sequentially)
2022-06-27T06:36:16.867600011Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 583, in bulk_restore
2022-06-27T06:36:16.867602824Z return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
2022-06-27T06:36:16.867605422Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1524, in restore_v2
2022-06-27T06:36:16.867608998Z name=name)
2022-06-27T06:36:16.867611624Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
2022-06-27T06:36:16.867614426Z attrs=attr_protos, op_def=op_def)
2022-06-27T06:36:16.867616967Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3485, in _create_op_internal
2022-06-27T06:36:16.8676198Z op_def=op_def)
2022-06-27T06:36:16.867622461Z File "/home/jovyan/.virtualenvs/basenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
2022-06-27T06:36:16.867625257Z self._traceback = tf_stack.extract_stack()
2022-06-27T06:36:16.86762792Z
2022-06-27T06:36:17.563235506Z SYSTEM: Finishing...