Works with tf<2.5. tf>=2.5 requires some minor fixes.
With tf==2.6, watcher.py throws the following exception:
➜ [private] tails git:(master) ✗ docker exec -it tails_sentinel_1 bash
(base) root@31f5ac54e255:/app# python watcher.py
2021-09-17 02:52:19.755559: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-17 02:52:19.755861: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "watcher.py", line 925, in <module>
fire.Fire(sentinel)
File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "watcher.py", line 811, in sentinel
dask_client.register_worker_plugin(worker_initializer, name="worker-init")
File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 4113, in register_worker_plugin
return self.sync(
File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 840, in sync
return sync(
File "/opt/conda/lib/python3.8/site-packages/distributed/utils.py", line 326, in sync
raise exc.with_traceback(tb)
File "/opt/conda/lib/python3.8/site-packages/distributed/utils.py", line 309, in f
result[0] = yield future
File "/opt/conda/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 4034, in _register_worker_plugin
raise exc.with_traceback(tb)
File "watcher.py", line 708, in setup
self.tails_worker = TailsWorker()
File "watcher.py", line 121, in __init__
self.model = Tails()
File "/app/tails/efficientdet.py", line 1473, in Tails
features = EfficientNetB0(input_tensor=x_input, weights=None)
File "/app/tails/efficientdet.py", line 560, in EfficientNetB0
return EfficientNet(
File "/app/tails/efficientdet.py", line 475, in EfficientNet
if not is_keras_tensor(input_tensor):
File "/opt/conda/lib/python3.8/site-packages/tensorflow-2.6.0-py3.8-linux-x86_64.egg/tensorflow/python/keras/backend.py", line 1281, in is_keras_tensor
raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) +
ValueError: Unexpectedly found an instance of type `<class 'keras.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance.
Works with
tf<2.5.tf>=2.5requires some minor fixes.With
tf==2.6,watcher.pythrows the following exception: