Skip to content

Update Tails' architecture to use TF >=2.6 #11

Description

@dmitryduev

Works with tf<2.5. tf>=2.5 requires some minor fixes.

With tf==2.6, watcher.py throws the following exception:

➜ [private]  tails git:(master) ✗ docker exec -it tails_sentinel_1 bash
(base) root@31f5ac54e255:/app# python watcher.py
2021-09-17 02:52:19.755559: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-17 02:52:19.755861: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "watcher.py", line 925, in <module>
    fire.Fire(sentinel)
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "watcher.py", line 811, in sentinel
    dask_client.register_worker_plugin(worker_initializer, name="worker-init")
  File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 4113, in register_worker_plugin
    return self.sync(
  File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 840, in sync
    return sync(
  File "/opt/conda/lib/python3.8/site-packages/distributed/utils.py", line 326, in sync
    raise exc.with_traceback(tb)
  File "/opt/conda/lib/python3.8/site-packages/distributed/utils.py", line 309, in f
    result[0] = yield future
  File "/opt/conda/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 4034, in _register_worker_plugin
    raise exc.with_traceback(tb)
  File "watcher.py", line 708, in setup
    self.tails_worker = TailsWorker()
  File "watcher.py", line 121, in __init__
    self.model = Tails()
  File "/app/tails/efficientdet.py", line 1473, in Tails
    features = EfficientNetB0(input_tensor=x_input, weights=None)
  File "/app/tails/efficientdet.py", line 560, in EfficientNetB0
    return EfficientNet(
  File "/app/tails/efficientdet.py", line 475, in EfficientNet
    if not is_keras_tensor(input_tensor):
  File "/opt/conda/lib/python3.8/site-packages/tensorflow-2.6.0-py3.8-linux-x86_64.egg/tensorflow/python/keras/backend.py", line 1281, in is_keras_tensor
    raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) +
ValueError: Unexpectedly found an instance of type `<class 'keras.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions