[FIXED] Tensorflow - Train.py - ValueError: ('%s is not decorated with @add_arg_scope', ('nets.mobilenet.mobilenet', 'depth_multiplier'))

Issue

I'm trying to make a custom object-detection model following this tutorial: https://towardsdatascience.com/custom-object-detection-using-tensorflow-from-scratch-e61da2e10087

Everything worked fine until the Step 10, where I have to train my model using the train.py script (with tensorflow 1).

When I run the script (macOS) :

python3 train.py     --alsologtostderr     --train_dir=train     --pipeline_config_path=/Users/sol.arabehety/Desktop/models-master/ssd_mobilenet_v2_coco.config

I get this error:

File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 152, in arg_scope
    _name_op(op))
ValueError: ('%s is not decorated with @add_arg_scope', ('nets.mobilenet.mobilenet', 'depth_multiplier'))

I found this solution https://stackoverflow.com/a/53038290/1269404 but I don't know where I have to add the @add_arg_scope tag, can you help me?

This is the arg_scope.py file: https://github.com/google-research/tf-slim/blob/master/tf_slim/ops/arg_scope.py (line 152)

This is the complete error:

models-master$ sudo python3 train.py     --alsologtostderr     --train_dir=train     --pipeline_config_path=/Users/sol.arabehety/Desktop/models-master/ssd_mobilenet_v2_coco.config
/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/platform/app.py:125: main (from __main__) is deprecated and will be removed in a future version.
Instructions for updating:
Use object_detection/model_main.py.
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/legacy/trainer.py:266: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/core/preprocessor.py:188: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/core/batcher.py:96: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/training/input.py:753: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/training/input.py:753: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/training/input.py:784: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Traceback (most recent call last):
  File "train.py", line 186, in <module>
    tf.app.run()
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "train.py", line 182, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/legacy/trainer.py", line 291, in train
    clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
  File "/Users/sol.arabehety/Desktop/models-master/research/slim/deployment/model_deploy.py", line 192, in create_clones
    outputs = model_fn(*args, **kwargs)
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/legacy/trainer.py", line 204, in _create_losses
    prediction_dict = detection_model.predict(images, true_image_shapes)
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/meta_architectures/ssd_meta_arch.py", line 568, in predict
    preprocessed_inputs)
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/object_detection/models/ssd_mobilenet_v2_feature_extractor.py", line 118, in extract_features
    [mobilenet.depth_multiplier], min_depth=self._min_depth):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Users/sol.arabehety/Library/Python/3.7/lib/python/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 152, in arg_scope
    _name_op(op))
ValueError: ('%s is not decorated with @add_arg_scope', ('nets.mobilenet.mobilenet', 'depth_multiplier'))

Solution

Your link shows the problem. There the programmer had to find bottleneck function at main. So in your case you have to find depth_multiplier at nets.mobilenet.mobilenet. IF the last one is something that is downloaded during selection of backbone for the model then you are at a dead end. Sorry but you have to train with Tensorflow v2. and better to follow instructions at Tensorflow github. I am happy to help if you have.... a dataset.

If I wanted to train I would follow this. It is the safest solution with examples. Inside you can find how to bring your own dataset....which is the hardest part as it takes time to prepare it right!

Answered By - Farmaker