convert.py doesn't work with RTX

[antongisli]

I get an error very similar to the memory issues seen on the detect scripts, but trying to set the TF2 options for gpu memory usage doesn't help/work in convert.py (I tried to import tf2 there and set the options the same as others have done in the other scripts but no dice).

converting and getting the checkpoints on another machine (AWS instance) and copying them over let's me proceed with using the detect scripts, but would be nice to be able to run the conversion locally.

Windows 10, RTX2080.

Traceback (most recent call last):
  File "convert.py", line 40, in <module>
    app.run(main)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "convert.py", line 31, in main
    output = yolo(img)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 708, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 860, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 708, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 860, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\layers\convolutional.py", line 197, in call
    outputs = self._convolution_op(inputs, self.kernel)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 1134, in __call__
    return self.conv_op(inp, filter)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 639, in __call__
    return self.call(inp, filter)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 238, in __call__
    name=self.name)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2010, in conv2d
    name=name)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1031, in conv2d
    data_format=data_format, dilations=dilations, name=name, ctx=_ctx)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1130, in conv2d_eager_fallback
    ctx=_ctx, name=name)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
**tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=25600, n=32, k=64 [Op:Conv2D]**

[vectorvp]

Hey, Anton!
I've seen your post here: opencv/opencv#14827.
Did you solve the problem?

[antongisli]

Hi, No, I didn't look into it more. I started to look at the TF2 based yolo implementation instead. Cheers, Anton

…

On Fri, 8 Nov 2019 at 13:02, VectorVP ***@***.***> wrote: Hey, Anton! I've seen your post here: opencv/opencv#14827 <opencv/opencv#14827>. Did you solve the problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#94?email_source=notifications&email_token=AFR5TKFHX25WL4M5CNLFUG3QSVIMZA5CNFSM4JKN7WLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDQ5OQI#issuecomment-551671617>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFR5TKG45FLLYMSJD2WH6TLQSVIMZANCNFSM4JKN7WLA> .

[NiklasWilson]

@antongisli Do you happen to have a link to this? Id really like to take a look at.

TF2 based yolo implementation

[zzh8829]

have you tried the latest tensorflow 2.1 version, unfortunately i can't test it for you since i don't have RTX2080

[bainro]

Convert.py worked for me with Ubuntu 18.04 and an RTX2080, if that helps.

[MrPerdition]

Add these lines to models.py right after the imports

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
# Currently, memory growth needs to be the same across GPUs
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Memory growth must be set before GPUs have been initialized
print(e)

[tgohblio]

Add these lines to models.py right after the imports

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:

Currently, memory growth needs to be the same across GPUs

for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:

Memory growth must be set before GPUs have been initialized

print(e)

Thanks MrPerdition.

Your code snippet from TF site (https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth) manage to solve the issue and weights are converted. Here's my succesful output if anyone is interested.
(note: despite the CUDA out of memory error, weights are still converted)
...
2020-03-14 16:14:02.204447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-14 16:14:02.969893: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2020-03-14 16:14:03.003407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-03-14 16:14:03.510546: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 3.90G (4190535680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0314 16:14:03.683123 140557873211200 convert.py:26] sanity check passed
I0314 16:14:04.121551 140557873211200 convert.py:29] weights saved

---

Ubuntu 18.04 LTS, RTX2070, CUDA 10.2, TensorRT 7

[elmira784]

I have the same problem if you could solve it please help me as well :(

I get an error very similar to the memory issues seen on the detect scripts, but trying to set the TF2 options for gpu memory usage doesn't help/work in convert.py (I tried to import tf2 there and set the options the same as others have done in the other scripts but no dice).

converting and getting the checkpoints on another machine (AWS instance) and copying them over let's me proceed with using the detect scripts, but would be nice to be able to run the conversion locally.

Windows 10, RTX2080.

Traceback (most recent call last):
  File "convert.py", line 40, in <module>
    app.run(main)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "convert.py", line 31, in main
    output = yolo(img)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 708, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 860, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 708, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 860, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\layers\convolutional.py", line 197, in call
    outputs = self._convolution_op(inputs, self.kernel)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 1134, in __call__
    return self.conv_op(inp, filter)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 639, in __call__
    return self.call(inp, filter)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 238, in __call__
    name=self.name)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2010, in conv2d
    name=name)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1031, in conv2d
    data_format=data_format, dilations=dilations, name=name, ctx=_ctx)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1130, in conv2d_eager_fallback
    ctx=_ctx, name=name)
  File "C:\Users\anton\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
**tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=25600, n=32, k=64 [Op:Conv2D]**