SSD example CUDA error: too many resources requested for launch

[ysh329]

It's okay to run CPU version SSD exmaple, but there's something wrong with CUDA. The main cuda error log is: example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Environment info

Operating System: TegraX1 Ubuntu 16.04.2 LTS
Linux tegra-ubuntu 3.10.96-tegra

Compiler: gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4)

Package used (Python/R/Scala/Julia):python

MXNet version:0.9.4

Or if installed from source: Yes

MXNet commit hash (git rev-parse HEAD):

If you are using python package, please provide

Python version and distribution: Python 2.7.12

Error Message:

Please paste the full error message, including stack trace.
(py-mxnet-SSD) yuanshuai@tegra-ubuntu:~/sdcard/code/mxnet_SSD_cuda_openblas/example/ssd$ python demo.py --epoch 0 --images ./data/demo/dog.jpg --thresh 0.5
[09:58:45] src/nnvm/legacy_json_util.cc:175: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 5 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d0) [0x7f74095db0]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (3) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (4) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

terminate called after throwing an instance of 'dmlc::Error'
what(): [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 5 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d0) [0x7f74095db0]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (3) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (4) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

Aborted

[ysh329]

I found there may be something wrong with mxnet-ssd code version.

I fixed it after I clone the original mxnet-ssd contributor's repo from his own code repo(https://github.com/zhreshold/mxnet-ssd) and then tried as its README says, finally it worked!

[ysh329]

Before I fixed it, I referred this issue titled src/operator/multibox_detection.cu:198: Check failed: (error) == (cudaSuccess) too many resources requested for launch · Issue #13 · zhreshold/mxnet-ssd, however it didn't work for me. Because there is no make files in official ./mxnet/examples/ssd dir, I couldn't compile the modified cuda codes.