Tensorrt in Python: Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)

[Eric-Zhang1990]

Description

When I run my code (accelerating caffe model using tensorrt) with detection network (using caffe2), this error occured: "ERROR: ../rtSafe/safeContext.cpp (133) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception".

Environment

TensorRT Version: 6.0.1.5
GPU Type: 2080 ti
Nvidia Driver Version: 418
CUDA Version: 10.0
CUDNN Version: 7.6.2
Operating System + Version: ubuntu 16.04
Python Version (if applicable): 2.7
Baremetal or Container (if container which image + tag): runing in docker

Can somebody help me? Thanks!

[rmccorm4]

Hi @Eric-Zhang1990,

Can you share the the model and the code you're running to repro the issue so I can further debug?

[Eric-Zhang1990]

@rmccorm4 I can run it correctly in my own computer, but when I load two models (one is caffe2 for detecting, one is my trt model), this error occurs, the code includes other infos, sorry.

[Eric-Zhang1990]

@rmccorm4 I mean, tensorrt model can not be loaded with other model together??

[Eric-Zhang1990]

@rmccorm4 This is my trt code, can you help me check the code? does it have error on it?
Thank you!
caffe_tensorrt.txt

[rmccorm4]

Hi @Eric-Zhang1990,

Can you share the output of nvidia-smi on your machine / in the container? I'm curious if the GPUs are in EXCLUSIVE_PROCESS mode, similar to here: #294

[Eric-Zhang1990]

@rmccorm4 This is output of nvidia-smi, I use card 5 for caffe2 detection, card 7 for my trt classify.

pid is 34544.

[rmccorm4]

@Eric-Zhang1990 can you show the full output? I wanted to see the top half as well to see each GPU's settings

[Eric-Zhang1990]

@rmccorm4 Ok.

[Eric-Zhang1990]

@rmccorm4 Can you see the picture?

[Eric-Zhang1990]

@rmccorm4 I have seen the issue #294 , but when I run it on one device , the error also occurs.

[rmccorm4]

Hi @Eric-Zhang1990,

I'm not quite sure what the issue is. The only thing that stands out to me is that nvidia-smi reports CUDA 10.2 + Driver 430, but CUDA 10.2 requires driver >= 440.33 per this table: https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver.

Compatability mode might help in this scenario with specific lower driver versions like 418, but that only applies to Tesla GPUs (V100, T4, etc.) and you have a GeForce GPU (RTX 2080)

Can you try upgrading your driver to >= 440.33? (or use CUDA 10.0 instead of 10.2 for both your CUDA version and for the TensorRT download as well which should be easy since you're using containers)

[Eric-Zhang1990]

@rmccorm4 Thank you for your kind reply, I will have a try.