libcudnn_cnn problem

[781574155]

INFO:faster_whisper:Processing audio with duration 03:52.176
Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}
Invalid handle. Cannot load symbol cudnnCreateConvolutionDescriptor

ctranslate2==4.5.0
torch==2.5.1
faster-whisper==1.1.0

[781574155]

after setup LD_LIBRARY_PATH, program run correctly.

export LD_LIBRARY_PATH=/usr/local/lib/python3.11/site-packages/nvidia/cudnn/lib/:$LD_LIBRARY_PATH

It's should be correctly setup automatically!

[zhou13]

Get the same problem.

On my machine, I have cuda 12.7, torch==2.5.1.
If I use ctranslate2==4.5.0 with torch, I got

Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}

If I use ctranslate2==4.4.0 with torch, I got

Could not load library libcudnn_ops_infer.so.8. Error: libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory

Using cuda with ctranslate2 or torch individually is fine. But you cannot invoke torch.cuda before ctranslate2.

The following files exist on my environment:

.venv/lib/python3.12/site-packages/nvidia/cudnn/lib/libcudnn_cnn.so.9
/usr/lib/libcudnn_cnn.so
/usr/lib/libcudnn_cnn.so.9
/usr/lib/libcudnn_cnn.so.9.5.1

Setting LD_LIBRARY_PATH="/path/to/.venv/lib/python3.12/site-packages/nvidia/cudnn/lib" works around the problem.

The problem does not exist when torch is not loaded. However, if I remove the system libcudnn under /usr/lib/, then ctranslate2 will fail even WITHOUT torch:

Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}

Weird problem. My hypothesis is that torch always loads /path/to/.venv/lib/python3.12/site-packages/nvidia/cudnn/lib but ld used by ctranslate2 tries to load /usr/lib/libcudnn_cnn.so. When used individually, it is fine. However, the later will fail if torch already loads the its cudnn in the same application.