kernel version does not match DSO version
I have a custom Docker image, which is based on "nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04". When I was starting a project a couple of months ago, I installed cuda 10.1.168-1 on host machine.
NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1
But now when I run my image on the same host, it gives me an error:
2019-10-07 09:13:00.903418: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2019-10-07 09:13:00.903440: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: predictions-worker-987657b8d-68mbs 2019-10-07 09:13:00.903445: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: predictions-worker-987657b8d-68mbs 2019-10-07 09:13:00.903850: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 418.87.0 2019-10-07 09:13:00.903875: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 418.67.0 2019-10-07 09:13:00.903880: E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:306] kernel version 418.67.0 does not match DSO version 418.87.0 -- cannot find working devices in this configuration
I suppose that there is a version mismatch between host and container.
apt-get shows me that my cuda on host can be updated to 10.1.243-1, which will most likely fix the problem now.
Seems like you update "nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04" image by overriding the same tag. It might break the running system, because image provides the latest version, which might be incompatible with host version. I guess it can be fixed by updating cuda on host, but I will have to do it every time on every machine when there is a new update from you.
Is there a way to lock to a specific version of "nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04" ? I know it is already a tag, but because it is updated without any version increment, it basically behaves like "latest" and I am not able to lock to specific version.