gromacs:gcc-11-cuda-11.7.1:buildMPI fails randomly with "undefined reference" to CUDA functions

Summary

gromacs:gcc-11-cuda-11.7.1:buildMPI job often fails with linking errors. Re-running typically helps.

Errors

/usr/bin/ld: warning: libcuda.so.1, needed by /opt/nvidia/hpc_sdk/Linux_x86_64/2022/math_libs/lib64/libcufftMp.so.10, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libnvidia-ml.so.1, needed by /opt/nvidia/hpc_sdk/Linux_x86_64/2022/math_libs/lib64/libcufftMp.so.10, not found (try using -rpath or -rpath-link)
...
/usr/bin/ld: /opt/nvidia/hpc_sdk/Linux_x86_64/2022/math_libs/lib64/libcufftMp.so.10: undefined reference to `cuEventCreate'
/usr/bin/ld: /opt/nvidia/hpc_sdk/Linux_x86_64/2022/math_libs/lib64/libcufftMp.so.10: undefined reference to `cuIpcCloseMemHandle'
/usr/bin/ld: /opt/nvidia/hpc_sdk/Linux_x86_64/2022/math_libs/lib64/libcufftMp.so.10: undefined reference to `nvmlInit_v2'
/usr/bin/ld: /opt/nvidia/hpc_sdk/Linux_x86_64/2022/math_libs/lib64/libcufftMp.so.10: undefined reference to `cuDeviceGet'
[and so on, many different function]

Exact steps to reproduce

Example of failed jobs:

For developers: Why is this important?

Failing CI jobs are not nice. Especially if it happens regularly.

Edited by Andrey Alekseenko