Matt Schramm commented on issue #5542 at GROMACS / GROMACS

Matt Schramm
2026-01-21T14:58:39Z

Hello

Full nvidia-smi output:

Wed Jan 21 14:47:50 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06 Driver Version: 580.65.06 CUDA Version: 12.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P8 16W / 70W | 3MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:05.0 Off | 0 |
| N/A 53C P8 12W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:06.0 Off | 0 |
| N/A 55C P8 13W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:07.0 Off | 0 |
| N/A 57C P8 17W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

1:

mpirun -np 4 nvidia-smi
Wed Jan 21 14:49:33 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06 Driver Version: 580.65.06 CUDA Version: 12.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
Wed Jan 21 14:49:33 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06 Driver Version: 580.65.06 CUDA Version: 12.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
Wed Jan 21 14:49:33 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06 Driver Version: 580.65.06 CUDA Version: 12.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
Wed Jan 21 14:49:33 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06 Driver Version: 580.65.06 CUDA Version: 12.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P8 13W / 70W | 3MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:05.0 Off | 0 |
| N/A 53C P8 13W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:06.0 Off | 0 |
| N/A 56C P8 12W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:07.0 Off | 0 |
| N/A 56C P8 12W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P8 13W / 70W | 3MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:05.0 Off | 0 |
| N/A 53C P8 13W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:06.0 Off | 0 |
| N/A 56C P8 12W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:07.0 Off | 0 |
| N/A 56C P8 12W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P8 13W / 70W | 3MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:05.0 Off | 0 |
| N/A 53C P8 13W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:06.0 Off | 0 |
| N/A 56C P8 15W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:07.0 Off | 0 |
| N/A 56C P8 12W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 51C P8 13W / 70W | 3MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:05.0 Off | 0 |
| N/A 53C P8 15W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:06.0 Off | 0 |
| N/A 56C P8 15W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:07.0 Off | 0 |
| N/A 56C P8 17W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
| No running processes found |
+-----------------------------------------------------------------------------------------+
| No running processes found |
+-----------------------------------------------------------------------------------------+

2:

ldd /opt/gromacs/bin/avx2/bin/gmx_mpi
linux-vdso.so.1 (0x00007fffe7d45000)
libdlfaker.so => /lib64/libdlfaker.so (0x00000d4de35c8000)
libvglfaker.so => /lib64/libvglfaker.so (0x00000d4de322b000)
libgromacs_mpi.so.11 => /opt/gromacs/bin/avx2/bin/../lib64/libgromacs_mpi.so.11 (0x00000d4ddcf76000)
libmpi.so.40 => /opt/openmpi/lib/libmpi.so.40 (0x00000d4ddcbbf000)
libgomp.so.1 => /lib64/libgomp.so.1 (0x00000d4ddc987000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000d4ddc767000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00000d4ddc3d2000)
libm.so.6 => /lib64/libm.so.6 (0x00000d4ddc050000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000d4ddbe38000)
libc.so.6 => /lib64/libc.so.6 (0x00000d4ddba62000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000d4ddb85e000)
libGL.so.1 => /lib64/libGL.so.1 (0x00000d4ddb5d8000)
libEGL.so.1 => /lib64/libEGL.so.1 (0x00000d4ddb3c4000)
libXv.so.1 => /lib64/libXv.so.1 (0x00000d4ddb1be000)
libX11.so.6 => /lib64/libX11.so.6 (0x00000d4ddae7a000)
libXext.so.6 => /lib64/libXext.so.6 (0x00000d4ddac67000)
/lib64/ld-linux-x86-64.so.2 (0x00000d4de37ca000)
libcufft.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so.11 (0x00000d4dcff37000)
librt.so.1 => /lib64/librt.so.1 (0x00000d4dcfd2f000)
libopenblas.so.0 => /lib64/libopenblas.so.0 (0x00000d4dcda80000)
libmuparser.so.2 => /opt/gromacs/bin/avx2/bin/../lib64/../lib64/libmuparser.so.2 (0x00000d4de398b000)
libucc.so.1 => /opt/lib/libucc.so.1 (0x00000d4de3950000)
libopen-pal.so.80 => /opt/openmpi/lib/libopen-pal.so.80 (0x00000d4de3815000)
libfabric.so.1 => /opt/lib/libfabric.so.1 (0x00000d4dcd915000)
librdmacm.so.1 => /opt/lib64/librdmacm.so.1 (0x00000d4dcd8f7000)
libefa.so.1 => /opt/lib64/libefa.so.1 (0x00000d4de3805000)
libibverbs.so.1 => /opt/lib64/libibverbs.so.1 (0x00000d4dcd8d6000)
libucp.so.0 => /opt/lib/libucp.so.0 (0x00000d4dcd7ba000)
libuct.so.0 => /opt/lib/libuct.so.0 (0x00000d4dcd778000)
libucs.so.0 => /opt/lib/libucs.so.0 (0x00000d4dcd705000)
libucm.so.0 => /opt/lib/libucm.so.0 (0x00000d4dcd6e6000)
libcuda.so.1 => /opt/lib/libcuda.so.1 (0x00000d4dcba78000)
libnl-route-3.so.200 => /opt/lib/libnl-route-3.so.200 (0x00000d4dcb9e3000)
libnl-3.so.200 => /opt/lib/libnl-3.so.200 (0x00000d4dcb9c1000)
libpmix.so.2 => /opt/openmpi/lib/libpmix.so.2 (0x00000d4dcb79b000)
libutil.so.1 => /lib64/libutil.so.1 (0x00000d4dcb597000)
libevent_core-2.1.so.7 => /opt/openmpi/lib/libevent_core-2.1.so.7 (0x00000d4dcb561000)
libevent_pthreads-2.1.so.7 => /opt/openmpi/lib/libevent_pthreads-2.1.so.7 (0x00000d4de37fc000)
libhwloc.so.15 => /opt/lib/libhwloc.so.15 (0x00000d4dcb500000)
libpciaccess.so.0 => /opt/lib64/libpciaccess.so.0 (0x00000d4dcb4f4000)
libcudart.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12 (0x00000d4dcb24c000)
libxml2.so.16 => /opt/lib/libxml2.so.16 (0x00000d4dcb10d000)
libz.so.1 => /opt/lib/libz.so.1 (0x00000d4dcb0e7000)
libGLX.so.0 => /lib64/libGLX.so.0 (0x00000d4dcaeb5000)
libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00000d4dcabff000)
libxcb.so.1 => /lib64/libxcb.so.1 (0x00000d4dca9d6000)
libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00000d4dca555000)
libXau.so.6 => /lib64/libXau.so.6 (0x00000d4dca351000)
libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00000d4dca110000)

3:

/opt/openmpi/bin/mpirun -np 1 -x GMX_GPU_PME_DECOMPOSITION /opt/gromacs/bin/avx2/bin/gmx_mpi mdrun -s ./benchMEM.tpr -quiet -maxh 0.1 -tunepme -nb gpu -bonded gpu
Reading file ./benchMEM.tpr, VERSION 4.6.3-dev-20130701-6e3ae9e (single precision)
Note: file tpx version 83, software tpx version 138
Changing nstlist from 10 to 100, rlist from 1 to 1.133

Update groups can not be used for this system because there are three or more consecutively coupled constraints

-------------------------------------------------------
Program: gmx mdrun, version 2026.0-dev
Source file: src/gromacs/taskassignment/findallgputasks.cpp (line 89)

Fatal error:
Cannot run short-ranged nonbonded interactions on a GPU because no GPU is
detected.

For more information and tips for troubleshooting, please check the GROMACS
website at https://manual.gromacs.org/current/user-guide/run-time-errors.html
-------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
Proc: [[7616,1],0]
Errorcode: 1

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node jarvice-job-223949-fccgv calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------