Matt Schramm opened issue #5289: numOpenmpThreadsForClearing_ error at GROMACS / GROMACS

Matt Schramm
2025-02-11T15:22:15Z

Summary

When trying to run older models (legacy benchmark files), I am getting the below error. This is the vanilla build pulled from https://gitlab.com/gromacs/gromacs/-/archive/release-2025/gromacs-release-2025.tar.gz.

GROMACS Build Options for Rocky Linux 8:

cmake \
-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.2 \
-DGMX_BUILD_OWN_FFTW=ON \
-DREGRESSIONTEST_DOWNLOAD=OFF \
-DGMX_GPU=CUDA \
-DGMX_MPI=on \
-DGMX_THREAD_MPI=off \
-DMPI_CXX_COMPILER=/opt/openmpi/bin/mpicxx \
-DCMAKE_INSTALL_PREFIX=/opt/gromacs/bin/avx2 \
-DGMX_SIMD=AVX2_256

Seen Error:

Program: gmx mdrun, version 2025.0-dev
Source file: src/gromacs/domdec/hashedmap.h (line 127)
Function: gmx::HashedMap<gmx_ga2la_t::Entry>::HashedMap(int, int)::<lambda()>
MPI rank: 2 (out of 16)

Assertion failed:
Condition: numOpenmpThreadsForClearing_ >= 1
Need at least one thread

Exact steps to reproduce

The error is generated by running the command:

OMP_NUM_THREADS=1 mpirun -np 16 -x PATH -x LD_LIBRARY_PATH /opt/gromacs/bin/avx2/bin/gmx_mpi mdrun -s /home/user/benchPEP.tpr -quiet -pin on -tunepme -maxh 0.0167

File was downloaded from: https://www.mpinat.mpg.de/grubmueller/bench

Relevant input files, logs and/or screenshots

Top of Log (Please let me know if you would like additional information):

GROMACS: gmx mdrun, version 2025.0-dev
Executable: /opt/gromacs/bin/avx2/bin/gmx_mpi
Data prefix: /opt/gromacs/bin/avx2
Working dir: /home/user
Process ID: 1619
Command line:
gmx_mpi mdrun -s /home/user/benchPEP.tpr -pin on -tunepme -maxh 0.0167

GROMACS version: 2025.0-dev
Precision: mixed
Memory model: 64 bit
MPI library: MPI
MPI library version: Open MPI v4.1.6rc4, package: Open MPI root@13bdafd8d0a3 Distribution, ident: 4.1.6rc4, repo rev: v4.1.6, Unreleased developer copy
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support: CUDA
NBNxM GPU setup: super-cluster 2x2x2 / cluster 8 (cluster-pair splitting on)
SIMD instructions: AVX2_256
CPU FFT library: fftw-3.3.10-sse2-avx-avx2-avx2_128
GPU FFT library: cuFFT
Multi-GPU FFT: none
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
C compiler: /opt/rh/gcc-toolset-12/root/usr/bin/cc GNU 12.2.1
C compiler flags: -Wno-array-bounds -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wall -Wno-unused -Wunused-value -Wunused-parameter -Wextra -Wno-sign-compare -Wpointer-arith -Wpedantic -Wundef -Werror=stringop-truncation -Wno-missing-field-initializers -pthread -O3 -DNDEBUG
C++ compiler: /opt/rh/gcc-toolset-12/root/usr/bin/c++ GNU 12.2.1
C++ compiler flags: -Wno-array-bounds -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wall -Wextra -Wpointer-arith -Wmissing-declarations -Wpedantic -Wundef -Wstringop-truncation -Wno-missing-field-initializers -pthread -Wno-old-style-cast -Wno-cast-qual -Wno-suggest-override -Wno-suggest-destructor-override -Wno-zero-as-null-pointer-constant -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
BLAS library: External - detected on the system
LAPACK library: External - detected on the system
CUDA compiler: /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2023 NVIDIA Corporation;Built on Tue_Aug_15_22:02:13_PDT_2023;Cuda compilation tools, release 12.2, V12.2.140;Build cuda_12.2.r12.2/compiler.33191640_0
CUDA compiler flags: -O3 -DNDEBUG
CUDA driver: 0.0
CUDA runtime: 12.20

Running on 1 node with total 16 cores, 32 processing units (GPU detection failed: No valid CUDA driver found)
Hardware detected on host add6584c236b (the node of MPI rank 0):
CPU info:
Vendor: AMD
Brand: AMD Ryzen 9 5950X 16-Core Processor
Family: 25 Model: 33 Stepping: 0
Features: aes amd apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt lahf misalignsse mmx msr nonstop_tsc pclmuldq pdpe1gb popcnt pse rdrnd rdtscp sha sse2 sse3 sse4a sse4.1 sse4.2 ssse3
Hardware topology: Basic
Packages, cores, and logical processors:
[indices refer to OS logical processors]
Package 0: [ 0 16] [ 1 17] [ 2 18] [ 3 19] [ 4 20] [ 5 21] [ 6 22] [ 7 23] [ 8 24] [ 9 25] [ 10 26] [ 11 27] [ 12 28] [ 13 29] [ 14 30] [ 15 31]
CPU limit set by OS: -1 Recommended max number of threads: 32