Incorrect forces from the pull code with more than 32 DD ranks

I have found some problem with pulling code (specifically dihedral pulling) in the current master branch of Gromacs. As for now I found it only if mdrun running without separate PME rank, which means gmp_mpi mdrun -npme 0 At random timestep total energy becomes "nan" and sim_util.cpp catches that in such kind of a message:

Source file: src/gromacs/mdlib/sim_util.cpp (line 537) MPI rank: 0 (out of 80)

Fatal error: Step 29: The total potential energy is nan, which is not finite. The LJ and electrostatic contributions to the energy are 5381.59 and -42407.4, respectively. A non-finite potential energy can be caused by overlapping interactions in bonded interactions or very large or Nan coordinate values. Usually this is caused by a badly- or non-equilibrated initial configuration, incorrect interactions or parameters in the topology.

I have investigated which energy component causes that and found it to be pulling energy (component 74). Coordinates of the system are fine, restarting from the last geometry also works fine, but then fails again after some time at different geometry. Because of the randomness and dependence on PME configuration, I think that it could be some deeper problem, possibly with memory leaks/corruption. For example, running the same file on a different node causes crash at a different timestep.

I have attached tpr file which has such a problem, it has been built with Gromacs 2020.2, the system is quiet big, but usually it crashes within 100-200 steps.

This affects both single/double precision versions. Also affects Gromacs 2020.2 Release as well as current master branch with different build setups.

pulling_bug.tpr

added 2020 Bug + 1 deleted label

changed the description

assigned to @acmnpv

@aracsmd what build configuration and execution commandline did you use?

CMake options: -DGMX_MPI=ON -DGMX_GPU=OFF -DGMX_USE_RDTSCP=ON -DGMX_HWLOC=OFF

build configuration: gcc/9.1.0, OpenMPI 4.x compatible (hpcx-mpi 2.4.0 is exact thing)

run commandline: srun -n $SLURM_NPROCS gmx_mpi mdrun -npme 0 -s pulling_bug.tpr

@acmnpv Any luck with locating that thing?

no, I tried running this on my machine but it didn't trigger the bug

@acmnpv I have an important update. It does not depend whether you use "-npme 0" or not. It purely depend on an actual number of PP ranks which you have in you simulation! For now I see problem happening when I have 36 or 40 PP (or mixed PP/PME ranks) used for job. So my conclusion for now that it is most probably Domain Decomposition problem. Seems like pulling code misses some coordinates of actual pulling groups, because they fall into different DD cells.

@berkhess could that be an issue with higher dimensionality domain decomposition?

That must be an issue with the pull optimization where only a sub-set of the ranks participates. This kicks in when there are more than 32 DD ranks. This can be turned of with the env.var. GMX_PULL_PARTICIPATE_ALL I assume that the problem goes away when you use this.

Can you get a checkpoint file that can reproduce this issue in a small number of steps? Then I can debug it.

Yes I've just checked export GMX_PULL_PARTICIPATE_ALL=1 solves the problem, however there is no possibility to give you *.cpt which immediately crushes. Because, surprisingly, simulation continue normally from that kind of cpt, but then crushes after some random number of steps again

That is not so surprising, as the group of ranks that participate depends on the history to avoid often regenerating communicators.

Then what kind of checkpoint file you want me to provide? From the starting tpr structure it usually happens after 20-100 steps

That's fast enough. I hope it also happens without DLB.

well with -dlb no crash also happens, but only with more than 60 PP ranks.

assigned to @berkhess

changed title from Pulling code and PME configuration to Incorrect forces from the pull code with more than 32 DD ranks

mentioned in issue #3735 (closed)

resolved with !753 (merged)

closed

I forgot that there already was an issue and added #3750 (closed) which is more general and now resolved.

Yes I saw #3750 (closed) I think it also resolved and maybe should be closed.

mentioned in issue #3750 (closed)

removed 1 deleted label