Different results from free energy perturbation simulations when run on GPU/CPUs vs. CPU only

Summary

We obtain different results from free energy simulations when running Gromacs only on CPUs or on CPUs/GPU. Specifically we used Gromacs version 2022.3, CUDA version 11.6.2 and free energy perturbation simulations using bar. Running only on CPUs, we reproduce the correct result from the literature for the solvation free energy of a magnesium ion in water (Grotz et al. J. Chem. Theory Comput. 2021, 17, 2530-2540). Running the exact same input files on one GPU and CPUs, the solvation free energies are 8 to 9 kJ/mol too high. In summary, we obtain: -1829.7 (for simulations on CPU/GPU) or -1838.5 (CPU only). Moreover, older gromacs versions (2019.6 and 2020.7) reproduce the correct values when running on CPUs/GPUs (or only on CPUs). The bugfix reported in !3164 (merged) did not resolve this issue, as we have also run these simulations with version 2022.4 and the beta release of 2023 on CPUs/GPU, but the results with GPU remain inconsistent relative to CPU only simulations.

GROMACS version

2022.3 and 2022.4

Steps to reproduce

The input files for reproduction of these results are included in the attached zip file: FEPInput.zip. The required simulations can be run by executing the included bash script and also the analysis with the BAR method is performed.

What is the current bug behavior?

The values for DG reported at the end of the file bar.log deviate between simulations run on CPUs only and on GPU/CPUs.

What did you expect the correct behavior to be?

That the values for DG deviate by not more than some statistical uncertainty, i.e. by less than 1 kJ/mol.

Possible fixes

No suggestions.