GPU PME Spread pipelining and FEP: wrong charges for B state used
Summary
When PME GPU pipelining was added in !1356 (merged), in the spread kernel, the charges for grid B were always loaded without offsetting by pipelineAtomStart
. As a result, all stages except the first one used wrong charges for the B state when doing spreading.
Exact lines of code to look at
Grid A charge prefetching, everything ok: https://gitlab.com/gromacs/gromacs/-/blob/d0bf2c16479d98b3b2c54f3c4c27d0b62a218b78/src/gromacs/ewald/pme_spread.cu#L225
Grid B charge prefetching, no offset applied: https://gitlab.com/gromacs/gromacs/-/blob/d0bf2c16479d98b3b2c54f3c4c27d0b62a218b78/src/gromacs/ewald/pme_spread.cu#L287
Prefetching is always enabled in CUDA: https://gitlab.com/gromacs/gromacs/blob/f327eecc60bcc487fda0fc157011761837db7909/src/gromacs/ewald/pme_gpu_calculate_splines.cuh#L113-113
For developers: Why is this important?
Using arbitrary charges for PME when doing FEP will produce incorrect results.
If this is a bug, (1) what happens, and (2) what did you expect to happen?
Relevant input files, logs and/or screenshots
Please attach at least the top of the GROMACS log file (it provides important information about the hardware and software versions), and likely a TPR file together with the raw GRO/MDP files required to regenerate it.
Possible fixes
pme_gpu_stage_atom_data<float, atomsPerBlock, 1>(sm_coefficients,
&kernelParams.atoms.d_coefficients[1][kernelParams.pipelineAtomStart]);