Hang and correctness issue observed with PME-PP force transfers NVSHMEM runs in certain cases

Summary

Hang is observed with PME-PP force transfers NVSHMEM runs in cases when iterThreads < 32 in the pme_gather kernel. And correctness issue is seen for certain regression tests where numGrids == 2 in the pme_gather kernel.

If this refers to a more general issue or feature, is it important to fix right now, and would you have time to help?

Exact steps to reproduce

Build with GMX_NVSHMEM=ON. For run, set GMX_ENABLE_NVSHMEM=1, and regressiontests env variables as below export REGRESSIONTEST_PME_RANK_NUMBER=1 export REGRESSIONTEST_TOTAL_RANK_NUMBER=2 export REGRESSIONTEST_OMP_RANK_NUMBER=1 export REGRESSIONTEST_PARALLEL="-np" export GMX_ENABLE_DIRECT_GPU_COMM=1

Run: perl gmxtest.pl $REGRESSIONTEST_PARALLEL $REGRESSIONTEST_TOTAL_RANK_NUMBER -ntomp $REGRESSIONTEST_OMP_RANK_NUMBER -npme $REGRESSIONTEST_PME_RANK_NUMBER $REGRESSIONTEST_DOUBLE $REGRESSIONTEST_MPI_RUN_COMMAND -xml all

Possible fixes

Filed fix MR - !4466 (merged)