## Summary `gmxapi` imports `mpi4py.MPI` (resulting in a call to `MPI_Init_thread`) more than it needs to. Other than just being annoying, this can cause strange interference with compute resource management in conjunction with process forks during common (but surprisingly complex) tasks like simply installing the package or client packages. `gmxapi` 0.3 is also affected. ## Exact steps to reproduce ```python >>> import sys >>> from gmxapi.utility import config >>> "mpi4py" in sys.modules True >>> hasattr(sys.modules["mpi4py"], "MPI") True >>> sys.modules["mpi4py"].MPI.Is_initialized() True ``` ## For developers: Why is this important? By default, `import mpi4py.MPI` calls `MPI_Init_thread` to initialize MPI. `gmxapi` has some module level mpi4py imports. 1. We should be more conservative about the circumstances under which `mpi4py.MPI` is imported. 2. We might want to be more conservative about automatically initializing MPI when importing `mpi4py.MPI`. 3. We probably shouldn't assume that MPI is initialized, just because we imported `mpi4py.MPI`, since the automatic behavior can be suppressed by a user or another module that is imported earlier. See also * https://mpi4py.readthedocs.io/en/stable/mpi4py.html#mpi4py.mpi4py.rc.initialize * https://mpi4py.readthedocs.io/en/stable/mpi4py.html#mpi4py.mpi4py.rc.finalize * https://mpi4py.readthedocs.io/en/stable/mpi4py.run.html#module-mpi4py.run * https://mpi4py.readthedocs.io/en/stable/reference/mpi4py.MPI.Is_initialized.html#mpi4py.MPI.Is_initialized ### Additional considerations The `mpi4py.run` wrapper (e.g. `python -m mpi4py myscript.py`) makes sure that `MPI_Abort` gets called if an exception escapes unhandled from `myscript.py`. A wrapped script will see `mpi4py` in `sys.modules`, but the wrapper does not import `mpi4py.MPI` or call `MPI_Init`. Any software component that calls `MPI_Init` should ensure that `MPI_Finalize` will be called. But we don't want a library to call `MPI_Finalize` too aggressively or prematurely. Is there a situation in which `brer-md` needs to insist on initalizing MPI (now or in the future)? If so, how should we make sure that MPI is finalized? Note that if we inspect `"mpi4py in sys.modules`, we can see whether mpi4py has already been imported and whether there is a chance that `mpi4py.rc` was adjusted, so we can save existing values before setting `mpi4py.rc.initialize = False` and/or `mpi4py.rc.finalize = False`. But there is not necessarily a clear scope at which to restore the caller's expected behavior if our plan is to `import mpi4py.MPI`. Whatever we decide to do, we should document that we `import mpi4py.MPI` and what affect we expect to have on that module's configuration and state. ## If this is a bug, (1) what happens, and (2) what did you expect to happen? We should not provoke libfabric to complain for things like package installation. We should be able to perform some basic actions, like checking the installed configuration, without triggering MPI initialization. In other cases, we might expect that gmxapi does not actually trigger MPI initialization unless work is actually being launched. We can at least try to delay initialization until MPI is actually needed rather than as modules are imported. ## Possible fixes A series of patches eliminates MPI initialization at import. * !3415 * !3416 * !3417 * !3418 I don't think there are any barriers to upgrading from gmxapi 0.3 to 0.4, so I don't think it is necessary or appropriate to back port a fix. See below for a possible problematic interaction---and mitigation option---with certain versions of libfabric. ## See also https://github.com/kassonlab/brer-md/issues/47 illustrates an error from libfabric 1.11.0 on the GitHub Actions compute resources. As a workaround, the caller can force libfabric to tolerate the `fork()` with `RDMAV_FORK_SAFE=1` in libfabric < 1.13 and with `FI_EFA_FORK_SAFE=1` in more recent versions. * For libfabric errors before the patches, see https://github.com/kassonlab/brer-md/actions/runs/3757421206/attempts/2 * For the RDMAV_FORK_SAFE workaround, see https://github.com/kassonlab/brer-md/actions/runs/3757704789 * For the mitigation provided by the above-linked merge requests, see https://github.com/kassonlab/brer-md/actions/runs/3772006950, or compare the exact same job with gmxapi 0.4.0rc1 vs. gmxapi 0.4.0rc2: https://github.com/kassonlab/brer-md/actions/runs/3757421206/jobs/6384661472 versus https://github.com/kassonlab/brer-md/actions/runs/3757421206/jobs/6412521225