Skip to content

ENH: Fix `MPI` class RecursionError w multiprocessing

Benjamin requested to merge (removed):multiprocess_fix into master

Hi, the use of the multiprocessing module with NEBs leads to RecursionError, as shown in the minimal example below. Using pickle on NEBs also gives the same error - I added another minimal example below for this. These problems when using multiprocessing and pickle seems to be due to __getattr__ being called on an empty instance of MPI, see for example, https://stackoverflow.com/a/34950256 and https://stackoverflow.com/a/50888571.

The RecursionError can be simply solved by checking if __getattr__ is trying to access the __setstate__ attribute, and raising an AttributeError if so. This merge request implements this fix.

Thanks for considering this!

Minimal example - RecursionError with multiprocessing

#!/usr/bin/env python

from ase import Atoms
from ase.neb import NEB
from multiprocessing import Pool

def print_neb_images(neb):
    print(neb.images)

images = [Atoms('H')] * 3
nebs = [NEB(images)] * 3

with Pool(2) as p:
    p.map(print_neb_images, nebs) # raises RecursionError

Output

Process ForkPoolWorker-2:
Traceback (most recent call last):
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/queues.py", line 354, in get
    return _ForkingPickler.loads(res)
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/site-packages/ase/parallel.py", line 92, in __getattr__
    if self.comm is None:
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/site-packages/ase/parallel.py", line 92, in __getattr__
    if self.comm is None:
  File "/home/benjamin/miniconda3/envs/test/lib/python3.7/site-packages/ase/parallel.py", line 92, in __getattr__
    if self.comm is None:
  [Previous line repeated 489 more times]
RecursionError: maximum recursion depth exceeded

Minimal example - RecursionError with pickle

#!/usr/bin/env python

import pickle
from ase import Atoms
from ase.neb import NEB

images = [Atoms('H')] * 3
nebs = [NEB(images)] * 3

pickled_nebs = pickle.dumps(nebs)
pickle.loads(pickled_nebs) # raises RecursionError
Edited by Benjamin

Merge request reports