ENH: Fix `MPI` class RecursionError w multiprocessing
Hi, the use of the multiprocessing
module with NEB
s leads to RecursionError
, as shown in the minimal example below.
Using pickle
on NEB
s also gives the same error - I added another minimal example below for this. These problems when using multiprocessing
and pickle
seems to be due to __getattr__
being called on an empty instance of MPI
, see for example, https://stackoverflow.com/a/34950256 and https://stackoverflow.com/a/50888571.
The RecursionError
can be simply solved by checking if __getattr__
is trying to access the __setstate__
attribute, and raising an AttributeError
if so. This merge request implements this fix.
Thanks for considering this!
Minimal example - RecursionError with multiprocessing
#!/usr/bin/env python
from ase import Atoms
from ase.neb import NEB
from multiprocessing import Pool
def print_neb_images(neb):
print(neb.images)
images = [Atoms('H')] * 3
nebs = [NEB(images)] * 3
with Pool(2) as p:
p.map(print_neb_images, nebs) # raises RecursionError
Output
Process ForkPoolWorker-2:
Traceback (most recent call last):
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/multiprocessing/queues.py", line 354, in get
return _ForkingPickler.loads(res)
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/site-packages/ase/parallel.py", line 92, in __getattr__
if self.comm is None:
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/site-packages/ase/parallel.py", line 92, in __getattr__
if self.comm is None:
File "/home/benjamin/miniconda3/envs/test/lib/python3.7/site-packages/ase/parallel.py", line 92, in __getattr__
if self.comm is None:
[Previous line repeated 489 more times]
RecursionError: maximum recursion depth exceeded
Minimal example - RecursionError with pickle
#!/usr/bin/env python
import pickle
from ase import Atoms
from ase.neb import NEB
images = [Atoms('H')] * 3
nebs = [NEB(images)] * 3
pickled_nebs = pickle.dumps(nebs)
pickle.loads(pickled_nebs) # raises RecursionError