Workers processes not executing any tasks with multiprocessing set to `spawn` (default for macOS and Windows)
The worker processes are not set up correctly if the multiprocessing method is set to spawn
instead of fork
, which is the default on macOS and Windows. The reason seems to be that the broker (main module) is not passed to dramatiq.cli:main
, so it is not imported in the sub-processes.
To reproduce (on Linux or macOS):
-
Do not start the broker (e.g. RabbitMQ).
-
Run this code somewhere in your application setup:
import multiprocessing
multiprocessing.set_start_method("spawn")
-
Start the worker with
flask worker
.
Expected result: The workers fail to start because they cannot connect to the broker.
Actual result: The workers start successfully, even though they cannot connect to the broker. However, they do not execute any tasks. Broker logs do not show any connections from the workers.
Workarounds:
- Option 1: Set
multiprocessing.set_start_method("fork")
(which is not available on Windows and deemed unsafe on macOS). - Option 2: Run worker with
dramatiq <name of flask main module>
.
Possible fix (not tested):
- Pass the flask main module to
dramatiq.cli:main
as thebroker
argument so it is imported when it spawns sub-processes. Currently,__name__
is passed instead, which does not lead to import of the flask main module.
I have tested this on Linux and macOS, but probably this also affects Windows (where fork
is not available).