-
Notifications
You must be signed in to change notification settings - Fork 602
Description
Recent Python releases on macOS and Linux use the spawn start‑method by default for new processes.
IIRC spawn starts a brand‑new interpreter and imports the target module it relies on pickling to transport the target function.
As only module‑level objects can be pickled and run_test is defined inside the if __name__ == "__main__" block, the child process tries to unpickle and look up as an attribute of the module __mp_main__. That attribute does not exist, and the unpickler raises:
Process ForkServerProcess-2:
File "/usr/lib/python3.14/multiprocessing/process.py", line 320, in _bootstrap
self.run()
~~~~~~~~^^
File "/usr/lib/python3.14/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.14/concurrent/futures/process.py", line 242, in _process_worker
call_item = call_queue.get(block=True)
File "/usr/lib/python3.14/multiprocessing/queues.py", line 120, in get
return _ForkingPickler.loads(res)
~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__mp_main__' has no attribute 'run_test'
I am not filing a pull request because there are at least three fixes, and which one is implemented depends on the project team's preferences.
The options I see that fit the file's stated minimal changes from upstream constraint:
- Move
run_testto the module level. - As this is I/O bound, change to the
ThreadPoolExecutor - Force the use of
fork()
import multiprocessing
multiprocessing.set_start_method('fork', force=True)All three options seem to benchmark close to each other on Linux/MacOS. Personally I lean towards the ThreadPoolExecutor option as that is my personal default for I/O bound concurrent operations.