Skip to content

echoclient.py fails on recent MacOS and Linux Python releases that default to the spawn start‑method #735

@gdahlm

Description

@gdahlm

Recent Python releases on macOS and Linux use the spawn start‑method by default for new processes.

IIRC spawn starts a brand‑new interpreter and imports the target module it relies on pickling to transport the target function.

As only module‑level objects can be pickled and run_test is defined inside the if __name__ == "__main__" block, the child process tries to unpickle and look up as an attribute of the module __mp_main__. That attribute does not exist, and the unpickler raises:

Process ForkServerProcess-2:
  File "/usr/lib/python3.14/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/usr/lib/python3.14/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/concurrent/futures/process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "/usr/lib/python3.14/multiprocessing/queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__mp_main__' has no attribute 'run_test'

I am not filing a pull request because there are at least three fixes, and which one is implemented depends on the project team's preferences.

The options I see that fit the file's stated minimal changes from upstream constraint:

  1. Move run_test to the module level.
  2. As this is I/O bound, change to the ThreadPoolExecutor
  3. Force the use of fork()
import multiprocessing
multiprocessing.set_start_method('fork', force=True)

All three options seem to benchmark close to each other on Linux/MacOS. Personally I lean towards the ThreadPoolExecutor option as that is my personal default for I/O bound concurrent operations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions