Why is multiprocessing forced to spawn processes rather than forking

[mbsabath]

❓ The question

Hi all, I'm working on a change to OLMo to support use of additional Pytorch Dataset Classes in our fork of OLMO, and I'm getting some OOM errors due to the use of process spawning rather than forming. I'm considering making process start method configurable, but wanted to understand more about the reasons for forcing all multiprocessing to be done with spawn before I went ahead with the change.

[aman-17]

The OOM errors you’re encountering might stem from increased memory usage associated with spawn due to the duplication of resources when processes are initialized(not 100% sure). We used Memmap implementation to minimize memory storage. Maybe you can reduce the workers/batch-size to ease up.

[dirkgr]

The main reason is that torch isn't safe when you fork the process. It should be, but it is not.