Dynamic workers based on RAM usage

[thoughtfuldata]

Description

Change the number of workers based on RAM availability

Use case / motivation

I would like for mapply number of workers to be dynamic with memory intensive operations or at least to consider RAM availability

Related Issues

I have detailed the issue in #43. The main complaint there is that there is no message telling me that not sufficient RAM

[ddelange]

Hi @thoughtfuldata 👋

This sounds like a cool feature! However in reality I think it's near impossible to estimate RAM usage in advance, without running through it (and crashing) at least once beforehand.

Currently, the amount of workers is calculated in advance, and I think dynamic pool resize would mean a lot of new (low-level) code to write and maintain.

In these scenarios, I think your best bet is to run a small sample and then adjust n_workers accordingly for the big run.

Hope that helps!

[ddelange]

Probably a duplicate of upstream issue uqfoundation/pathos#243