Include swapped memory in process memory

[crusaderky]

Add swapped-out memory to all process memory measures.
This PR prevents incorrect warnings in the worker heartbeat (#7419).

Typically you won't swap out in dask. The most realistic ways to cause a swap out are:

1. set the memory_limit higher than the available memory / number of workers on the host. This makes sense if you expect that only one worker will hold most of the memory at any given time.
2. you have other stuff going on on the same host (e.g. it's a dev box) which causes the dask process to be swapped out.
3. you disabled dask spilling completely and you're relying on OS-level swap mechanics instead. In [DNM] File-based shared memory model #7449 I posted a benchmark that shows this is a viable option.

I'm unsure how swapping looks like on Windows or MacOSX - if a volunteer could allocate enough numpy arrays and then post what they read on memory_full_info(), I'll be able to incorporate.

No unit tests possible as you really don't want to write tens of GiB of memory on a dev box, and most CI boxes won't mount a swap file at all.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       22 files  ±  0         22 suites  ±0   9h 57m 29s ⏱️ + 19m 50s
  3 292 tests +  1    3 204 ✔️  - 2       86 💤 +1  2 ❌ +2 
36 143 runs  +11  34 580 ✔️ +6  1 561 💤 +3  2 ❌ +2 

For more details on these failures, see this check.

Results for commit 85a8983. ± Comparison against base commit 9d8e18c.

♻️ This comment has been updated with latest results.

[fjetter]

I don't understand why we can add this. swap is defined as the data that is swaped to disk

swap (Linux): amount of memory that has been swapped out to disk.

and is not actually in memory

[crusaderky]

We measure "process" memory for these purposes:

• Calculate unmanaged memory (bokeh + prometheus + AMM optimistic memory heuristics). If you don't add swap, then managed memory may exceed rss. You will also see be misled into thinking that your host mounts enough physical RAM, whereas it could benefit from more.
• terminate threshold. If the user configured the memory_limit so that spilling is possible, then we should expect that the memory_limit was in fact set to physical memory + part of the swap file. If you omit swap from your measure, then you may end up never hitting the terminate threshold while the swap mount fills up. When it's fully saturated, the whole host will hang.
• In addition to the previous point: it's entirely legitimate and performant to disable target, spill, and pause thresholds, leaving everything to the os swapping mechanism, and just leave the terminate threshold on.
• To appreciate how much memory you reclaimed aftr gc and after spilling. Again, you have no control if some of the memory was swapped out or not. As a matter of fact, if you have both spill and swap systems going on, it's very likely that dask will choose to spill memory that was swapped out to begin with - because both algorithms are LRU(ish). This is obviously poorly performant, but a very realistic use case on dev boxes where you have a web browser and your IDE running at least.
• To notice memory leaks in tests. Again, if you're leaking you might be leaking swapped out memory.

[fjetter]

If you don't add swap, then managed memory may exceed rss. You will also see be misled into thinking that your host mounts enough physical RAM, whereas it could benefit from more.

I'm not terribly concerned about this. If that was a concern and we wanted to raise awareness we should rather visualize swap on the dashboard

If you omit swap from your measure, then you may end up never hitting the terminate threshold while the swap mount fills up. When it's fully saturated, the whole host will hang.

True but this doesn't warrant us introducing a poorly/untested functionality. If this was a concern I would go for a better visualization instead of applying a "magical" fix

In addition to the previous point: it's entirely legitimate and performant to disable target, spill, and pause thresholds, leaving everything to the os swapping mechanism, and just leave the terminate threshold on.

Yes but this still doesn't explain why dask should add these values.

This is obviously poorly performant, but a very realistic use case on dev boxes where you have a web browser and your IDE running at least.

How would this help? if the OS starts spilling, the new process_memory definition would stay roughly the same and won't reduce, i.e. you will not help avoiding the "spill + swap" scenario. Quite the opposite, really since while RSS is lower and we might not spill if additional data is put on the cluster, this new metric would suggest that we're hitting the actual hard limit sooner and would increase the chance of entering this "spill+swap" space.

Besides, I would refrain from arguing about how the OS swaps. The OS swaps memory pages and is using algorithms we don't have any control over. We're operating at a way too high level of abstraction of trying to compare these systems

To notice memory leaks in tests. Again, if you're leaking you might be leaking swapped out memory.

[crusaderky]

Besides, I would refrain from arguing about how the OS swaps. The OS swaps memory pages and is using algorithms we don't have any control over. We're operating at a way too high level of abstraction of trying to compare these systems

Exactly. Dask should not know about how many pages are swapped and how many aren't.
Dask does care about how much memory dask itself (well, the Python interpreter actually) malloc'ed. That's what we call process memory. The total amount of malloc'ed memory is rss+swap.

[fjetter]

I'm unsure how swapping looks like on Windows or MacOSX

swapping in OSX is complicated and cannot be properly mapped to how it is understood in the linux context. OSX performs a lot of in-memory page compression and I'm not even sure if it is swapping in the traditional sense, i.e. writing to disk, at all. According to psutil docs, this is only valid for Linux

[crusaderky]

swapping in OSX is complicated and cannot be properly mapped to how it is understood in the linux context. OSX performs a lot of in-memory page compression and I'm not even sure if it is swapping in the traditional sense, i.e. writing to disk, at all. According to psutil docs, this is only valid for Linux

I know you don't have the "swap" measure in psutil. However I would like to know what psutil.Process().memory_full_info() returns when you have memory swapped to a compressed area and/or to disk. Does it all show up as RSS?