Recovery tasks can create excessively large chunks, resulting in out of disk

[bryanlb]

Describe the bug

Recovery tasks are created based on message counts, which is a static config value. If this value is not correctly optimized, it can cause the cache nodes to download too much data and crash with an out-of-disk.

Expected behavior

There are several ways we could resolve this - the cache nodes could dynamically calculate and advertise slots based on available disk space; the recovery process could use disk size instead of message counts (somewhat complex to create tasks based on an anticipated size).

Of these options the former may be the most attractive, as it also accounts for the inverse issue where lots of small recovery tasks take up slots but not much disk space.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days.