S3 SDK list_objects_v2 pagination would hang after a long time of running. How to debug?

[jrcavani]

I'm trying to use the paginated listObjectsV2 API to query a flat list of UUID types in a bucket on S3, by permuting the first four hex digits as the prefix, like 0000, 0001, ..., ffff totaling 65536. There are two ways to structure the iterations:

1. Keep using tokio, and spawn many tasks with a TaskTracker bound by a semaphore.
2. Use tokio's current_thread to provide the sync bridge, and use rayon's iter_par() to go through the prefix list.

Both lead to hanging for some prefixes. Because I go through all the pages to build a result vector for each prefix before writing to file, I can see some files are never started, indicating some tasks just hang going through pages. I bubble up the errors by using anyhow, so I should be seeing something if there is any.

It's an external cloud storage that adopts the S3 protocol. It's possible some servers just ate up the requests. What I've tried to resolve some occurrences successfully is explicitly setting the timeout operation_attempt_timeout.

I would like to find a way to debug this, or have some community insight on what could be hanging, and resolve the issue.

In addition, what's the best timeout config to set in order to counter occasional S3 server glitches while ensuring the call eventually succeeds? operation_attempt_timeout seems to do exactly that, with infinite retries (is it possible in the SDK natively?). What are the uses for other timeout settings?

The code is in a repo: https://github.com/jrcavani/s3-ops-rust/blob/main/src/bin/generate-obj-manifest.rs (or https://github.com/jrcavani/s3-ops-rust/blob/main/src/bin/generate-obj-manifest.rs for the rayon version)

#611 seems related, but my situation looks to be some tasks hanging, using up the semaphore permits / rayon thread pool, instead of the async task executor hanging.

[jdisanti]

There is a very definite possibility of hanging if no operation timeout is set, so setting that timeout is definitely what you want to do. Finding the correct timeout value will require you to do some experimentation.