parallel forget opendal s3

[pierreedbrg]

Context:
rustic v0.9.5
no-cache
When running forget --filter-tags backupName:backup.GetName() --keep-none in parralel using 4 go routines, error may appear with message:

[INFO] Repository opendal:backups: password is correct.
[INFO] Using no cache.
[WARN] service=s3 name=backups path=snapshots/8c4f124122a7b8dffc5b57690336e5c7b3caea44bf4957592a259257eda457eb: stat failed NotFound (persistent).
[WARN] service=s3 name=backups path=snapshots/8e7e634885aac4a11c37c524310c4a880b4318f04cec191509da5b8f2254ef04: stat failed NotFound (persistent).
[WARN] service=s3 name=backups path=snapshots/453899b85fe7e8e2137bc0a90eb006af296581398621c21275320b02e010668b: stat failed NotFound (persistent).
[WARN] service=s3 name=backups path=snapshots/fe97472c3d0e8a8702af4c68a527671555752495886a68febf43c64270d7d6df: stat failed NotFound (persistent).

[ERROR] rustic_core encountered a backend-related error.

Message:
Reading file snapshots/8c4f124122a7b8dffc5b57690336e5c7b3caea44bf4957592a259257eda457eb failed in the backend. Please verify if the path is correct.

Context:

• Type: Snapshot
• Snapshot ID: 8c4f1241

Caused by:
NotFound (persistent) at stat, context:
URI: http://minio.minio:9000/backups/prod2/snapshots/8c4f124122a7b8dffc5b57690336e5c7b3caea44bf4957592a259257eda457eb
Response:

• HTTP Status: 404 Not Found

• MinIO Error Code: NoSuchKey

• Error Description: "The specified key does not exist."

• Same thing happens with index when running prune and backup at the same time (still using s3), is there a possilibty to add a retry mechanism ? for example for backuping, if i'm piping pg_dump to rustic backup stdin, i dont really want to re-run pg_dump command each time a backup fails because of index not found,

• Same error when running forget and backup in parallel, in backup command i get

[INFO] using no config file, none of these exist: /root/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml │
│ [INFO] repository opendal:backups: password is correct. │
│ [INFO] using no cache │
│ [WARN] service=s3 name=backups path=snapshots/a9a3e534b33d3e13c4c0a364c190cff93d4272d95775c45bee64a3e82d111c24: stat failed │
│ NotFound (persistent) at stat, context: { uri: http://minio.minio:9000/backups/prod2/snapshots/a9a3e534b33d3e13c4c0a364c19 │
│ 0cff93d4272d95775c45bee64a3e82d111c24, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes" │
│ , "content-length": "0", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Or │
│ igin", "vary": "Accept-Encoding", "x-amz-id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8", "x-amz- │
│ request-id": "1829E091F4840E17", "x-content-type-options": "nosniff", "x-minio-error-code": "NoSuchKey", "x-minio-error-des │
│ c": ""The specified key does not exist."", "x-xss-protection": "1; mode=block", "date": "Wed, 05 Mar 2025 10:14:08 GMT"} │
│ }, service: s3, path: snapshots/a9a3e534b33d3e13c4c0a364c190cff93d4272d95775c45bee64a3e82d111c24 }