S3 export: uploads recorded as present without any content-integrity check (no ETag/MD5 comparison, HEAD-only presence check on fsck --fast)

[yarikoptic]

CORRECTION (2026-08-01)

Important

The original report below contained a factual error about how fsck --from <s3remote> behaves on exported files, corrected by @joeyh in this comment. fsck does download and hash the object; it is not HEAD-only. See the follow-up retraction comment and the strike-through edits below.

The upload-side observations (no Content-MD5 set, InfoPresent logged unconditionally, ETag never compared) are correct. The fsck-side framing was wrong.

---

Summary

Extra disclaimer as not posted from @yarikoptic-gitmate: it is not yet curated analysis by claude code, hinted on HI-to-HI communication, and my prior experiences in designing uploads for dandiarchive.org

Interest triggered by a discussion with OpenNeuro folk (cc @effigies) about relying on git annex fsck to guarantee that files in an S3-backed archive are actually intact. In the current S3 export code path, fsck --from <s3remote> cannot deliver that guarantee — see below. [CORRECTION: fsck does verify content by download+hash even for exports — see Command/Fsck.hs:163-217. The remaining concern is narrower: cost of periodic re-fsck on archive-scale exports, and the window between upload and first fsck during which InfoPresent is trusted.]

When git annex export uploads a file to an S3 remote (exporttree=yes), git-annex marks the file as present on the remote as soon as the HTTP upload returns without an exception. There is no end-to-end integrity check between the file's content and what actually landed in the bucket:

• The ETag returned by putObject / postCompleteMultipartUpload is captured but never compared against anything (Remote/S3.hs:394, Remote/S3.hs:433; discarded at Remote/S3.hs:550 via void $ storeExportS3').
• cleanupExport writes InfoPresent to the location log and updates the export DB unconditionally on non-exception return (Command/Export.hs:391-396).
• checkPresentExportS3 on the authenticated code path only does a HEAD request and returns true if any object metadata is present — no size, no checksum, no ETag compare (Remote/S3.hs:542-547, Remote/S3.hs:585-587). The public-URL fallback path does verify size via Url.checkBoth u (fromKey keySize k) at Remote/S3.hs:589-590, so the authenticated (common) case is actually weaker than the anonymous one. Because the exported object is keyed by tree-path (not by git-annex key/hash), a bare HEAD cannot detect content mismatch even in principle. [EDIT: this is about checkPresent in isolation — used by whereis-style queries and as fsck's first step. fsck itself does more than HEAD, see the CORRECTION above.]
• Only retrieveExportS3 verifies content (Remote/S3.hs:563-573, via verifyKeyContentIncrementally AlwaysVerify) — but by then the false InfoPresent has already been recorded, and other copies may have been dropped based on it. [EDIT: retrieveExportS3 is also what fsck --from invokes, so fsck triggers this verification too. Under --fast mode however, Command/Fsck.hs:207-210 skips the download and reverts to HEAD-only.]

So a partial / truncated / silently-corrupted S3 upload (buggy S3-compatible endpoint, transient network corruption, a botched multipart complete, etc.) can be recorded as a valid copy and only surface as an error much later, when the user tries to fetch it back [EDIT: … surface as an error only on the next full-download step — either a git annex get, or a fsck --from without --fast. During the window between upload and that next full-verify, InfoPresent is trusted for numcopies/whereis accounting.]

Realistic failure dataloss path

1. git annex export <tree> --to s3remote
2. putObject (or postCompleteMultipartUpload) returns HTTP 200 with an ETag; the object in S3 is truncated / corrupted / a leftover from a prior failed part upload.
3. storeExportS3 returns normally → cleanupExport records InfoPresent for that key at the S3 remote's UUID.
4. git annex fsck --from s3remote → checkPresentExportS3 → HEAD → object exists → "OK". [CORRECTION: git annex fsck --from s3remote (without --fast) downloads the object via retrieveKeyFile and hashes it against the key. Corruption is detected here — this step catches the fault. Under --fast, however, only the HEAD check runs and corruption goes undetected. So the failure path applies to (a) users who don't run fsck between the upload and their next drop-based-on-numcopies decision, and (b) users who run fsck --from --fast.]
5. User trusts S3 as a valid copy; local content is eventually dropped elsewhere on the strength of that.
6. Later git annex get --from s3remote finally trips verifyKeyContentIncrementally and discovers the hash mismatch — after the safety margin has already been eroded. [EDIT: or, equivalently, git annex fsck --from s3remote (non---fast) discovers it, at the cost of a full download of the object.]

Related but distinct from the well-known "S3 doesn't guarantee the ETag equals MD5 for encrypted / multipart objects" caveat — the issue here is that git-annex isn't attempting any comparison at all.

Suggested mitigations

Both strategies below are used in production by dandi/dandi-cli. Either or both could apply to git-annex's S3 export path:

1. Ask S3 to validate on its side via Content-MD5 / SigV4 payload hash. For single-part upload, git-annex knows (or can compute during upload) the MD5 of the object and can pass it as Content-MD5; S3 will reject the PUT with BadDigest if the body doesn't match. dandi-cli does exactly this for the small-file path — see dandi/files/zarr.py around L940 in _upload_zarr_file:

   headers = {"Content-MD5": item.base64_digest}
   ...
   storage_session.put(upload_url, data=fp, ..., headers=headers, ...)

   The aws Haskell library exposes poContentMD5 on PutObject for exactly this. Same idea for S3.uploadPart.

2. Reconstruct and compare the multipart ETag client-side. git-annex fully controls the multipart flow (Remote/S3.hs:396-435) — it can accumulate the per-part MD5s during upload, compute md5(concat(part_md5s)) and append -<partcount>, and compare that against S3.cmurETag returned by postCompleteMultipartUpload. Mismatch → fail the export before cleanupExport runs.

   This is exactly what dandi-cli's DandiETag does — the algorithm lives in dandischema/digests/dandietag.py (DandiETag class at L96, as_str() at L132-140: f"{md5(concat(md5(part_i))).hexdigest()}-{len(parts)}"), and the comparison against the server-returned ETag is in dandi/files/bases.py L373-471, boiling down to:

   etagger = get_dandietag(self.filepath)     # bases.py:374
   filetag = etagger.as_str()                 # bases.py:375
   ...
   final_etag = rxml.findtext(f"{ns}ETag").strip('"')   # bases.py:463-465
   if final_etag != filetag:
       # bail out — server and client disagree on final ETag (bases.py:466-471)

   Per-part sizes are also cross-checked against etagger.get_part(...) before upload (bases.py around L681).

Note that mitigation (a) subsumes (b) at the S3-transit level: if each part's upContentMD5 is validated server-side, the combined ETag mismatch cannot occur (barring post-ack corruption on the S3 side, which is a different threat model). Recommend (a) alone for the S3 code path — it's a smaller, more robust patch.

Library availability check: git-annex.cabal line 296 requires aws >= 0.24.1; stack.yaml currently resolves aws-0.25.2. Both poContentMD5 :: Maybe (Digest MD5) on PutObject and upContentMD5 :: Maybe (Digest MD5) on UploadPart are present at that version, so mitigation (a) needs no library bump — just wiring an MD5 computed alongside the existing stream through httpBodyStorer / handlePopper. The current putObject helper at Remote/S3.hs:1107-1116 explicitly leaves poContentMD5 = Nothing (populates only storage class / metadata / auto-make-bucket / ACL / tagging).

3. The correct semantic fix, longer-term: neither (a) nor (b) validates against the git-annex key hash. They only prove "what S3 stored matches what we sent" — not "what we sent matches the key". A full end-to-end fix would hash the file with the key's own backend (SHA256E, etc.) during the upload stream and refuse to record InfoPresent on a mismatch. This is more invasive but is what the key-hash contract implies.

Even without any of the above, at minimum: if verification isn't structurally possible for a given endpoint/backend, refuse to record InfoPresent after export without an explicit verify-roundtrip (or gate on an option), so users at least have a way to opt into "upload-then-verify" semantics.

Impact

Silent-corruption exposure for any git-annex user relying on an S3 remote as a durable copy of an exported tree. Especially concerning for S3-compatible (non-AWS) endpoints where transport-layer integrity guarantees are weaker than on AWS proper. And — bringing it back to the trigger — fsck --from <s3remote> cannot currently be used to reassure downstream users (e.g. OpenNeuro) that archived files are intact; it only confirms that some object with the right name exists. [CORRECTION: fsck DOES download and hash for exports — it can be used to reassure downstream users that archive files are intact, at the cost of downloading the archive (or the delta since the last incremental-fsck pass). The remaining concern is the download cost for archive-scale exports and the window between upload and the next full-verify.]

Code references (git-annex 10.20260717, master @ ccf99dd)

Upload — Remote/S3.hs:

• L386-395 single-part: S3.porETag resp returned but never validated.
• L396-435 multipart: per-part S3.uprETag and final S3.cmurETag returned but never validated.
• L549-561 storeExportS3: void $ storeExportS3' — ETag and version-id both discarded.

Location log — Command/Export.hs:

-- Command/Export.hs:391-396
cleanupExport r db ek loc sent = do
    liftIO $ addExportedLocation db ek loc
    when (sent && not (isGitShaKey ek)) $
        logChange NoLiveUpdate ek (uuid r) InfoPresent
    return True

Called from performExport (Command/Export.hs:312-330) on any non-exception return from storer.

Presence check — Remote/S3.hs:

-- Remote/S3.hs:542-547
checkKeyHelper' info h o limit = liftIO $ runResourceT $ do
    rsp <- sendS3Handle h req
    extractFromResourceT (isJust $ S3.horMetadata rsp)
  where
    req = limit $ S3.headObject (bucket info) o

-- Remote/S3.hs:585-591
checkPresentExportS3 hv r info k loc = withS3Handle hv $ \case
    Right h -> checkKeyHelper info h (Left (T.pack $ bucketExportLocation info loc))
    ...

Retrieval (verifies content; called by git annex get AND by fsck --from in the non---fast case) — Remote/S3.hs:563-573:

retrieveExportS3 hv r info k loc f p =
    verifyKeyContentIncrementally AlwaysVerify k $ \iv ->
        withS3Handle hv $ \case
            Right h -> retrieveHelper (gitconfig r) info h (Left (T.pack exportloc)) f p iv
            ...

Fsck-from-remote flow (Command/Fsck.hs:163-217) — performRemote first calls Remote.hasKey (HEAD-only for exports), then downloads via Remote.retrieveKeyFile … (RemoteVerify remote) at line 214. For exports, retrieveKeyFile is adjusted at Remote/Helper/ExportImport.hs:239-244 to route through retrieveFromImportOrExport → retrieveExport above. --fast mode skips the download at Command/Fsck.hs:207-210.

Related sites in the same code path

Adjacent places that inherit or extend the same "trust the HTTP response" pattern — worth touching in the same fix:

• storeExportWithContentIdentifierS3 (Remote/S3.hs:796-807) does retain the ETag, but only as an opaque content-identifier — never compared against an expected content hash. A corrupt upload therefore gets a stable-but-meaningless CID; checkPresentExportWithContentIdentifierS3 (Remote/S3.hs:819-825) will then succeed forever on a HEAD that matches that same corrupt ETag via limitHeadToContentIdentifier. Also note the dead-branch at lines 797-799: both versioning info and otherwise branches do the same go — probably an unfinished intention to differentiate versioned vs unversioned verification (versioned case ought to be able to do a version-id round-trip verify).
• Annex-object → export rename shortcut (Command/Export.hs:372-388, tryrenameannexobject): when the annex-object is already known to be present on the remote, git-annex issues a server-side S3 copy (renameExport) instead of re-uploading. It then calls addExportedLocation unconditionally on success (line 382). This inherits any pre-existing corruption at the source key without noticing.
• No fsck / verify hook currently touches the export path (grep -n 'export\|Export' Command/Fsck.hs → nothing; Logs/Export.hs and Logs/Export/Pure.hs manage the export tree log without triggering content verification). So there is currently no post-hoc way for fsck --from <s3remote> to catch a corrupt export short of downloading and re-hashing. [CORRECTION: downloading and re-hashing is exactly what fsck does. The routing goes via retrieveKeyFile (adjusted for exports by Remote/Helper/ExportImport.hs:239-244), not via a fsck-specific export hook.]

Notes / caveats

• With SSE-KMS / SSE-C / CSE, the returned ETag is not the MD5 of the plaintext object, so the client-side ETag reconstruction (mitigation b) needs to account for that. The server-side Content-MD5 header (mitigation a) still works fine — S3 verifies the pre-encryption body.
• InfoPresent is skipped for isGitShaKey ek (Command/Export.hs:394), i.e. for non-annexed git blobs being exported. For annexed keys — the case that matters for durability guarantees — InfoPresent is written unconditionally on any non-exception return of the storer.
• The same reasoning likely applies to other export-capable remotes (Remote/WebDAV.hs, Remote/Rclone.hs, …) — but this issue is scoped to the S3 code path we actually read.

---

Reported after a static code review of the S3 export path (Remote/S3.hs, Command/Export.hs) at master ccf99dd4fe, followed by an independent second-pass verification. Not a live reproducer. Filed with the ai-uncurated label per repo convention; please double-check line numbers / logic before treating as authoritative. Edited 2026-08-01 to correct incorrect claims about fsck --from behaviour after @joeyh's correction; see the retraction comment for details.

[yarikoptic-gitmate]

Turns out there is substantial prior discussion of this exact question in git-annex itself — filing a comment for the record so the design tension is visible on the issue.

Prior discussion (Jan 2026)

Joey has already considered upload-side content hashing for export and decided against it. Same trigger — the OpenNeuro ds005256 incident (OpenNeuroOrg/openneuro#3446). See doc/todo/recover_from_export_of_corrupted_object/comment_8 (2026-01-26), verbatim:

"I thought about making git-annex export checksum files before uploading, but I don't see why export needs that any more than a regular copy to a remote does. In either case, annex.verify will notice the bad content when getting from the remote, and fscking the remote will also detect it, and now, recover from it."

The chosen path was recovery, not prevention — landed in 10.20260213 as "fsck: Support repairing a corrupted file in a versioned S3 remote" (new repairKey Remote method for S3).

The design position

Two design docs speak directly to this:

• doc/design/exporting_trees_to_special_remotes.mdwn:157-162 — exports are always-untrusted, verification is deliberately download-time via annex.verify, enforced in Annex/Verify.hs.
• doc/design/passthrough_proxy.mdwn:715-721, on the Content-MD5 header specifically:

  "S3 can optionally verify the upload of a presigned url by using the Content-MD5 header. […] Added complexity in this particular case does not seem likely to be worthwhile. git-annex does not usually have S3 verify the checksum."

So the poContentMD5 = Nothing at Remote/S3.hs:1107-1116 is a considered omission, not an oversight.

Where I'd push back

Joey's "get / fsck will catch it" argument in comment 8 depends on two things that don't hold cleanly for the OpenNeuro-class case:

1. The user has to actually run get or fsck at some point after the export. For archive-scale S3 uses (OpenNeuro, DANDI, similar), exported files sit for years without a get. In the meantime the false InfoPresent erodes the safety margin — local copies elsewhere may be dropped because git-annex records the S3 remote as holding a valid copy.
2. fsck --from <s3remote> on an exported file has to actually verify content — but as this issue documents, it doesn't. Today it's HEAD-only for the authenticated path (Remote/S3.hs:542-547, 585-587). So the "fsck will detect it" fallback is broken for this exact code path — the fallback needs to be repaired or prevention added.

Comment 8 also treats "the local annex-object was already corrupt" and "the transport truncated the upload" as the same case. They aren't:

• Local-corruption case: annex.verify on retrieval catches it eventually; prevention wouldn't help unless we hash against the key backend during export (a bigger change). Agreed with Joey here.
• Transport-corruption case (the OpenNeuro trigger — "more likely that the upload was somehow not of the whole file", comment 8): S3-side or client-side ETag verification would catch it at the moment of upload, before InfoPresent is logged. This is the case that Content-MD5 / dandi-style ETag comparison actually addresses.

A hybrid design that respects the prior position

A tri-state config gate, explicit about what the mechanism actually does — "provide MD5 to S3 so it validates via Content-MD5":

annex.s3-provide-md5 = no | key-only | always     (default: no)

• no (default): current behaviour, no Content-MD5 header, no verification. Nothing changes for existing users.
• key-only: when the git-annex key already carries an MD5 (MD5 / MD5E backends), extract it from the key (Backend/Hash.hs:187-188: keyHash = fst . splitKeyNameExtension) and set poContentMD5 on the request. For other backends, skip. Zero I/O, zero CPU — the MD5 is right there in the key. Covers DataLad's default backend at literal zero cost.
• always: for keys without a free MD5 (SHA256E — used by OpenNeuro's datasets, see e.g. OpenNeuroDatasets/ds004504 .gitattributes — plus SHA1, WORM, URL, etc.), pre-hash the file (or part range for multipart), then upload with poContentMD5 / upContentMD5 set. Costs one extra read per part; S3 rejects with BadDigest pre-commit on mismatch. Archive-scale users opt into this explicitly.

The name annex.s3-provide-md5 is S3-specific because the mechanism is S3-specific — Content-MD5 is what S3 accepts for server-side pre-commit validation. Analogous knobs would exist per-remote-type if we ever want the same for WebDAV, GCS, etc.

This addresses the specific "any more than a regular copy" objection: for MD5/MD5E under key-only it's not "any more" — it's literally free. And for always the user has explicitly asked for the extra cost.

Incremental fsck note

For completeness — git annex fsck --from s3remote --incremental / --more / --incremental-schedule=1d (see Command/Fsck.hs:749-770 + Database/Fsck.hs) already provides the memoisation for large-archive re-fsck. But — the fsck DB lives at .git/annex/fsck/<uuid>/fsckdb/ per Annex/Locations.hs:431-451, i.e. strictly local, not on the git-annex branch. That means if a large-archive user runs their periodic fsck from an ephemeral clone (fresh checkout per CI job), each fsck starts from scratch and re-verifies the entire export — which effectively rules out "just make fsck actually verify content" as a general fallback for this class of user.

Also: the fsck DB is keyed by git-annex key, so re-export of the same key with different S3-side content would still show as "verified" from a prior pass. Distinct issue, worth its own todo.

@effigies — does OpenNeuro's setup have a persistent local git-annex clone for the fsck workflow, or is it re-cloned per run? Answer changes the shape of a follow-up fix.

Suggested cross-references

• doc/todo/recover_from_export_of_corrupted_object (parent context)
• doc/design/exporting_trees_to_special_remotes.mdwn:157-162 (always-untrusted design)
• doc/design/passthrough_proxy.mdwn:715-721 (Content-MD5 explicitly declined for presigned-URL case)
• Commit fcc052bed8 (When proxying an upload to a special remote, verify the hash. — Jul 2024) — precedent for upload-side hash verification in the proxy path, i.e. the pattern already exists in git-annex where the trust boundary matters.
• doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn (real-world minio-console 200-OK-with-no-content failure)

With @yarikoptic we might iterate on the config-knob shape (annex.s3-provide-md5 vs annex.s3.provide-md5, tri-state vs boolean-with-implicit-key-only, etc.) and to prepare a .patch file under patches/ per the contributing pattern if the general direction is acceptable.

[joeyh]

@effigies note that I do not usually engage with LLM generated text as posted here. If you have a real issue, I suggest filing a bug report on the git-annex website.

FWIW, git-annex fsck does not operate as described above. It does download the entire object from S3, and hash it. This happens whether or not the S3 remote uses exporttree=yes.

[yarikoptic-gitmate]

@joeyh — thank you for the correction. You're right, and the report indeed has a real error.

Retraction

I traced Command/Fsck.hs:163-217 end-to-end after your comment: performRemote calls Remote.hasKey first (which for exports lands in checkPresentExport — HEAD-only for S3), and then, if present, calls Remote.retrieveKeyFile … (RemoteVerify remote) at line 214. For export-configured remotes, retrieveKeyFile is adjusted at Remote/Helper/ExportImport.hs:239-244 to route through retrieveFromImportOrExport → retrieveExport → for S3, Remote/S3.hs:563-573 which uses verifyKeyContentIncrementally AlwaysVerify k. So fsck does download and hash — including for exporttree=yes. My mistake was collapsing checkPresent (which is HEAD-only for exports) with the full fsck flow (which is not).

The specifically wrong claims made before:

• Original issue body, "Realistic failure dataloss path" step 4 — asserted fsck would say "OK" on a corrupt object. Wrong; fsck would detect the hash mismatch on download. The wider dataloss framing still holds via the narrower path in points 1-3 below, but this step's specific mechanism doesn't apply.
• Original issue body, "Impact" — asserted fsck --from <s3remote> "only confirms that some object with the right name exists". Wrong; it downloads and verifies.
• Follow-up comment, "Where I'd push back" point 2 — asserted the "fsck will detect it" fallback is "broken for this exact code path". Wrong; the fallback works as you designed it.

I'll edit the issue body to strike-through those specific sentences with inline corrections so future readers don't get the wrong picture.

Historical check — no version of git-annex ever supported the incorrect claim

For the record: git log -S on the code paths involved shows fsck --from has been "download + verify content" since it was introduced.

• 90319afa41 (2012-01-19, "fsck --from"), from @joeyh's own commit message: "Fscking a remote is now supported. It's done by retrieving the contents of the specified files from the remote, and checking them, so can be an expensive operation."
• ccc0684d21 (2019-02-20) introduced Remote/Helper/ExportImport.hs; from that point retrieveKeyFile for export-configured remotes routes through retrieveFromImportOrExport.
• 2f2701137d (2022-05-09, "incremental verification for retrieval from all export remotes") wired AlwaysVerify explicitly into retrieveExportS3. This added streaming/incremental verification on top of the already-present download-and-hash. Sponsored-by trailer: Dartmouth College's Datalad project — so Datalad literally paid for the verification path I incorrectly said didn't exist.
• ee076b68f5 (2024-08-04) added strong verification for the annexobjects location on top of that.

So the mistake wasn't from misreading an older version of the codebase — it was a straightforward reading error on the current code path. Noting that here so the report's confidence level is reset appropriately.

What's still true, restated more narrowly

Setting aside the fsck errors, three narrower observations remain:

1. InfoPresent is written unconditionally after upload (Command/Export.hs:391-396). Between an upload and the next fsck --from, whereis/numcopies accounting treats the S3 copy as valid. If the user drops other copies based on that log before the fsck runs, and fsck then detects corruption, the safety margin has already been eroded.
2. fsck --from <remote> --fast (Command/Fsck.hs:207-210) skips the download and only does the HEAD check. Under --fast, corruption is not caught.
3. Cost of full fsck on archive-scale exports: for a multi-TB S3 archive, fsck --from downloads the entire archive to hash it. Incremental fsck reduces re-verification cost on subsequent passes, but new keys still require full download. Content-MD5 at upload time short-circuits this by having S3 reject corruption pre-commit — cheaper by orders of magnitude for the "assure the archive is intact" workflow.

These are legitimately weaker arguments than what the original report made. It's not "corruption goes undetected", it's "detection is deferred and can be expensive" and thus not necessarily done.