disk: store client-compressed cas uploads as received, verify digests in the background

[shreyas-blacksmith]

Summary

Stacked on #53 (base is that branch). Off by default.

Today a client-compressed upload costs the L1 more CPU than a raw one. The gRPC layer decodes the client's zstd stream, then casblob.WriteAndClose hashes the raw bytes and re-encodes them into 1 MiB chunks. So turning on --remote_cache_compression on the client makes write storms worse, not better.

CAS entries are already namespaced per storage prefix (FileLocationForStoragePrefixID). A blob that does not match its digest can only hurt the tenant that uploaded it. That makes inline verification optional, so this PR moves it off the request path.

With BAZEL_REMOTE_VERBATIM_ZSTD_UPLOADS=<n> set (n = verifier workers):

flowchart LR
  subgraph before [Before]
    direction TB
    a1[Write compressed-blobs/zstd] --> a2[Go zstd decode] --> a3[sha256] --> a4[zstd encode 1 MiB chunks] --> a5[disk + proxy]
  end
  subgraph after [After]
    direction TB
    b1[Write compressed-blobs/zstd] --> b2[frame to disk as one chunk] --> b3[commit, reply OK]
    b3 -.-> b4[background: decode once, check size + sha256]
    b4 -->|ok| b5[proxy write-through]
    b4 -->|mismatch| b6[evict entry]
  end
  before ~~~ after

Loading

Flow in code:

• server.(*grpcServer).putUpload — compressed uploads call cache.PutZstd first. It returns disk.ErrVerbatimUnavailable before reading any bytes when the feature is off, the blob is larger than a uint32 chunk, or more than 65536 blobs are already waiting for verification. Then the existing decode + Put path runs unchanged. Identity uploads are untouched.
• disk.(*diskCache).PutZstd → put(..., verbatim) → casblob.WriteZstdFrameAndClose: writes the normal casblob header with chunkSize = size and one chunk, copies the client's stream into it, fills the offset table. No decode, hash, or encode. The copy stops at MaxCompressedSize(size) (zstd's compress bound), so a stream longer than a size-byte blob can legitimately compress to is rejected before commit.
• After commit, the job (key, hash, size, random, detached ctx) goes on the verifier queue. verifyVerbatim reopens the file and decodes it once through GetUncheckedReadCloser (no size cap, so a frame that expands past the declared size is seen), checks byte count and sha256. On any failure it RemoveKeys the entry if the random still matches (so a later re-upload is not evicted). On success it does the proxy Put that put skipped, using context.WithoutCancel(ctx) so the storage prefix and S3 backend selection survive the request.
• Reads: the file is a valid single-chunk casblob. A compressed Read at offset 0 streams the stored frame as-is (zero decode, the point of the change), so until the verifier has run it can carry whatever the client sent; the client checks the digest. Identity reads and ranged reads of any single-chunk blob (header.singleChunk(), i.e. two offsets in the table) decode through skipDecoded and are capped at the header's uncompressedSize, so they never emit more than the declared size and no longer DecodeAll the whole blob into memory for a ranged read.

Metrics: bazel_remote_verbatim_zstd_puts_total, ..._fallback_total{reason=disabled|size|backlog}, ..._verify_total{result=ok|hash_mismatch|size_mismatch|decode_error|read_error|gone}, ..._verify_pending.

Not covered: BatchUpdateBlobs with Compressor_ZSTD still decodes inline (grpc_cas.go); Bazel uses ByteStream for anything over its batch limit, which is what matters for the write storms.

Risks

Verdict: Data pending — staging write storm with --remote_cache_compression on the client. Unit tests cover round trip (incl. exactly 1 MiB), ranged reads, unknown-expected-size reads, mismatch/oversize eviction and the no-consume fallback; no load measurement yet.
Worry about: between commit and verification a wrong blob is servable to its own tenant: a compressed Read at offset 0 returns the frame the client sent, uncapped; identity and ranged reads are capped at the declared size. Bazel checks digests on download so the symptom is a download error for that tenant on that blob until the verifier evicts it (seconds to minutes, visible as verify_pending). Only applies when a client sends bad data.
Checked, not worried: cross-tenant poisoning (CAS keyed under the storage prefix); existing on-disk blobs (format unchanged, only chunkSize differs for new blobs, readers already handle arbitrary chunk sizes); disk over-consumption (compressed body bounded by MaxCompressedSize before commit); unbounded unverified data (backlog cap falls back to inline verification); proxy receiving unverified blobs (write-through deferred until ok); feature off (env var unset → every compressed upload takes the old path, tests pass with fakes returning ErrVerbatimUnavailable).
If it goes wrong: rising verify_total{result!="ok"} or tenant download errors; unset BAZEL_REMOTE_VERBATIM_ZSTD_UPLOADS and restart, already-stored single-chunk blobs remain readable by the old binary.

Link to Devin session: https://app.devin.ai/sessions/29afc24d202a4deab5cfbb2c46c87104
Open in Devin Desktop: https://app.devin.ai/desktop/session/29afc24d202a4deab5cfbb2c46c87104?variant=devin
Requested by: @shreyas-blacksmith

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

✅ Compressed upload stored as received, verified in background

After still stores the client's zstd frame byte-for-byte as one chunk, reads back intact both ways, and the verifier records ok; before re-encodes into eight 1 MiB chunks.

_Merge-base

_{This PR b707af6f}

8 of 9 claims checked

	Claim	What Proof saw
	Compressed upload stored as received, verified in background	After still stores the client's zstd frame byte-for-byte as one chunk, reads back intact both ways, and the verifier records ok; before re-encodes into eight 1 MiB chunks.
	Server CPU for compressed uploads drops On an earlier commit	Server CPU for the same 200 compressed uploads drops about 3x, background verification included, and every blob verifies ok.
	Wrong-digest blob evicted after verification	After accepts the wrong-digest blob, serves it once, then the verifier evicts it as hash_mismatch; before rejects it inline.
	Ranged reads of a single large chunk	Every identity and zstd ranged read returned the right bytes, for verbatim single-chunk blobs, regular single-chunk blobs that now take the streaming branch, and multi-chunk blobs.
	Feature off keeps the old path On an earlier commit	With the env var unset, compressed uploads are re-encoded into 1 MiB chunks and wrong digests are rejected inline, as before. Uncompressed storage mode also falls back, and a non-numeric value stops startup.
	Oversized compressed upload rejected On an earlier commit	A ~1 MiB frame declared as 4096 bytes is rejected with the 4256-byte bound error, leaving no file and a NotFound read; a 4110-byte incompressible frame for 4096 bytes is still accepted and verifies ok.
	Unverified reads capped at declared size	Identity reads and ranged zstd reads are capped at the declared size before verification, but a zstd read at offset 0 still returns the full oversized frame; the verifier later evicts it as size_mismatch.
	Frame with trailing data fails verification	After accepts the oversized frame, then the verifier evicts it as size_mismatch and both reads return NotFound; before rejects it inline.
	Exactly-1 MiB blob verifies ok	All three blobs around 1 MiB, including exactly 1048576 bytes, verify ok, stay on disk as one chunk and read back intact.

Warning

2 things look different from the PR description

• Unverified reads capped at declared size: Identity reads and ranged zstd reads are capped at the declared size before verification, but a zstd read at offset 0 still returns the full oversized frame; the verifier later evicts it as size_mismatch.
• Zstd offset-0 reads still uncapped (not in the plan): Before verification, identity reads and ranged zstd reads are capped at the declared size, but a zstd read at offset 0 still streams the stored frame, which decodes to 4194304 bytes for a declared 4194000.

_{Proof on b707af6f (from bb4b1ea9)}

View full Proof report

---

^{Need help on this PR? Tag @codesmith-bot with what you need. (Staging)}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bb4b1ea. Configure here.}