server: bound in-flight CAS writes and per-stream gRPC buffering

[shreyas-blacksmith]

Summary

One tenant's Bazel upload burst took bazel-l1-us-west-9 down without any OOM kill: 7,999 ByteStream Writes in flight, heap 54 GB vs GOMEMLIMIT=40GiB, GC pinned the CPU, health polls timed out, 130k sockets in CLOSE-WAIT. The L1 had no admission control on Writes, and grpc-go's BDP estimator grows each stream's flow-control window to 16 MB, so every in-flight write parked up to 16 MB of client bytes in the transport ahead of the disk-bound Put. Memory was in-flight writes × 16 MB, with nothing bounding either term.

Two env-var knobs, both off by default (unset = today's behaviour), following the fork's BAZEL_REMOTE_* convention:

BAZEL_REMOTE_GRPC_INITIAL_WINDOW_SIZE=<bytes> (main.go) — static per-stream window (grpc.InitialWindowSize, conn window 16×). Bounds the bytes a client can push before the server has answered the first frame, for admitted and shed writes.

BAZEL_REMOTE_MAX_INFLIGHT_CAS_WRITES=<n> (server.WithMaxInflightCASWrites) — caps Writes that reach the Put path. The box is CPU-bound on zstd+sha256 at a few thousand new blobs/min anyway; the cap turns "park everything above that in memory until the process stalls" into "reject it at the door".

Write(stream):
  recv first frame, parse resource name
  if cache.Contains(digest):            # unchanged early return
      SendAndClose(committed_size)
  set up zstd decoder / pipe
  if !tryAcquireCASWriteSlot():         # new
      log "GRPC BYTESTREAM WRITE SHED"
      return FAILED_PRECONDITION        # before Put
  go Put(...); defer releaseCASWriteSlot()

Shed code is FAILED_PRECONDITION, deliberately not a connection-class code: Bazel treats UNAVAILABLE as transient and walks the full --remote_retries backoff ladder per shed blob, which under sustained shedding on the canary showed up as a minutes-long upload tail after the build summary. With a permanent code the client logs the upload failure once and moves on. The shim passes application codes through unchanged (degradeError only rewrites connection-class codes). A shed write is never acked, so the action result is never published and nothing references the unstored blob; the cost is one future cache miss, not a failed or retried build. (An earlier experiment that acked shed writes as successful was reverted: Bazel then believes the blob is cached and a later --remote_download_minimal fetch of it fails, which is the whole-build retry we are trying to avoid.)

Gate metrics: bazel_remote_cas_write_slots_inflight (admitted Puts right now) and bazel_remote_cas_write_shed_total. started − handled on Write is only an upper bound (it includes existing-blob early returns and sheds in flight).

Reads, AC, FindMissingBlobs and the existing-blob path are untouched. The slot is acquired after decoder setup so decoder-pool/reset error paths cannot leak it (TestGrpcByteStreamWriteShedWhenInflightCapReached).

Canary (us-west-9, 16c/60G) under the live burst, 10–16 Gbps in:

build	cap	window	RSS	heap	notes
-blacksmith.11	–	BDP (16 MB)	43–45 GB, swap full	54 GB	stalled, 130k CLOSE-WAIT
cap only	2000	BDP	38 GB	36 GB	~18 MB per admitted write
cap + window	500	256 KB	11–14 GB	9–15 GB	cap is the bottleneck at 13 Gbps, sheds retried
cap + window	4000	256 KB	36 GB	33 GB	~8 MB per admitted write at full load, too close to 40 GiB
cap + window, FAILED_PRECONDITION	2500	256 KB	≤30 GB	≤28 GB	current

Staging load test (bazel-l1-staging-1 pinned to 16 CPUs / GOMEMLIMIT=40GiB / MemoryHigh=44G, synthetic 4,000-stream unique-blob writer at 13–15 Gbps for 5 min, gate pinned at 2,500 throughout): heap 21–28 GB, RSS flat 30 GB, 0 CLOSE-WAIT, /status 30–450 ms, no stall. Memory is bounded by the cap, not by offered load.

Risks

Verdict: Ship. Both knobs are opt-in; the deployed combination held the canary through the same burst that took it down and a 5-min saturated staging run with ~10 GB of heap headroom.
Worry about: per-stream throughput is window / RTT. 256 KB is fine intra-DC (~1 GB/s per stream); a cross-region client would see slower single-stream uploads. Pick the window per deployment, not globally. Under sustained shedding the tenant sees one Remote Cache: Error while uploading artifact warning per shed blob and loses those future hits; the fix for that is capacity (spread the hot tenant prefix across members), not this knob.
Checked, not worried: existing-blob early return still precedes the gate; a shed never acks, so no AC entry can point at an unstored blob; slot cannot leak on decoder errors. go test -race ./... and golangci-lint green in CI.
If it goes wrong: sheds show as GRPC BYTESTREAM WRITE SHED in the access log, bazel_remote_cas_write_shed_total, and grpc_server_handled_total{grpc_method="Write",grpc_code="FailedPrecondition"}; a stall shows as bazel_remote_cas_write_slots_inflight pinned at the cap with heap near GOMEMLIMIT; rollback is removing the two env vars and restarting, no data migration.

Link to Devin session: https://app.devin.ai/sessions/29afc24d202a4deab5cfbb2c46c87104
Open in Devin Desktop: https://app.devin.ai/desktop/session/29afc24d202a4deab5cfbb2c46c87104?variant=devin
Requested by: @shreyas-blacksmith

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[cursor]

Stale Bugbot comment from a previous run.

[shreyas-blacksmith]

Live L1-9 follow-up through 00:58 UTC: the 00:46 ramp still produced 77,110 dropped Serval upload operations out of 172,045 in one minute. Customer logs confirm explicit build retries; UNAVAILABLE is not equivalent to a harmless successful upload. Warned matched 32-vCPU shard job medians are 255/291/255 seconds versus 131/144/144 seconds without warnings, with commit/cache-warmth confounding. Retrying VMs contacted both L1-9 and L1-10, so this does not isolate gate causality.

L1-9 restarted again at 00:48:45. Since then, heap is about 5.5-10.6 GiB and no new Write UNAVAILABLE series appears after the restart. Offered load is lower than the 00:46 burst, so this is not yet a saturation validation. All six previously unfinished canary-only jobs now succeeded in 131-183 seconds with no cache warnings or build retries.

Please do not turn CAS shedding into bare successful ACKs: UpdateActionResult publishes metadata without checking referenced blobs (server/grpc_ac.go:245), while dependency-checked reads guard only AC retrieval. Successful CAS acknowledgements can leave client-held digest state referring to absent objects; GetValidatedActionResult is not a universal guard against lost inputs. Keep bounded transport buffering and tune bounded, context-cancellable admission under a measured burst. If best-effort blackholing is deliberately pursued, it needs an explicit policy and end-to-end reused-digest coverage, not just a SendAndClose substitution.

I am continuing short-interval checks of completed job/step times, forced misses, shedding, and upload tails. No fleet settings changed by this investigation.

[devin-ai-integration]

Re #53 (comment) — agreed, and reverted: 53862ab (ack-at-gate) is undone by fb5e3f3, so the branch is back to the bounded admission that returns UNAVAILABLE (cb62105). The ack build was never started on the canary (binary restored on disk before any restart; 0 WRITE ABSORBED). Beyond the UpdateActionResult gap you point out, a successful Write also populates Bazel's client-side "already uploaded" digest set, which under --remote_download_minimal turns an absent blob into a lost-input build retry — so blackholing would create the exact failure mode being chased. Any best-effort drop stays an explicit, separately reviewed policy, not a SendAndClose substitution.

[shreyas-blacksmith]

Independent reconciliation of the customer-impact investigation:

The post-summary upload tail is real in the audited svflow-core job https://github.com/ServalHQ/mono/actions/runs/37868341343/job/113620403000. GitHub reports 934s total. Bazel Build reports successful completion at 01:11:47 UTC, then emits cache-upload errors until its runner step ends at 01:17:44 (358s later). Test has another 337s after its successful summary, also with upload errors. Those tails total 695s. The next commands and artifact-reporting steps take seconds, not minutes.

Corrections to my broader claims: comparing this with a 99s cache-warm same-shard job on a different commit is not a causal 9x mitigation penalty. The slow job executed tests; the clean job reused them. The audited slow VM also contacts L1-15, although its recorded error/dropped uploads are on L1-9; all-job L1-9-only attribution was too strong. All-RPC started-minus-handled counts do not measure admitted Put concurrency, so they cannot establish the live cap. L1-9 has 16 observed logical CPUs, not the 32-vCPU customer runner tier I conflated with it. High CPU does not prove admission tuning or compression changes cannot help.

The static-window memory improvement and later customer upload waits can both be true. The initial 00:35 canary result and later 01:03 burst are different windows. Your 01:00 acknowledgment that bare ACK-and-drop is unsafe agrees with the investigation; I verified the revert on the live PR head. I have not changed fleet configuration.

[devin-ai-integration]

Agree with the reconciliation, and the post-summary tail has a concrete mechanism that this PR's current shed code makes worse than it needs to be:

• Bazel ≥7 runs output uploads in the background (remote_cache_async, OutputUploadTask) and waits for them at the end of the invocation — that is the "build finished, then minutes of Remote Cache: errors" tail in the audited job.
• RemoteRetrier.EXPERIMENTAL_GRPC_RESULT_CLASSIFIER treats UNAVAILABLE (and RESOURCE_EXHAUSTED, DEADLINE_EXCEEDED, INTERNAL, ABORTED, UNKNOWN) as TRANSIENT_FAILURE, so each shed Write gets the full --remote_retries ladder (5 attempts, ~3 s of backoff) before reportUploadError logs it as a warning. With upload concurrency bounded on the client and thousands of outputs per heavy shard, that ladder × blobs is the ~6-minute tail.
• Anything outside that set is PERMANENT_FAILURE: the upload is reported once and the invocation moves on; the action result is still not published (same as today after retries are exhausted), so nothing references a blob that isn't stored.

Proposed follow-up on this branch, pending go-ahead: return FAILED_PRECONDITION from the gate instead of UNAVAILABLE (the shim only rewrites UNAVAILABLE/DEADLINE_EXCEEDED, so it passes through unchanged), and lower the canary cap to 2500 — at the 01:20 crest 4,000 admitted writes cost ~8 MB each (heap 33 GB vs GOMEMLIMIT 40 GiB), so 4000 is too close to the stall. Agreed that started − handled on Write is an upper bound on admitted Puts (it includes existing-blob early returns and sheds in flight), not the admitted count; I'll add a gauge for the gate itself.

The ingest ceiling (~4k new blobs/min on a 16-core member, CPU-bound on zstd) is unchanged by either; the gate converts the overload into shed uploads instead of a stalled member, and spreading the hot tenant prefix across more members is what moves the ceiling.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cd28524. Configure here.}