obs(api): add expiration index size and evictor sweep duration metrics - #3606
Open
AdaAibaby wants to merge 1 commit into
Open
obs(api): add expiration index size and evictor sweep duration metrics#3606AdaAibaby wants to merge 1 commit into
AdaAibaby wants to merge 1 commit into
Conversation
Closes e2b-dev#3605. Two new histograms: api.redis_storage.expiration_index.size ({entry}) ZCARD of sandbox:storage:global:expiration sampled once per heal pass (every 5 min). Use max/p99 to detect unbounded growth caused by stale ZSET entries that are not being swept. api.redis_storage.expiration_index.sweep_duration (ms) Wall-clock duration of each ExpiredItems call (ZRANGEBYSCORE + MGET pipeline). Rising p99 alongside rising index size confirms the O(log N + K) cost growth described in e2b-dev#3605. Collection points: - heal.go: ZCARD after each healExpirationIndex pass — one extra Redis round-trip per 5 min, negligible next to forEachSandboxBatch. - items.go: defer-based timer wrapping the full ExpiredItems body, recorded on every evictor tick.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 24, 2026 09:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #3605.
Adds two histograms to make the global expiration ZSET observable in production:
api.redis_storage.expiration_index.size{entry}healExpirationIndex, every 5 min)api.redis_storage.expiration_index.sweep_durationmsExpiredItemscall (every evictor tick)Why these two signals
sandbox:storage:global:expirationis a singleton ZSET shared by every sandbox.With 22,458 members (~3 MB) observed in production on 2026-08-14 and the evictor
calling
ExpiredItemsatpollInterval = 50 ms, this key is read 20 × N timesper second (N = API allocation count) — yet neither its size trend nor the
per-sweep cost was visible in any metric.
expiration_index.size— use max/p99 to detect unbounded growth beforeit causes latency spikes; use as a before/after signal when fix(api): prune stale team index entries when ZSET orphans are swept #3567 (orphan SREM
fix) or future sharding work lands.
expiration_index.sweep_duration—ZRANGEBYSCOREis O(log N + K);rising p99 alongside rising size confirms the cost is growing and justifies
remediation.
Implementation
heal.go— oneZCARDcall appended afterhealExpirationIndexcompletes.Cost: one Redis round-trip per 5 min, negligible next to the
forEachSandboxBatchSSCAN scan that precedes it. Sampling here avoids adding any load to the 50 ms
evictor hot path.
items.go—defer-based timer wrapping the fullExpiredItemsbody.Records on every evictor tick, capturing the end-to-end cost of
ZRANGEBYSCORE + MGET pipeline + orphan cleanup.Related
cjson.decodehot path (separate Redis bottleneck)/cc @jakubno @dobrac @ValentaTomas