Summary
Two observability gaps: nothing measures what a sandbox actually consumes (CPU seconds, RSS) — so no noisy-neighbor detection, no usage-based billing beyond wall-clock seconds, no overcommit data — and claim latency is exported only as counter totals (averages), hiding tails. Three proposals, independently landable.
Current state
/metrics (hand-rendered, server/metrics.go): pool/claim gauges, lifecycle counters, *_seconds_total for average latency only.- The usage journal bills wall-clock seconds by lifecycle interval folding; it never sees actual resource consumption.
- Per-claim VMM pids are known to cocoon; nothing reads their
/proc stats.
- No trace correlation between an SDK call and the node's logs/journal.
Proposal
1. Per-sandbox resource stats
Source: the VMM process (/proc/<pid>: utime+stime, VmRSS) — works for CH and FC alike, no guest cooperation, no cgroup dependency.
GET /v1/sandboxes/{id}/stats (sandbox token) and GET /v1/stats (root, all claims) — sampled on demand, {cpu_seconds, rss_bytes, state}.- Per-tenant aggregates on
/metrics (sandboxd_tenant_cpu_seconds_total, sandboxd_tenant_rss_bytes), sampled by the existing telemetry tick. Deliberately NO per-sandbox Prometheus series — unbounded cardinality; the operator index + stats endpoint covers spot checks.
- Billing:
release/reap/hibernate usage events gain cpu_seconds/max_rss — read once at teardown, off the claim path. Cumulative across hibernate cycles: accumulate into the claim record at each hibernate (the VMM dies there; the counter must survive it). A reap whose VMM is already gone records the last accumulated value — honest, not fabricated.
2. Claim-path histograms
Replace averages-only with fixed-bucket histograms per tier (warm/clone/cold) and per phase (queue → provision → probe), hand-rendered like the rest of metrics.go. Buckets chosen from the numbers of record (0.5ms/5/50/100/250/500/1000). The phase timestamps largely exist for the *_seconds_total counters; this changes the export shape, not the instrumentation.
3. Trace propagation (cheap, no OTel dependency)
Accept traceparent on the resource-creating endpoints; echo the trace id into the structured claim log line and the usage event. Correlation happens in the operator's collector. Full OTLP export is rejected for now: an SDK dependency + config surface for something a log pipeline already does; revisit when someone actually runs a tracing backend against this.
Hot-path cost
Claim path: zero new syscalls (phase timestamps already taken; histogram observe is arithmetic). Stats: on-demand reads + one /proc read per claim per telemetry tick for tenant aggregates (bounded by live claims, off the data plane). Teardown: one /proc read before destroy. State the measured delta in the PR per the perf discipline.
Acceptance evidence
- Unit: stat parsing, hibernate-cycle accumulation, reap-after-death fallback.
- e2e: spin CPU in a sandbox →
stats shows monotonic cpu_seconds; release event carries plausible cpu_seconds; /metrics shows tenant gauges and tier histograms; rpcbench p50/p99 unchanged vs baseline in the same run.
Estimated size: ~200-250 lines + tests; proposals land separately (1 is the substance, 2 and 3 are small).
Summary
Two observability gaps: nothing measures what a sandbox actually consumes (CPU seconds, RSS) — so no noisy-neighbor detection, no usage-based billing beyond wall-clock seconds, no overcommit data — and claim latency is exported only as counter totals (averages), hiding tails. Three proposals, independently landable.
Current state
/metrics(hand-rendered,server/metrics.go): pool/claim gauges, lifecycle counters,*_seconds_totalfor average latency only./procstats.Proposal
1. Per-sandbox resource stats
Source: the VMM process (
/proc/<pid>: utime+stime, VmRSS) — works for CH and FC alike, no guest cooperation, no cgroup dependency.GET /v1/sandboxes/{id}/stats(sandbox token) andGET /v1/stats(root, all claims) — sampled on demand,{cpu_seconds, rss_bytes, state}./metrics(sandboxd_tenant_cpu_seconds_total,sandboxd_tenant_rss_bytes), sampled by the existing telemetry tick. Deliberately NO per-sandbox Prometheus series — unbounded cardinality; the operator index + stats endpoint covers spot checks.release/reap/hibernateusage events gaincpu_seconds/max_rss— read once at teardown, off the claim path. Cumulative across hibernate cycles: accumulate into the claim record at each hibernate (the VMM dies there; the counter must survive it). A reap whose VMM is already gone records the last accumulated value — honest, not fabricated.2. Claim-path histograms
Replace averages-only with fixed-bucket histograms per tier (warm/clone/cold) and per phase (queue → provision → probe), hand-rendered like the rest of metrics.go. Buckets chosen from the numbers of record (0.5ms/5/50/100/250/500/1000). The phase timestamps largely exist for the
*_seconds_totalcounters; this changes the export shape, not the instrumentation.3. Trace propagation (cheap, no OTel dependency)
Accept
traceparenton the resource-creating endpoints; echo the trace id into the structured claim log line and the usage event. Correlation happens in the operator's collector. Full OTLP export is rejected for now: an SDK dependency + config surface for something a log pipeline already does; revisit when someone actually runs a tracing backend against this.Hot-path cost
Claim path: zero new syscalls (phase timestamps already taken; histogram observe is arithmetic). Stats: on-demand reads + one
/procread per claim per telemetry tick for tenant aggregates (bounded by live claims, off the data plane). Teardown: one/procread before destroy. State the measured delta in the PR per the perf discipline.Acceptance evidence
statsshows monotonic cpu_seconds; release event carries plausiblecpu_seconds;/metricsshows tenant gauges and tier histograms;rpcbenchp50/p99 unchanged vs baseline in the same run.Estimated size: ~200-250 lines + tests; proposals land separately (1 is the substance, 2 and 3 are small).