feat(bay): chart GitHub throttle history

[brokemac79]

Summary

• replace Bay's ambiguous cumulative GitHub-throttle sentence with a compact, closed-bucket time series
• show actual observed GitHub wire responses split by 403/429 for 6h, 24h, and 7d, with peak, recent-hour comparison, and closed physical-pool-class totals
• reuse the existing bounded v2 SQLite rollups; no parallel telemetry store, raw identity, or production mutation is introduced

Problem and data semantics

The old Bay sentence summed legacy publication.github_request_metrics.counters keys containing :throttle:. That is a cumulative Durable Object/SQLite total with no public start timestamp or reset boundary. Receipt replay is deduplicated, but distinct retries, revisions, and pagination pages are distinct observations, so the number can grow through repeated real wire attempts. It does not represent a time window and must not be graphed as a rate.

The replacement projection counts only complete, attempted wire_attempt rollup rows with outcome=throttle and exact status 403 or 429. It excludes invocations, circuit skips, pre-wire deferrals, and avoided requests. Each response carries a closed pool_class; no pool identity, repository/item/workflow identity, URL, query, token, or request body is exposed.

The store already retains five-minute rollups for 7 days and hourly rollups for 30 days. throttle_series re-aggregates those rows into at most 1,344 entries (168 hourly buckets × four closed pool classes × two statuses). The current bucket is excluded. Missing/pre-instrumentation, retention, cap truncation, and incomplete attribution fail closed through first_available_bucket_start, coverage_complete, rows_truncated, excluded_incomplete_count, and complete; the UI never invents zero history for an incomplete series. Coverage requires a retained bucket before the requested lower-bound bucket, so telemetry beginning partway through the boundary bucket remains explicitly incomplete.

Raw sanitized rate-limit-header detail remains retained for 24 hours. A 7-day response therefore discloses that boundary and can have a complete throttle series while the overall raw-detail query is incomplete.

Implementation

• extend the existing observability endpoint to accept hours=168
• add the bounded throttle_series projection from existing rollups, with explicit completeness and first-available boundaries
• render a dedicated small GitHub-throttle card in Bay, compatible with legacy cached v2 responses
• cache and switch matching 6h/24h/7d requests, with 403/429 paths, point tooltips, pool-class totals, peak, recent/prior-hour comparison, and honest empty/incomplete states
• update the existing deterministic Bay browser proof to exercise all three ranges and the 24-hour header-detail boundary
• classify proof-network traffic from the parsed exact api.github.com hostname, with lookalike-host regression coverage

Validation

• pnpm run build:all
• node --test test/openclaw-bay-proof-network.test.ts test/github-egress-telemetry.test.ts test/dashboard-public-observability.test.ts test/dashboard-worker-observability.test.ts test/dashboard-worker-bay-records-routes.test.ts — 90 passed, 0 failed
• pnpm run check:docs
• pnpm run check:dashboard-strict
• git diff --check origin/main...HEAD
• dirty Codex reviews: clean after fixing the 7d raw-detail boundary, incomplete-response zero filling, range-switch race, tooltip attribution, privacy-rebase State writer regression, formatter finding, and partial first-bucket completeness finding
• hosted CodeQL identified an incomplete URL-prefix comparison in the proof harness; the accepted fix uses the parsed exact hostname and adds lookalike-domain coverage. Dirty Codex re-review was clean and the focused gate was rerun before commit.
• committed Codex review against origin/main 61276c87cbb73cfec39e9649cee6862cf84a935f: clean at exact head efbe0ac96a431889d4dcdf4a4028cd3aafd3ff8a; no actionable correctness findings. The final loop accepted and fixed two late findings: any incomplete closed egress observation now fails the throttle series closed, and reset-header detail is accurately labelled as retained 403/429 observations rather than implying that every recorded 403 was classified as a throttle.
• local ClawSweeper committed-range review: complete at base 61276c87cbb73cfec39e9649cee6862cf84a935f / head efbe0ac96a431889d4dcdf4a4028cd3aafd3ff8a; decision keep_open, confidence high, no actionable code finding. Its offline-only proof concern is resolved by the exact-head browser receipt below.

Native Windows pnpm run format:check reports the checkout-wide CRLF mismatch, so no line-ending or platform-policy change is included. The authoritative clean Linux gate passed at exact head: pnpm install --frozen-lockfile && pnpm run check, Crabbox resolved provider aws, c7a.8xlarge, run run_9fecc9c69bff, lease cbx_aa44306637df (amber-prawn-c221), exit 0 and leaseStopped=true. The full coverage lane reported 3,537 tests: 3,529 passed, 8 platform skips, 0 failed; aggregate coverage was 81.49% lines, 74.11% branches, and 87.35% functions.

Real Behavior Proof

• Claim: Bay renders actual closed GitHub throttle observations as a legible 403/429 series across 6h, 24h, and 7d, switches ranges without stale-response attribution, discloses incomplete and header-retention boundaries, and performs no GitHub API or mutation request.
• Exercised surface: real /bay Worker route and UI with the repository's deterministic, fully synthetic/redacted status and observability fixtures.
• Scenario/fixture: three closed throttle buckets across repository_actions, target_app, and public_read_fallback; range controls fetch 6/24/168 hours; 7d omits raw header detail and states its 24h retention.
• Command/environment: Docker-backed Crabbox local-container, target linux/amd64, image mcr.microsoft.com/playwright:v1.60.0-noble, fresh PR checkout, --no-hydrate; exact head efbe0ac96a431889d4dcdf4a4028cd3aafd3ff8a; run run_97017c5031cc; lease cbx_5bebead8cb6d (tidal-barnacle). The required-artifact contract matched proof-summary.json, trace.zip, and the storyboard; the command receipt reports runStatus=succeeded, exitCode=0, and leaseStopped=true.
• Observed result: 11 focused assertions passed, three evidence frames cover 6h/24h/7d, zero direct GitHub API requests, and zero mutation requests.
• Artifacts/trace: retained Crabbox artifact bundle run_97017c5031cc-artifacts.tgz (2,307,391 bytes), SHA-256 986B2EE66524445AA92D8C7CD6CDE120E112289BDEE7FC1D1ECDE193EB417478; proof-summary.json SHA-256 104BFCDF8E80DA70A6BE2079644B28A61DCFFAF081FFFE15FB8F1CFDE0CD70FE; trace.zip SHA-256 179464FFA5360CDED46BBAA7F84817F2DB6986D85E9608A7173CD17708D9BF1F; storyboard SHA-256 45AD73C778EF921E2B91AF918B03FB92DAB8880196AD56C935FC35325FAD29EB; 6h frame SHA-256 9A42964B8802D8320C0B21E4233A4C2825055F33DFB9DD7FC250C0A843410502; 24h frame SHA-256 A2BFFEDBF696240B764F0FEB3A1299524A297B957CF45242673B84D2250A6874; 7d frame SHA-256 41E0961A88A36259B383DAF1377B857AF79719D463249656D9D8561AC9B7B19F.
• Limits: fixtures are synthetic and redacted; this proves rendering, range requests, completeness semantics, and network/mutation boundaries, not production traffic volume or deployment state.

Privacy and overlap

• Maintainer decision: Martin explicitly approved the aggregate-only public throttle-history boundary and authorized this PR to merge on 2026-08-16 after reviewing the security-boundary risk. That approval is limited to the closed, bounded projection documented here; it does not authorize identities, raw request detail, or production mutation.
• feat(telemetry): observe GitHub publication egress #1140, fix(telemetry): add a 15-minute egress window #1142, Recover exact-review quota handoffs #1150, and fix(dashboard): make Phase 0.5 evidence activation-grade #1156 supply the existing Phase 0/0.5/0.6 observer, rollups, retention, and completeness model reused here.
• fix(bay): correlate direct re-review completions #1172 is already in the base.
• Privacy PR fix: contain public worker status metadata #1171 merged as 43799a11fc2612df7d80918e0012d6f03b339747 and is contained by the current base 61276c87cbb73cfec39e9649cee6862cf84a935f. This PR adds only the bounded throttle projection after its strict public sanitizers. Reconciliation removed a State writer caller that would otherwise have reintroduced a surface fix: contain public worker status metadata #1171 withholds; no removed identity field or route returns.
• Draft Phase 1 PR fix(publication): stop repository pool on first throttle #1145 remains inactive and is not enabled, modified, or depended on here.

Risks, rollout, and rollback

Risk is limited to the public read-only telemetry projection and Bay presentation. Legacy cached v2 responses fall back to the detailed rows they already contain. The new projection is bounded and can fail closed independently of raw header-detail completeness.

Rollback is a normal revert of this commit: the underlying v2 rollups and retention remain unchanged. The final read-only pre-merge health check observed deployment SHA 61276c87cbb73cfec39e9649cee6862cf84a935f, matching the PR base; this PR does not claim that its head is deployed. It makes no deployment, schedule, capacity, queue, credential, DLQ, or Phase 1 change.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 16, 2026, 5:22 AM ET / 09:22 UTC.

ClawSweeper review

What this changes

Adds a bounded aggregate 403/429 GitHub-throttle history API projection and a read-only OpenClaw Bay card with 6-hour, 24-hour, and 7-day views.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep this PR open for normal maintainer landing: the current head has no actionable correctness or security finding, and its exact-head browser proof addresses the prior stale-proof blocker.

Priority: P3
Reviewed head: efbe0ac96a431889d4dcdf4a4028cd3aafd3ff8a

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	A well-scoped public-observability feature with exact-head behavior evidence, focused regression coverage, and no actionable review finding.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (live_output): The PR body records an exact-head Crabbox browser receipt covering the real Bay route, all range requests, visible 403/429 behavior, and zero direct GitHub API or mutation requests.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body records an exact-head Crabbox browser receipt covering the real Bay route, all range requests, visible 403/429 behavior, and zero direct GitHub API or mutation requests.
Evidence reviewed	5 items	Bounded server projection: The PR accepts the 168-hour view and derives the chart only from complete attempted wire observations with exact 403/429 statuses, excludes the open bucket, and fails the series closed for incomplete evidence or insufficient coverage. Public boundary validation: The public projector caps series rows, accepts only closed pool classes and 403/429 buckets, validates time boundaries and conservation, and recomputes completeness before serializing the new field. Bay behavior and proof: Bay maps its existing range selector to 6/24/168-hour requests and renders a read-only card; the focused runner loads the real Bay route, exercises all three ranges, validates labels/tooltips, and asserts zero mutations and direct GitHub API requests.
Findings	None	None.
Security	None	None.

How this fits together

GitHub publication egress is retained as bounded SQLite rollups and exposed through a fail-closed public observability projection. OpenClaw Bay fetches that projection and renders aggregate operator telemetry without controls or identifying request data.

flowchart LR
A[GitHub egress observations] --> B[Bounded rollups]
B --> C[Closed throttle series]
C --> D[Public data validation]
D --> E[Bay range request]
E --> F[Read-only throttle chart]

Loading

Before merge

• Resolve merge risk (P1) - This expands a public observer contract, so the aggregate-only pool-class/status boundary and fail-closed completeness semantics remain merge-critical even though the reviewed projector preserves them.
• Resolve merge risk (P1) - The supplied browser proof uses synthetic redacted fixtures; it demonstrates rendering and request boundaries, not deployed traffic volume or deployment state.
• Complete next step (P2) - No repair lane is needed: the review found no mechanical defect, and the remaining action is ordinary maintainer landing under the documented aggregate-only privacy approval.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Diff footprint	16 files; +970 / -51 lines	The feature crosses rollup storage, public projection, Bay rendering, proof, documentation, and focused tests.
Added-line split	runtime +225, proof/docs +418, tests +327	The implementation adds substantial contract and proof coverage alongside the runtime change.
Exact-head proof	3 ranges, 11 assertions, 0 mutation requests	The recorded browser receipt directly covers the changed Bay interaction and its observer-only boundary.

Merge-risk options

Maintainer options:

1. Accept the bounded observer surface (recommended)
   Land the reviewed aggregate-only 403/429 projection with its current caps, sanitization, and fail-closed coverage behavior.
2. Pause for a new disclosure review
   Pause this PR if maintainers no longer approve exposing closed pool-class totals or status-specific throttle counts publicly.

Technical review

Best possible solution:

Preserve the bounded aggregate-only projection, merge it only under the documented privacy approval, and verify the deployed read-only card against normal operational telemetry after rollout.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a defect report; the PR supplies an exact-head browser proof that exercises the real Bay route with deterministic redacted fixtures across all three ranges.

Is this the best way to solve the issue?

Yes. Reusing the existing bounded rollups and validating a narrow aggregate projection is the maintainable path; it avoids a parallel telemetry store and preserves the public privacy boundary.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 61276c87cbb7.

Labels

Label changes:

• add proof: sufficient: Contributor real behavior proof is sufficient. The PR body records an exact-head Crabbox browser receipt covering the real Bay route, all range requests, visible 403/429 behavior, and zero direct GitHub API or mutation requests.
• add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body records an exact-head Crabbox browser receipt covering the real Bay route, all range requests, visible 403/429 behavior, and zero direct GitHub API or mutation requests.
• remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

• P3: This is a bounded operator-observability enhancement rather than a regression in the review or publication workflow.
• merge-risk: 🚨 security-boundary: The PR adds a public telemetry projection, making its closed aggregate disclosure and sanitizer contract merge-critical.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body records an exact-head Crabbox browser receipt covering the real Bay route, all range requests, visible 403/429 behavior, and zero direct GitHub API or mutation requests.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body records an exact-head Crabbox browser receipt covering the real Bay route, all range requests, visible 403/429 behavior, and zero direct GitHub API or mutation requests.

Evidence

What I checked:

• Bounded server projection: The PR accepts the 168-hour view and derives the chart only from complete attempted wire observations with exact 403/429 statuses, excludes the open bucket, and fails the series closed for incomplete evidence or insufficient coverage. (dashboard/github-egress-telemetry.ts:519, efbe0ac96a43)
• Public boundary validation: The public projector caps series rows, accepts only closed pool classes and 403/429 buckets, validates time boundaries and conservation, and recomputes completeness before serializing the new field. (dashboard/public-observability.ts:864, efbe0ac96a43)
• Bay behavior and proof: Bay maps its existing range selector to 6/24/168-hour requests and renders a read-only card; the focused runner loads the real Bay route, exercises all three ranges, validates labels/tooltips, and asserts zero mutations and direct GitHub API requests. (docs/proof/openclaw-bay/run-proof.mjs:941, efbe0ac96a43)
• Feature provenance: The surrounding egress telemetry and public-boundary work on main dates to merged PRs including the Phase 0 telemetry merge and the public-status containment merge; this PR builds on those existing ownership boundaries rather than introducing a parallel store. (dashboard/public-observability.ts:976, 43799a11fc26)
• Current-head proof: The PR body records a Crabbox browser receipt at exact head efbe0ac, with 11 passing focused assertions, redacted artifact hashes, and no GitHub API or mutation requests. (efbe0ac96a43)

Likely related people:

• brokemac79: Authored the merged Phase 0 telemetry PR identified in the supplied history and the current branch's five commits. (role: introduced adjacent telemetry behavior and current implementation contributor; confidence: high; commits: ac340908bf69, efbe0ac96a43; files: dashboard/github-egress-telemetry.ts, dashboard/bay-page.ts, docs/proof/openclaw-bay/run-proof.mjs)
• Martin Cleary: Current-main history attributes the public observability serializer to the merged public-status containment work and shows repeated merged egress/dashboard work. (role: recent public-boundary contributor; confidence: high; commits: 43799a11fc26, 9fb6d6bfe562, 56cb78d60734; files: dashboard/public-observability.ts, dashboard/github-egress-telemetry.ts)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (12 earlier review cycles; latest 8 shown)

• reviewed 2026-08-15T23:17:44.178Z sha c805a36 :: needs real behavior proof before merge. :: [P2] Refresh the Bay proof at the current head
• reviewed 2026-08-15T23:41:27.495Z sha c805a36 :: found issues before merge. :: [P1] Rebase the throttle chart onto current main
• reviewed 2026-08-15T23:54:17.399Z sha ff88024 :: needs real behavior proof before merge. :: [P2] Refresh the throttle proof at the rebased head
• reviewed 2026-08-16T00:00:48.088Z sha ff88024 :: needs changes before merge. :: [P3] Match the GitHub API host exactly
• reviewed 2026-08-16T00:54:33.263Z sha c0917b3 :: needs maintainer review before merge. :: none
• reviewed 2026-08-16T00:59:49.219Z sha c0917b3 :: needs maintainer review before merge. :: none
• reviewed 2026-08-16T08:50:45.008Z sha 16d8243 :: needs real behavior proof before merge. :: [P2] Refresh the browser proof at the current head
• reviewed 2026-08-16T09:12:54.414Z sha efbe0ac :: needs real behavior proof before merge. :: [P2] Refresh Bay proof for the final head

[brokemac79]

@clawsweeper review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31909863963
• Updated: 2026-08-15T21:35:52.744Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31910653665
• Updated: 2026-08-15T21:53:28.468Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31918420971
• Updated: 2026-08-16T00:56:54.178Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/31938975117
• Updated: 2026-08-16T09:25:09.658Z