Recover exact-review quota handoffs

[brokemac79]

Summary

• Convert complete, attributable GitHub egress reset observations into bounded durable exact-review credential circuits.
• Stagger matching queue work with deterministic 1–30 second recovery jitter and wake the queue at the earliest pending member boundary.
• Expose bounded review-handoff recovery causes through the read-only queue/status/Bay surfaces.
• Preserve replay, migration, attribution, privacy, and observer-only boundaries.

This is the Phase 0.5 follow-up selected from CSW-127 post-merge observation. Phase 1 remains separately held and inactive.

Problem

Phase 0 records sanitized GitHub publication egress telemetry, but the queue did not consume credible reset evidence as durable admission state. A quota-affected cohort could therefore retry without a shared, attributable recovery boundary. Review workflows that failed, cancelled, or never claimed their lease also requeued without an operator-visible bounded recovery cause.

The post-merge observation did not establish a provider-side cause for cancelled runs, so this change records objective queue/workflow facts without assigning one.

Related work:

• feat(telemetry): observe GitHub publication egress #1140 introduced Phase 0 egress telemetry.
• fix(telemetry): add a 15-minute egress window #1142 added the bounded 15-minute observer window used by CSW-127.
• fix(publication): stop repository pool on first throttle #1145 is the held Phase 1 draft; this PR does not enable or implement its enforcement.

Implementation

• Accept a circuit candidate only from complete repository_actions or public_read_fallback observations with a credible future retry/reset boundary, bounded to two hours.
• Keep ownerless target_app observations observational so an ambiguous installation cannot block unrelated owners.
• Persist the accepted circuit candidates with the telemetry receipt. A duplicate replays the originally stored candidates; a conflicting body cannot introduce or extend a circuit; legacy receipts migrate fail-closed.
• Apply accepted circuits through the existing authenticated queue transaction and reschedule the Durable Object alarm.
• Hold each matching publication member, authority reservation, or command intake until its raw reset plus deterministic 1–30 second jitter; report the raw reset separately from the latest aggregate recovery_until boundary.
• Count bounded claim_timeout, execution_timeout, workflow_cancelled, and workflow_failed handoff causes without misclassifying typed throttle or coordination outcomes.
• Render the cause in OpenClaw Bay as read-only health context and synchronize the public API, dashboard, Bay, scheduler, and limits guides.

No workflow, permission, credential, configuration gate, deployment, queue, DLQ, or production state is changed by this PR.

Validation

Executed on the final branch/base unless the proof source is called out below:

• pnpm run build:all
• node --test test/github-egress-telemetry.test.ts test/exact-review-publication-batches.test.ts test/dashboard-worker-command-intake.test.ts test/dashboard-worker-queue-policy.test.ts test/dashboard-worker-queue-runtime.test.ts test/exact-review-health.test.ts test/dashboard-worker-bay-records-routes.test.ts
• pnpm run lint:dashboard
• pnpm run lint:scripts
• pnpm run check:docs
• pnpm run check:limits
• git diff --check

The controlled container proof passed 324 focused tests with zero failures. The final committed Codex review against ae36d608d01701af7e06c313be96689068b5c890 found no actionable correctness issues and independently completed the dashboard TypeScript build.

The final read-only local ClawSweeper review covered base ae36d608d01701af7e06c313be96689068b5c890 through head 960488d7d98c897c12cc0b581c1701f52ab6de7a: review complete, high confidence, patch correct, no review comments, security cleared, Real Behavior Proof sufficient, overall/proof rating A/A. Earlier review findings about stale operator wording were accepted, fixed across all active guides, and re-reviewed cleanly.

Real Behavior Proof

Claim: Complete, attributable signed egress evidence creates one bounded durable circuit; replay conflicts cannot change it; matching publication work recovers only at its deterministic reset-plus-jitter boundary; cancelled review handoffs remain queued with a bounded visible cause; public projections remain privacy-clean and observer-only.

Exercised surface: The production Worker and Durable Object code paths, signed internal telemetry ingest, receipt storage and migration, queue admission/alarm behavior, public queue projection, and OpenClaw Bay rendering.

Scenario: A local GitHub fixture supplied complete repository Actions/public-read-fallback quota evidence, ambiguous target-App evidence, a conflicting receipt replay, pending publication work around the reset boundary, and one cancelled review handoff.

Command/environment: Docker-backed Crabbox local-container, image mcr.microsoft.com/playwright:v1.60.0-noble, lease cbx_1df4ca79b0b1 (blue-krill-5534). The executed source was 3a95d75996e654d9a97294d55a3235998fc3cd43 on current base ae36d608d01701af7e06c313be96689068b5c890; subsequent commits only refresh the receipt and synchronize documentation.

Observed result: 324/324 focused tests passed; signed ingest deduplicated; conflicting receipt evidence was fenced; exactly one attributable circuit was created; raw-reset claim was rejected; reset-plus-jitter claim succeeded; Bay rendered 1 recovering after workflow cancelled; public output was privacy-clean.

Artifacts:

• docs/proof/csw-127-phase0-5/container-receipt.json
• docs/proof/csw-127-phase0-5/README.md
• Executable fixture: docs/proof/csw-127-phase0-5/run-proof.sh

Limits: The loopback fixture proves ClawSweeper response handling and durable transitions, not GitHub's external scheduler or actual quota refill. It does not assign a provider-side cause to earlier cancelled runs. Phase 1 remains inactive and was not exercised. Broad Windows coverage retained unrelated baseline limitations from shell-only fixtures (/bin/bash unavailable in WSL) and temporary-directory EPERM cleanup; the seven affected test files were green on Windows and Linux. Repository-wide Linux format checking of the Windows checkout also encountered the existing CRLF representation across untouched files; scoped lint and all non-format gates were clean.

Risks and rollout

• This changes maintenance automation admission, so the main risk is an incorrectly attributed reset deferring a credential pool. Attribution fails closed, target-App circuits require an owner, resets are future-only and capped at two hours, and receipt replays cannot inject new candidates.
• Recovery remains automatic and staggered. blocked_until is the raw credential reset; recovery_until is the latest current member boundary, not a command or manual gate.
• Bay and public status remain observer-only. No recovery, workflow, queue, deploy, or rollback control is added.
• Normal deployment and live smoke are required after maintainer approval and merge. This branch performs neither.

Release-note context

Exact-review work now uses credible GitHub quota reset evidence to recover in a bounded, staggered way and reports objective handoff recovery causes to operators.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 12, 2026, 8:16 PM ET / August 13, 2026, 00:16 UTC.

ClawSweeper review

What this changes

This PR converts eligible signed GitHub egress reset observations into bounded exact-review recovery circuits, adds reset-plus-jitter scheduling, and reports recovery causes through status and Bay.

Merge readiness

⚠️ Ready for maintainer review - 4 items remain

Keep open for maintainer direction. Current main keeps egress telemetry observational, while this well-proven draft makes it bounded queue-admission state; that production-policy change needs explicit sponsorship before landing.

Priority: P2
Reviewed head: 960488d7d98c897c12cc0b581c1701f52ab6de7a
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	The patch is focused, guarded by targeted regression coverage, and supported by strong controlled runtime proof; only the production-policy decision remains.
Proof confidence	🦞 diamond lobster (5/6) ✨ media proof bonus	Sufficient (linked_artifact): A Docker-backed local-container run exercised the real Worker and Durable Object, recorded signed-ingest and jittered-recovery results, and includes a directly inspectable Bay screenshot; redactable artifacts contain no production mutation.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (linked_artifact): A Docker-backed local-container run exercised the real Worker and Durable Object, recorded signed-ingest and jittered-recovery results, and includes a directly inspectable Bay screenshot; redactable artifacts contain no production mutation.
Evidence reviewed	7 items	Current-main boundary: Current main explicitly describes reset authority as observational and says it does not alter retry behavior, so the PR is a new production policy rather than an already-fixed bug. Bounded admission implementation: Eligible complete repository-Actions and public-read-fallback observations are the only telemetry converted to circuits; unattributable target-App observations are excluded and resets are capped at two hours. Durable queue handoff: The telemetry endpoint persists accepted receipt candidates through the existing queue transaction and reschedules the next wake-up only when circuit state changed.
Findings	None	None.
Security	None	None.

How this fits together

Exact-review work flows through a durable queue before publication and review workflows run. This change feeds sanitized quota observations into that queue’s recovery timing and exposes bounded health state to operator-facing status surfaces.

flowchart LR
A[Signed GitHub egress telemetry] --> B[Receipt validation]
B --> C[Circuit eligibility]
C --> D[Durable exact-review queue]
D --> E[Jittered recovery alarm]
D --> F[Public status API]
F --> G[OpenClaw Bay]

Loading

Decision needed

Question	Recommendation
Should ClawSweeper move from observe-only egress telemetry to default-on bounded automatic repository-Actions queue recovery in this Phase 0.5 scope?	Sponsor bounded Phase 0.5 recovery: Accept the capped, attributable recovery policy and require normal deployed smoke observation after merge.

Why: The safeguards establish a technically bounded implementation, but only maintainers can decide whether automatically deferring queue work from telemetry is the desired production policy.

Before merge

• Resolve merge risk (P1) - Merging changes telemetry from observer-only data into automatic repository-Actions publication deferral; a mistaken attribution can defer recovery for up to the two-hour cap.
• Resolve merge risk (P1) - The receipt migration and queue/status contract need normal post-deploy observation, since the supplied proof uses a local Worker and loopback GitHub fixture.
• Complete next step (P2) - A maintainer must choose whether the new default automatic recovery policy should ship; no narrow mechanical repair is indicated.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Runtime versus tests	dashboard +310/-39, tests +257/-12	The durable-state behavior is accompanied by focused regression coverage across telemetry, queue, command-intake, health, and Bay surfaces.
Proof package	324 focused tests; 1 local Worker/Durable Object scenario	The supplied after-fix proof exercises the production queue boundary and records observable recovery results.

Merge-risk options

Maintainer options:

1. Sponsor bounded recovery (recommended)
   Explicitly accept the new automatic deferral behavior and validate deployed queue recovery before any broader enforcement work.
2. Pause at the observer boundary
   Keep the existing telemetry-only behavior if maintainers do not want reset observations to influence publication timing yet.

Technical review

Best possible solution:

Explicitly sponsor the bounded Phase 0.5 recovery policy, then land with post-deploy monitoring of capped circuits and staggered recovery while keeping Phase 1 enforcement inactive.

Do we have a high-confidence way to reproduce the issue?

Yes — the linked local-container proof runs the real Worker and Durable Object through signed ingest, queue recovery, status projection, and Bay rendering; the final head retains the exercised product code.

Is this the best way to solve the issue?

Unclear — the implementation is narrow and guarded, but whether telemetry should automatically alter queue admission is a maintainer policy decision rather than a purely mechanical fix.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ae36d608d017.

Labels

Label changes:

• add P2: This is a bounded reliability improvement to exact-review recovery rather than a confirmed urgent outage.
• add merge-risk: 🚨 compatibility: The PR migrates stored telemetry receipts and changes the public circuit-status contract.
• add merge-risk: 🚨 availability: Incorrect circuit attribution could temporarily defer repository-Actions publication work.
• add merge-risk: 🚨 automation: The patch changes durable queue admission, alarm scheduling, and workflow-handoff recovery behavior.
• add proof: sufficient: Contributor real behavior proof is sufficient. A Docker-backed local-container run exercised the real Worker and Durable Object, recorded signed-ingest and jittered-recovery results, and includes a directly inspectable Bay screenshot; redactable artifacts contain no production mutation.
• add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (linked_artifact): A Docker-backed local-container run exercised the real Worker and Durable Object, recorded signed-ingest and jittered-recovery results, and includes a directly inspectable Bay screenshot; redactable artifacts contain no production mutation.

Label justifications:

• P2: This is a bounded reliability improvement to exact-review recovery rather than a confirmed urgent outage.
• merge-risk: 🚨 compatibility: The PR migrates stored telemetry receipts and changes the public circuit-status contract.
• merge-risk: 🚨 availability: Incorrect circuit attribution could temporarily defer repository-Actions publication work.
• merge-risk: 🚨 automation: The patch changes durable queue admission, alarm scheduling, and workflow-handoff recovery behavior.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (linked_artifact): A Docker-backed local-container run exercised the real Worker and Durable Object, recorded signed-ingest and jittered-recovery results, and includes a directly inspectable Bay screenshot; redactable artifacts contain no production mutation.
• proof: sufficient: Contributor real behavior proof is sufficient. A Docker-backed local-container run exercised the real Worker and Durable Object, recorded signed-ingest and jittered-recovery results, and includes a directly inspectable Bay screenshot; redactable artifacts contain no production mutation.

Evidence

What I checked:

• Current-main boundary: Current main explicitly describes reset authority as observational and says it does not alter retry behavior, so the PR is a new production policy rather than an already-fixed bug. (docs/github-egress-telemetry.md:133, ae36d608d017)
• Bounded admission implementation: Eligible complete repository-Actions and public-read-fallback observations are the only telemetry converted to circuits; unattributable target-App observations are excluded and resets are capped at two hours. (dashboard/github-egress-telemetry.ts:568, 960488d7d98c)
• Durable queue handoff: The telemetry endpoint persists accepted receipt candidates through the existing queue transaction and reschedules the next wake-up only when circuit state changed. (dashboard/exact-review-queue.ts:710, 960488d7d98c)
• Regression coverage: Focused tests cover attributable Actions-only circuit creation, receipt replay, conflicting receipt fencing, and fail-closed legacy receipt migration. (test/github-egress-telemetry.test.ts:611, 960488d7d98c)
• Real behavior proof: The committed local-container receipt records a real Worker/Durable Object run with 324 focused passing tests, jittered recovery, receipt fencing, privacy checks, and Bay rendering; final-head changes after the exercised source are documentation and proof artifacts only. (docs/proof/csw-127-phase0-5/container-receipt.json:1, 960488d7d98c)
• Current-main and release check: The PR head is not an ancestor of current main and no local tag contains it, so the central behavior is neither implemented on main nor released. (960488d7d98c)

Likely related people:

• Peter Steinberger: Recent current-main history covers durable command intake, queue recovery, and related operator behavior. (role: recent queue-area contributor; confidence: high; commits: e0dc54438e5e, ae36d608d017, 6bf7f483fb03; files: dashboard/exact-review-queue.ts, dashboard/exact-review-read-model.ts, dashboard/exact-review-health.ts)
• Martin Cleary: Merged the Phase 0 egress telemetry and its follow-up window work that this PR extends. (role: merged telemetry contributor; confidence: high; commits: ac340908bf69, 6ca63d912cb1; files: dashboard/github-egress-telemetry.ts, docs/github-egress-telemetry.md)
• brokemac79: Authored the merged Phase 0 telemetry PR and this directly related Phase 0.5 continuation. (role: prior merged telemetry contributor; confidence: high; commits: 41bdd3bd2ad9, f5dc1b270b55; files: dashboard/github-egress-telemetry.ts, dashboard/exact-review-queue.ts)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.