fix(dashboard): retain Bay projection across queue probe failures - #1194
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
clawsweeper
Bot
added
merge-risk: 🚨 other
|
Codex review: needs maintainer review before merge. Reviewed August 17, 2026, 9:20 AM ET / 13:20 UTC. ClawSweeper reviewWhat this changesThe PR retains a recent verified public Bay queue projection after an optional queue-status probe rejects, while clearing live activity and showing lower-bound queue counts. Merge readinessKeep open for maintainer review. The patch has no actionable correctness finding, and current main does not contain this bounded Bay fallback. Priority: P2 Review scores
Verification
How this fits togetherThe dashboard Worker combines queue and worker observations into a privacy-filtered public status document. OpenClaw Bay reads that document to display public queue references without treating stale queue data as a live worker census. flowchart LR
A[Queue status probe] --> B[Dashboard status refresh]
C[Prior public status snapshot] --> B
B --> D{Probe rejected?}
D -->|recent complete sample| E[Retain reprojected queue]
D -->|invalid or expired| F[Fail closed]
E --> G[Clear live activity]
G --> H[Bay lower-bound display]
F --> H
Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Merge the bounded, allowlist-reprojected queue fallback while preserving unavailable live activity and lower-bound Bay presentation. Do we have a high-confidence way to reproduce the issue? Yes. The PR supplies an exact-head Crabbox trace for a rejected queue probe, and the added source-level regression covers the retained, expired, malformed, and unbound cases; this read-only review did not rerun it. Is this the best way to solve the issue? Yes. It uses the existing stale-window and public-projection boundaries, then clears generation-coupled live activity rather than fabricating a complete census. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against ecf222f5bd14. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (3 earlier review cycles)
|
9a488ba to
4590605
Compare
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
4590605 to
bea3ef7
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
rating: 🦞 diamond lobster
1 participant
Summary
/statsprobe rejects, including after the ordinary 60-second status cache has expired≥N) countsProblem
The Bay intermittently becomes entirely empty while the direct
/api/exact-review-queueendpoint still returns a complete public projection with active and pending work. During status refresh,attachExactReviewQueueStatustolerates optional queue probe failures withPromise.allSettled, but a rejected queue probe leavesexact_review_queueasnull. Thatnullis persisted and cached, erasing the last verified public snapshot and making a transient dependency failure look like zero work.The ordinary cached-status read also expired after 60 seconds, earlier than the intended 15-minute stale fallback window. Retaining a queue projection without adapting the Bay client was insufficient because the client previously discarded all queue references whenever the live activity census was incomplete.
This is an observability defect. It can hide a real review/publication backlog, but it does not itself stop reviews. The durable-lifecycle
over_capstate and the terminal-review completion outage are separate conditions.Implementation
When—and only when—the optional queue probe rejects, status refresh may load and retain the prior queue projection through the existing 15-minute stale window if the nested queue projection is complete and its own generation timestamp remains inside that bound. Before retention, the prior object is reprojected through
publicExactReviewQueueProjectionwith the current verified-public repository allowlist.Generation-coupled live activity is replaced with the canonical incomplete activity shape. The Bay client still renders verified queue references and queue-stage counts, labels them as lower bounds (
≥N), and does not invent live workers. A fulfillednullresult, an absent queue binding, malformed/partial input, or a nested queue sample older than 15 minutes still fails closed.Validation
git diff --check origin/main...HEADReal Behavior Proof
Claim: A status refresh that loses only the optional queue probe retains the last complete, privacy-filtered public Bay queue projection through the bounded stale window, clears unverifiable live activity, and renders queue-only references/counts without presenting them as a complete census. An environment with no queue binding and an expired nested sample both fail closed.
Exercised surface: the real in-memory ExactReviewQueue,
/api/exact-review-queue,/api/status, persisted status storage, response caching, health diagnostics, public projection filtering, and the production Bay client parser/renderer.Scenario/fixture: enqueue public
openclaw/openclaw#125204, obtain its complete public projection through the direct queue API, seed it as the prior status snapshot, then make/statsreject with a private marker while keeping the events endpoint available. Repeat after the ordinary 60-second cache age but within 15 minutes. Negative cases remove the queue binding, return fulfilled null/malformed data, and age the nested queue projection beyond 15 minutes.Command/environment: AWS Crabbox Linux (
c7a.8xlarge), provideraws, runrun_904311dcdb8d, leasecbx_68e44965aaf4, exact committed-tree headbea3ef7083fc79dd8cb48d81c2a1ef6e4c7e6416, proof base866506fb0f57da42f4664647b3d8619f9944475d.Observed result:
/api/statusretained the allowlisted queue reference in response, persistence, and cache projections after a rejected probe and after the ordinary cache age elapsed. It cleared live activity, remained degraded/incomplete, withheld the private failure marker, and the Bay rendered queue-only references with lower-bound counts. No binding, fulfilled null, malformed/partial input, and a nested projection older than 15 minutes all failed closed. All 139 tests and the boundary/type/lint/format gates passed.Artifact/trace: Crabbox run
run_904311dcdb8d, leasecbx_68e44965aaf4; terminal proof line:CSW-132 Bay projection Linux behavior proof: PASS.Limits: This deterministic exact-runtime proof does not deploy or mutate production. The optional
pr-behavior-proofskill named by workspace policy was unavailable in this session, so its required contract is reproduced explicitly here. Main later advanced through unrelated agent-runner and live-proof changes in #1196/#1198; the proved four-file tree remains unchanged.Reviews
866506fb0f57da42f4664647b3d8619f9944475d: cleandecision=keep_open,confidence=high,action=kept_open; no GitHub mutationRisks and rollout
The retained projection may be stale, which is why health and activity remain degraded and counts are lower bounds. Retention is bounded by the existing 15-minute stale window, requires a previously complete public projection, and occurs only on an actual optional-probe rejection. It cannot introduce unverified repositories, preserve live activity, or convert an absent binding into apparent data.
Rollback is to revert this PR. No data migration, queue repair, workflow change, or production-state mutation is required.
This PR fixes Bay projection continuity only. It does not repair the separate durable-lifecycle over-cap state, publication backlog, acute model/output generation failures, or hot-intake terminal-comment routing defect tracked in #1189.
Related