Skip to content

docs(board): a worker restart during a repo index loses the run #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sshlg wants to merge 1 commit into from
Closed

docs(board): a worker restart during a repo index loses the run #238

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/qa-audit/issues.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,11 @@ frontend **A** (563 smells, 7 SOLID).
|---|---|
| 🔴 Critical | 0 |
| 🟠 High | **0** |
| 🟡 Medium | 1 |
| 🟡 Medium | 2 |
| 🟢 Low | 25 |
| ⚪ Info | 12 |

*Counted 2026-08-27, not estimated: **33 open `F-` rows and 76 struck** by `grep -cE '^\| F-'` / `grep -cE '^\| ~~F-'` over this file, plus **5 open `CB-` rows** those two commands do not see. The severity table above counts all 38 open rows of both kinds, which is why it does not match the `F-` figure — the two measure different sets and each says which. Both are derived from the rows themselves.*
*Counted 2026-08-27, not estimated: **33 open `F-` rows and 76 struck** by `grep -cE '^\| F-'` / `grep -cE '^\| ~~F-'` over this file, plus **6 open `CB-` rows** those two commands do not see. The severity table above counts all 39 open rows of both kinds, which is why it does not match the `F-` figure — the two measure different sets and each says which. Both are derived from the rows themselves.*

*(R1+R2 closed 4 High + 8 Medium + 3 Low. R3 (`fbf8112`) closed 2 High (F-SSH-08, F-RULE-01) +
5 Medium (F-RULE-05, F-DG-07/09, F-GRAPH-01, F-LEARN-07) + 1 Low (F-SSH-06). The 2026-07-19 UX
Expand Down Expand Up @@ -403,6 +403,7 @@ maintainability / reliability risks.
| CB-UX1 | ⚪ | **102 UX scenarios carry a verification older than 30 days.** 110 of 127 were dated 2026-07-19 while 152 commits had landed since; five were re-audited 2026-08-26 and the ceiling now stands at 105, of which 102 still have a changed Coverage file under them. Ordered and computable: `python3 scripts/ux_verification_status.py --backlog 2026-07-19`. The ceiling in `tests/unit/docs/test_ux_scenarios.py` may fall but not rise. | Re-audit in batches, worst first; date each verdict and add an `SCN-NNN` anchor so a machine can check it (21 of 127 have one). |
| ~~CB-OPS1~~ | 🟢 | **CLOSED, measured 2026-08-27.** A forced full re-index of the 9 981-file customer repository ran on Standard-2X and logged **zero `R14` and zero `R15`**, with `mem=` absent from the whole window — Heroku emits those only over quota, so absence is the evidence. `graph_build` completed over the full symbol set, which is the case this row said no run had exercised. Before the resize: 170 × R14 and 2 × R15 in 6.5 h with a 1 143 MiB peak against a 512 MiB quota. |
| CB-OPS2 | 🟡 | **The nightly cron can rebuild a repository the "Re-index repository" button never can.** `run_repo_index_task` is called by two ARQ jobs carrying two ceilings: the cron's `run_daily_project_knowledge_sync` at 7200 s, and `run_repo_index` at 1800 s. Measured on the same repository from `indexing_runs`: nightly `completed` in **42.4 min** (08-25 22:00), manual `TimeoutError` at **exactly 1800.02 s** inside `_run_code_symbol_embed` (08-27 09:30). Diagnosed once already — AUD-0819-20 added the knob on 2026-08-19 for this failure and left the default at the value just measured as too small. **Fix written, not yet in production:** `repo_index_job_timeout_seconds` defaults to 3600 on branch `fix/repo-index-ceiling`, with both orderings asserted in `tests/unit/services/test_repo_index_ceiling.py`. | Deploy, then force one manual full re-index and require it to reach `pipeline_end`; strike this row on that evidence, not on the merge. |
| CB-OPS3 | 🟡 | **A worker restart during a repo index loses the run, and nothing retries it.** Measured 2026-08-27: release `v279` restarted the worker at 10:59:34 UTC, 37 min into a 42-min index. arq logged `shutdown on SIGTERM ◆ 0 jobs complete ◆ 2 failed ◆ 0 retries ◆ 1 ongoing to cancel`, exited 143, and the fresh worker started at 10:59:45 with **no job re-queued** — grep over the next 30 min of worker log finds no `run_repo_index` or `run_daily_project_knowledge_sync` start. The reaper then correctly flipped both rows to `failed / stale run reaped` (visible in `error_log` since N3). `WorkerSettings` sets neither `retry_jobs` nor `max_tries`, so arq's defaults apply and still did not retry. Blast radius is bounded — the checkpoint survived with its completed steps recorded, so the next run resumes cheaply, and the nightly cron re-runs within 24 h — so this is a resilience gap, not lost data. | Decide the semantics before coding: re-enqueue on shutdown (risking a double run against the `_indexing_locks` / advisory-lock guarantees), or leave it to the cron and make the loss explicit in the UI. Not a silent implementation choice. |

**Verified-good in the codebase audit (no issue):** SQL identifier quoting (`connectors/base.py:262`
doubles quotes correctly), credential exposure (`ConnectionResponse` returns no secrets; Fernet at
Expand Down