Repository navigation
Rules-aware skip memo: re-decode only changed records, invalidate on any rule change #154
Description
Prompt
Add to crawlkit a small helper that lets a crawler skip re-decoding source records whose bytes have not changed since the last sync, and that invalidates every skip automatically when the crawler's decoding rules change. API sketch:
state.RulesSignature(versions map[string]int, rules any) []bytehashes the declared decoder and derivation versions plus a canonical, path-tagged rendering of the rule tables (strings quoted, maps and sets in key order, compiled regular expressions by pattern; panic on an unknown kind so a new table cannot be skipped silently), and a per-record memo compares(source key, byte digest, signature)before the crawler decodes. Keep record storage in the app; crawlkit owns only the signature and the comparison. Reference implementation: https://github.com/ourostack/teamscrawl/blob/main/internal/teamsdesktop/memo.go (MemoSignature,hashValue).
Why
A crawler that re-reads a large local cache on every sync pays to decode records that have not changed. teamscrawl skips them with a per-record byte digest, which made a re-sync of a real Teams cache (about 62,000 messages) do work in proportion to what changed. The subtle part is invalidation: a new credential-scrub key, deny term or regular expression must re-read everything, and hashing the rule tables themselves (rather than relying on someone remembering to bump a version) is what makes that safe.
Why an issue, not a PR
Only teamscrawl uses this today, and crawlkit's boundary asks for at least two apps. If another crawler (slacrawl, discrawl, imsgcrawl) would adopt it, we can send a PR from the proven code; say the word.
Activity
clawsweeper commentedon Oct 6, 2026
Codex review: this still needs some work.
Summary
This issue remains unimplemented; repository policy permits closure only for implemented changes, while shared adoption still needs a maintainer decision.
Reproducibility: Source inspection confirms the requested helper is absent; the reported Teamscrawl resync improvement was not independently exercised.
Maintainer decision needed
Next step
Decide whether to sponsor a second crawler adopter before defining the shared memo API.
Review details
Best possible solution:
Keep the helper downstream until two crawlers establish a compatible signature and comparison contract with app-owned storage.
Do we have a high-confidence way to reproduce the issue?
Source inspection confirms the requested helper is absent; the reported Teamscrawl resync improvement was not independently exercised.
Is this the best way to solve the issue?
The credited Teamscrawl implementation is a plausible starting point after a second adopter validates the contract, with explicit version bumps still needed for code changes.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against ca93c9f70a4d.
Label changes
Label changes:
No label changes.
Label justifications:
P3: This optional performance capability establishes no broken crawlkit contract and awaits demonstrated shared adoption.Evidence reviewed
What I checked:
docs/boundary.md:3, ca93c9f70a4d)state/state.go:95, ca93c9f70a4d)worker/worker.go:25, ca93c9f70a4d)state/adapters.go:39, 3ff7edef8b33)internal/teamsdesktop/memo.go:28, ab5e739b4f66)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.Reviewed October 19, 2026, 8:01 AM ET / 12:01 UTC.
62 remaining items