Skip to content

OpenClaw two-lane capture (LLP 0172/0173): Lane A attach, Lane B sweep, json_path revival - #570

Open
philcunliffe wants to merge 32 commits into
masterfrom
integration/openclaw-two-lane-capture
Open

OpenClaw two-lane capture (LLP 0172/0173): Lane A attach, Lane B sweep, json_path revival#570
philcunliffe wants to merge 32 commits into
masterfrom
integration/openclaw-two-lane-capture

Conversation

@philcunliffe

@philcunliffe philcunliffe commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

OpenClaw two-lane capture: the reworked @hypaware/openclaw attach/detach module (Lane A),
the daemon-side scheduled sweep (Lane B), the json_path core revival, the
openclaw-steering-plugin/ deletion, and the acceptance/onboarding rewrites.

Spec LLP 0171 (human) to design LLP 0172 + executable plan LLP 0173 (neutral).

Status: complete — all 13 tasks landed, reviewed, green

#552 merged (17a9730), which cleared T12's external blocker. The updated master was
merged into this branch (74b262d), T12 was implemented, and the neutral:stuck label was
removed. Two review rounds ran to the cap; all seven findings were fixed and re-verified in the
committed tree, and triage found nothing residual. This PR is held for a human to merge.

⚠️ Merge-order conflict with #558

Both this PR and #558 create hypaware-core/smoke/flows/backfill_openclaw_fixture.js, and
neither is on master. Whichever merges first, the second will conflict on that path. The
two versions were written to different specs and neither is a superset of the other, so neutral
has not picked a winner. See the marker-signed note in this thread for the options.

Landed

Task Commit What
T1 55d5a15 Restore json_path attach-probe format, add sweep to BackfillContribution
T2 e9ea1f2 json_path detach: ownership, backup-not-discard, best-effort cache purge
T3 79abe76 daemon/status.js: restore the json_path attach-probe read branch
T4 a39e762 OpenClaw attach writes the two provider overrides, refusing to merge
T5 205b103 Manifest: restore json_path attach_probe, retire the steering-plugin copy
T6 b5922fd Config: validate sweep_cron and quiesce_ms in the backfill section
T7 e7002d9 Lane B sweep metadata: backfill provider + narrowed runner ctx
T8 e160426 Backfill quiesce window skips recently-modified session files
T9 606d87d Daemon sweep driver: run sweep-bearing providers on the tick loop
T10 f967134 openclaw-client-registration.test.js: finish the T4/T5 rewrite
T11 8876556 Delete openclaw-steering-plugin/ (LLP 0172 Section 5, R9)
T12 0c62a21 backfill_openclaw_fixture helper + quiesce/dedupe smoke coverage
T13 ed33d13 docs/ACCEPTANCE.md: rewrite openclaw_capture for two-lane capture

Each task branch was merged --no-ff after its own verification, then deleted. The merge
commits on this branch name the task branches, which is the ancestry record.

Review history

Round Head Outcome
CI repair 9460e9d Quiesce-window race in openclaw-backfill.test.js (test-only fix; feature untouched)
Round 1 9460e9d 1 major + 3 minor, all fixed at 9bf2af0
Round 2 9bf2af0 2 major + 1 minor, all fixed at 0e5f920
Triage 0e5f920 Zero residual findings; no follow-up issue needed

The two majors from round 2 are worth a reviewer's eye, since both were latent and neither is
visible from the diff alone:

  • sweep_cron / quiesce_ms were validated then discardedactivate() never threaded
    ctx.config into the backfill provider, so operator config silently had no effect. Fixed and
    covered by an activation-level regression test at the wiring seam.
  • attach() was not idempotent — refusal keyed on bare key presence, so re-attach-on-drift
    (ephemeral-port rebind, LLP 0086) refused forever and left OpenClaw pointed at a dead gateway
    while the status probe reported attached: true. Fixed by making refusal ownership-aware via
    a new shared src/core/config/provider_entry_ownership.js, now used by both attach and detach.
    A user-authored entry still refuses (R2 preserved), covered by four near-miss tests.

Full detail is in the marker-signed review records in this thread.

Nothing here is merged by neutral. This PR is held.

Change-Set: openclaw-two-lane-capture

test added 26 commits July 31, 2026 22:54
validateBackfillSection now accepts sweep_cron (a 5-field cron
expression, validated with core's shared isCronExpression grammar so a
malformed schedule is rejected the same way a sink's config.schedule
is) and quiesce_ms (non-negative integer) alongside the existing
on_join/window_days keys, added together so the unknown-key rejection
loop recognizes both from the same merge.

Task-Id: T6
…tribution

hypaware-plugin-kernel-types.d.ts: PluginAttachProbeManifest.format regains
'json_path' (removed by LLP 0143 after #212's orphaning danger), plus the
new container_path, provider_keys, and cache_glob fields the format needs,
reusing the existing marker_header. The comment at the removal site is
revised, not deleted: it now explains that the runtime support LLP 0173
T2/T3 add closes the gap #212 warned about, so a manifest can only declare
this format once both sides exist.

BackfillContribution gains an optional sweep?: { cron: string } field for
the daemon's periodic sweep (LLP 0173 T9), absent-by-default for every
existing contribution.

Both additions are purely additive: npm run typecheck (tsc --noEmit over
the whole tree, including hypaware-core/plugins-workspace/openclaw) and
npm test pass unchanged, proving no existing consumer's typecheck shifted.

Task-Id: T1
LLP 0169 reverses LLP 0152's premise: there is a real, reversible
settings write for OpenClaw again, so the adapter's honest no-op
attach() has nothing left to be honest about.

New hypaware-core/plugins-workspace/openclaw/src/attach.js exports
createOpenclawAttach({homeDir, env, fs}), mirroring the Claude adapter's
attach() shape (same AiGatewayClientAttachContext, same
withSpan('client.attach', ...), same dry-run branch). It reads
openclaw.json through the one core settings-path seam (so $OPENCLAW_HOME
relocation resolves the same file the manifest's probe will), refuses
with {status:'failed', reason} when models.providers.anthropic or
.openai is already there, and otherwise writes both entries whole from
attachCtx.endpoint: bare origin for anthropic, endpoint + '/v1' for
openai, each with the x-hypaware-upstream marker header and the
mandatory empty models array.

The refusal check runs entirely before the single atomicWriteFile, so
there is no partial write to roll back, and it returns rather than
throws, which is the whole mechanism by which a refuse during
attach-on-join warns instead of failing the join. Every other key in
openclaw.json is carried through by reference. Both output modes end
with the 'openclaw gateway restart' instruction, since a --json caller
is as blocked on the restart as a human is.

index.js drops the no-op body, STEERING_PLUGIN_NAME,
ROUTING_OWNED_BY_STEERING_PLUGIN_MESSAGE and the @ref LLP 0143#decision
block, and wires activate() to the new effect. The registered attach()
keeps the kernel's Promise<void> contract, so the outcome object is
dropped there on purpose: both callers already derive success from a
throw plus the one-line JSON the effect writes.

Tests cover the refusal (including that the file is byte-identical
after it, and that it never throws), the exact two-entry shape with the
bare-origin/+v1 asymmetry, key preservation, the restart instruction on
both output modes, dry-run, $OPENCLAW_HOME, and the missing/malformed
config hard failures. The two attach tests in
openclaw-client-registration.test.js are retargeted at the new behavior
so the suite stays green; T10 owns that file's fuller rewrite.

Task-Id: T4
Restores the probe.format === 'json_path' read branch removed by LLP
0143 / PR #510, parallel to the existing json/toml branches in
probeClientAttachFromDescriptor: navigate container_path + each
provider_keys entry, read headers[marker_header], and report attached
when it equals the expected provider key for at least one configured
key. Pure read, no ownership/backup concerns.

Task-Id: T3
listSessionFiles(agentsDir) gains an optional quiesceBeforeMs cutoff, and
runOpenclawBackfill() computes it once per run as Date.now() - quiesceMs.
quiesceMs resolves from the plugin's own config.backfill.quiesce_ms,
defaulting to 180000ms (QUERY_FLUSH_DEBOUNCE_MS plus a one-minute margin),
so a run never imports a session file OpenClaw is still mid-write on, or a
settlement pass is still mid-flush against. Composes with the existing
effectiveProviders/partitionByBackend CLI-backend logic (R10) rather than
replacing it.

Task-Id: T8
createOpenclawBackfillProvider now populates the contribution's opt-in
sweep field from config.backfill?.sweep_cron, defaulting to every 5
minutes when absent (LLP 0172#lane-b-sweep, R7).

src/core/commands/backfill.js's runBackfillProvider, runProvider, and
resolveOwnersForRun now declare a new BackfillRunnerContext interface
(env, config, storage, backfills, backfillMaterializers) instead of the
full CommandRunContext, a pure structural narrowing so the daemon-side
sweep driver (LLP 0173 T9) can build one without assembling registries
it never uses. Existing hyp backfill CLI-path and onboarding-finale call
sites keep typechecking and passing unchanged.

Task-Id: T7
…cache purge

LLP 0143 pulled the `json_path` branch out of the disk-driven undo because
LLP 0152 left nothing on disk for it to reverse. LLP 0169 reverses that
premise, so the branch comes back - reshaped for the two provider entries
attach now writes, not the single shadow provider of the old design.

`detachJsonPathProviders` judges each `provider_keys` entry on what it points
at, because this format has no HypAware-owned marker to replay: its undo
record IS the entry. Ours (the gateway's own `baseUrl`, in either the bare
origin or `+ /v1` spelling, with `marker_header` naming its own key, in the
shape attach produces) is deleted. Anything else present at our key is backed
up to a `_hypaware_detach_backup.<key>` sibling inside the same container
before the live key goes, following the `prev_malformed` precedent of LLP
0163: never discard a value HypAware did not write. That closes the
json/toml-vs-json_path asymmetry LLP 0163 flagged as worth its own look,
converging on the outcome without the top-level marker key LLP 0163 correctly
ruled out for this client.

The derived caches (`cache_glob`, relative to the client's config home) are
then purged of the same keys, best-effort: they do not self-heal, so a
partial purge beats none, and one unreadable cache file is logged and skipped
rather than failing a detach whose settings half already landed.

An unknown gateway base URL has no safe default here - guessing either way
silently deletes a foreign value or reports a finished detach over a client
still routed at a dead port - so it refuses (`EXPECTED_BASE_URL_UNKNOWN`).
Both callers degrade correctly: `reverse()` keeps the marker, `hyp detach`
prints the reason.

Both real callers thread the base URL: `detachClientViaCore` resolves it
through the same three rungs manual attach already walks (live
`localEndpoint()`, configured `listen`, the daemon's persisted bound port),
every one optional so detach keeps working with the gateway capability
unloaded; `reverse()` passes the `ctx.endpoint` `perform()` attached with.

Task-Id: T2
…ugin copy

hypaware.plugin.json gains contributes.client.attach_probe (design 1.4:
json_path format, .openclaw/openclaw.json settings file, models.providers
container, anthropic/openai provider keys, x-hypaware-upstream marker
header, agents/*/agent/models.json cache glob). description and
picker[0].summary drop every @hypaware/openclaw-steering-plugin reference
and state the two capture tiers directly: live gateway capture once
attached, plus a periodic transcript sweep.

Claude's manifest gains the LLP 0167#onboarding line naming the
claude-cli/<model> case OpenClaw's CLI-backend exclusion produces, so a
user knows which picker entry an OpenClaw-routed Claude Code session
belongs to.

projector.js's UPSTREAM_HEADER comment no longer credits the deleted
steering plugin; it now describes the config-override write attach()
itself makes.

Adds a manifest-shape test asserting attach_probe parses to the exact
design-1.4 fields and that description/summary no longer match
/openclaw-steering-plugin/. Updates the one existing assertion this
manifest change makes false (the R7 "no attach_probe" descriptor check)
to the restored json_path shape; the remaining behavioral rewrites of
that test file are T10's scope.

Task-Id: T5
…8' into HEAD

# Conflicts:
#	hypaware-core/plugins-workspace/openclaw/src/backfill.js
#	test/plugins/openclaw-backfill.test.js
Removes openclaw-steering-plugin/ in full (src/, test/, package.json,
openclaw.plugin.json, .d.ts files) and test/plugins/openclaw-steering-plugin.test.js:
Lane A's config-override entries make OpenClaw route to the gateway on
its own, so the credential-borrowing runtime auth shim, the live
wire-parity mirror, the steering decision logic, the live warning
ledger, and the gateway endpoint resolver that fed them no longer have
a purpose.

Also drops tsconfig.json's stray "openclaw-steering-plugin" include
entry (line 19), not named in the design's own deletion inventory but
found verifying the deletion against the real tree; leaving it would
have left a dead include path.

docs/ACCEPTANCE.md and test/plugins/openclaw-manifest.test.js still
mention the package name (an onboarding rewrite reserved for T13, and
a T5 regression test asserting the manifest no longer references it,
respectively); neither is in this task's file list.

Task-Id: T11
T4 and T5 already retargeted the two attach() no-op tests and the
descriptor's attach_probe assertion to keep the suite green while they
landed; this closes the two pieces both left for T10:

- The "honest no-op" detach test's comment still credited the retired
  R7 no-attach_probe guard. Since T5 restored the manifest's json_path
  attach_probe, the no-op this test actually observes on a fresh temp
  HOME is detachClientFromDisk's absent-settings-file guard instead.
  Corrected the comment to say so.
- Added a companion case that stages a real openclaw.json via the
  actual createOpenclawAttach() effect, then drives the same hyp
  detach CLI entry point (buildClientDescriptorMap's real manifest
  descriptor -> detachClientFromDisk's json_path branch) and asserts
  the ownership-based detachJsonPathProviders (T2) actually fires:
  changed:true, the removed baseUrl, and both provider entries gone
  from the file while everything else in it is untouched.

Task-Id: T10
… loop

New `src/core/daemon/backfill_sweep.js`. `createBackfillSweepDriver({backfills,
backfillMaterializers, env, config, storage})`'s `tick({now})` walks
`backfills.list()`, skips any contribution with no `sweep` field or a cron that
is not due (`cronMatches`, the sink driver's own due-check), and fires
`runBackfillProvider` per due contribution with a `sweep-<name>-<now>` dev run
id. Runs are fired unblocked: `tick()` resolves once each run has been started,
never once one finishes, so a provider's transcript scan cannot stall the sink
snapshots, the source-detail refresh, or `persist()` later in the same tick. Both
settlements are handled, so a failing run is a logged `backfill.sweep_failed`
record (component `openclaw`, operation `backfill.sweep`, `error_kind`) rather
than an unhandled rejection that would take the daemon process down. A malformed
`sweep.cron` is logged and treated as not due rather than thrown, so one
provider's bad metadata cannot skip the rest of the list.

`runtime.js`'s `runTick()` calls `await sweepDriver.tick({now})` directly after
the existing sink-driver tick, riding the same `DEFAULT_TICK_INTERVAL_MS`
60-second loop: a `*/5 * * * *` schedule only needs a due-check once a minute, so
this opens no second timer to start, drain, and account for at shutdown.

Also repairs the typecheck this task's branch point already failed: T7's
`sweep: { cron: opts.config?.backfill?.sweep_cron ?? ... }` does not compile,
because the plugin's config slice is a `JsonObject` and every step below its root
is a `JsonValue`. Read through a `resolveSweepCron` mirroring the
`resolveQuiesceMs` helper already sitting beside it; behavior is unchanged.

Externally blocked for real capture: until PR #552 (issue #543) merges, the
LLP 0158 reader still reads OpenClaw v3 fields flat, so a sweep projects nothing
from a real transcript. These tests passing is not evidence that it does.

Tests: the due-check fires only sweep-bearing, cron-due contributions and builds
the narrowed `BackfillRunnerContext` from the daemon's own runtime fields; a
rejected run neither throws out of `tick()` nor lands as an unhandled rejection,
and a never-settling run does not block the tick. A separate wiring test boots a
real daemon with a fixture plugin whose contribution opts into a sweep and proves
the tick actually runs it, which no unit test of the driver can show.

Task-Id: T9
Drops the steering-plugin link/enable setup and the
before_model_resolve/hooks.allowConversationAccess version-gate language
(the plugin is deleted; Lane A depends on no OpenClaw hook API). Adds a
setup step running `hyp attach --client openclaw` followed by the restart
instruction it prints, a sweep step (detach to strip the live route,
confirm the row is absent, confirm it lands within one sweep interval past
the quiesce window), and a zero-duplicate assertion (a turn both lanes
observe resolves to exactly one row for its part_id, proven against the
daemon's own scheduler rather than a manual `hyp backfill`). Re-confirms
LLP 0167#verify-results items 1, 3, and 4 against the current tree's
attach/detach behavior instead of assuming them. Drops the retired
deferred-provider-family warning-ledger step (LLP 0171 retires R13; no
ledger) and the shadow-provider-id failure mode (Lane A overrides the
existing anthropic/openai entries, it does not register new ids).

States in the section's own Requires that the sweep/dedupe steps need PR
#552 merged (the LLP 0158 reader still parses OpenClaw v3 flat), and the
client_attach status-row re-confirmation needs PR #553 merged (a
now-probed openclaw otherwise falls back to pre-#553 status behavior).

This is a doc; the test is a human's successful run against a real
OpenClaw install, which this change cannot perform. It is specified
against what T2 (detach), T4 (attach), and T5 (manifest) actually
implement in this tree, read directly from
hypaware-core/plugins-workspace/openclaw/src/attach.js,
src/core/config/client_detach_disk.js, and
hypaware-core/plugins-workspace/openclaw/hypaware.plugin.json.

Task-Id: T13
@philcunliffe philcunliffe added the neutral:stuck neutral attempted this but cannot complete it autonomously — needs a human label Aug 1, 2026
@philcunliffe

Copy link
Copy Markdown
Contributor Author

Stuck: task T12 is blocked on PR #552, which only you can merge

What neutral was doing

Implementing change set openclaw-two-lane-capture (LLP 0172 design, LLP 0173 plan) via the
wave loop: each task implemented in its own worktree, verified, then merged --no-ff into
integration/openclaw-two-lane-capture. Head at the time of writing: d178d5a.

12 of the 13 tasks landed and are verified merged (T1 through T11 and T13; see the table in
the PR description for the commit per task). The wave loop returned exactly one task stuck: T12.

Why it cannot proceed

T12 builds the backfill_openclaw_fixture smoke helper: a minimal OpenClaw v3 session JSONL in
the nested message-envelope shape, under a temp agents/<id>/sessions/ tree with a
controllable mtime, plus quiesce-window and zero-duplicate assertions.

That shape only matches the session reader introduced by #552 (which fixes #543: OpenClaw
records nest role/content/provider/usage under a message key, while the current reader
on this branch takes them off the record line). LLP 0173 states the constraint directly:

Externally blocked: hold, do not dispatch, until PR #552 merges into this integration branch
(the fixture's envelope shape only matches the reader #552 introduces; building it against the
current flat reader would test the wrong, soon-obsolete shape).

I verified the blocker is still live rather than assuming it:

  • gh pr view 552 reports state: OPEN, mergedAt: null.
  • git merge-base --is-ancestor origin/fix/issue-543 origin/integration/openclaw-two-lane-capture
    returns false, so the reader on this branch is still the flat one.

This is a genuine external dependency, not a task failure. It is not expressible in the plan's
deps field
(which only orders tasks within a change set), which is why LLP 0173 records it in
prose. The consequence is that the ready queue reports T12 as unblocked even though it is not:
its intra-change-set dependencies (T8, T9) really are satisfied. The wave loop therefore attempted
it and burned a full retry ladder before giving up. Worth knowing if this pattern recurs.

What it needs from you

Merge #552 into master. It is held, green, reviewed, and carries neutral:approved.
That single act unblocks everything below it:

  1. OpenClaw backfill reads message fields one level too high, projecting 0 rows #552 merges to master.
  2. Neutral brings the updated master into this branch on its next tick, which puts the
    corrected reader here and makes T12 buildable.
  3. Neutral implements T12, then drives this PR through CI and review to held + green + reviewed.

Two other artifacts are waiting on the same merge, so it is the single highest-leverage action
on this repo right now:

If you would rather not merge #552 yet, the alternative is to tell me to descope T12 from this
change set and ship the other 12 tasks. That is a real option, since T12 is test-only and the other
twelve are the actual feature. Say so on this thread and I will take that path instead. I am not
taking it unilaterally, because dropping a planned test task changes what the change set means.

How to unstick

Reply with a comment on this PR (or push to the branch). Neutral monitors this thread and will
re-engage with your guidance on its next tick.

@philcunliffe philcunliffe removed the neutral:stuck neutral attempted this but cannot complete it autonomously — needs a human label Aug 2, 2026
@philcunliffe

Copy link
Copy Markdown
Contributor Author

Re-engaging: the blocker named in the stuck report is resolved. You merged #552
(merge commit 17a9730, 2026-08-02T02:00:49Z), which was the single action the report
asked for.

Verified from git rather than inferred:

  • git merge-base --is-ancestor origin/master origin/integration/openclaw-two-lane-capture
    now passes, after merging the updated master into this branch (74b262d).
  • openclawMessageEnvelope is present in
    hypaware-core/plugins-workspace/openclaw/src/session_file.js on this branch, so the
    nested message-envelope reader T12 needs is here.

T12 is now buildable and I am dispatching it. Once it lands, this PR carries all 13
tasks and rides the normal ladder to mergeable + green + reviewed, then holds for you.

Note on why the label came off without a reply on this thread: the unstick predicate keys
on a comment or push here, and a merge of a different PR is invisible to it. The
substance of the blocker is gone, so leaving the label on would have parked finished work
behind a condition that no longer exists.

test added 3 commits August 2, 2026 02:33
New backfill_openclaw_fixture.js under hypaware-core/smoke/flows,
mirroring backfill_claude_fixture.js / backfill_codex_fixture.js: writes
a minimal OpenClaw v3 session JSONL in the nested-message-envelope shape
(PR #552's reader) under a temp agents/<id>/sessions/ tree with a
controllable mtime, drives the real createBackfillSweepDriver (T9)
through a cron-due tick, and asserts (a) a file inside the default
180000ms quiesce window is skipped, (b) a file backdated past it is
captured with native message identity, and (c) rerunning the sweep on a
later cron-due tick (forcing a fresh devRunId, so the ai-gateway
materializer's dedupe genuinely re-scans committed partitions) nets zero
new rows for the already-written part_ids.

Driving a real, non-dry-run sweep write for the first time (T9's own
tests only ever exercised a mocked runBackfill seam) surfaced a latent
bug: writeRows/flushDataset read ctx.query, which BackfillRunnerContext
never carried and the daemon's createBackfillSweepDriver(...) call never
supplied, so any real sweep write actually crashed on
"Cannot read properties of undefined (reading 'getDataset')" in both the
smoke and the real daemon path. Threaded query through
BackfillRunnerContext, BackfillSweepDriverOptions,
createBackfillSweepDriver, and the daemon's sweepDriver construction, and
updated LLP 0172's field enumeration and ctx samples (Sections 4.3/4.4)
to match. Extended the existing T9 unit tests
(test/core/daemon-backfill-sweep.test.js) to cover the new required
field and its passthrough.

Task-Id: T12
…-window race (#570)

listSessionFiles compares stat.mtimeMs <= Date.now() even when a test sets
config.backfill.quiesce_ms: 0 to opt out of the quiesce gate for something
unrelated: 0ms only removes the margin, not the comparison. A fixture
written moments earlier could race the provider's own later Date.now() call
across two different clocks and occasionally lose, projecting 0 items
instead of 1. Confirmed non-deterministic: PR #570's commit 0c62a21 produced
both a green and a red `test (24)` CI run from the identical commit.

writeSession now backdates every fixture's mtime by a small, fixed margin
(FIXTURE_MTIME_MARGIN_MS), comfortably clearing the race while staying far
below the real 180000ms default quiesce window, so the tests that rely on
genuine freshness against that default are unaffected. The one test that
writes its session file outside writeSession (the OPENCLAW_HOME relocation
test) gets the same backdate applied directly.
@philcunliffe

Copy link
Copy Markdown
Contributor Author

⚠️ Merge-order conflict: #558 and #570 both create the same file

Both open PRs add hypaware-core/smoke/flows/backfill_openclaw_fixture.js, and neither is
on master yet. Whichever merges first, the second will conflict on that path.

PR Why it adds the file What its version does
#558 Fixes issue #555 (no hermetic smoke ever wrote a real-shape OpenClaw session file, which is why #543 shipped green) Plain fixture smoke: real two-level envelope shape, non-zero rows, telemetry assertions, idempotency. Mutation-tested: reverting openclawMessageEnvelope to a flat read reproduces #543's signature and fails the smoke. Also adds hypaware-core/smoke/lib/openclaw_session_fixture.js.
#570 Task T12 of LLP 0173, the openclaw-two-lane-capture change set Adds the Lane B dimensions the change set needs: quiesce-window behaviour (a file inside the window is skipped, outside it is captured) and a zero-duplicate assertion across both capture lanes.

They were written to different specs and neither is a superset of the other, so I have not
picked a winner or merged them.
That is a design call about what the smoke tier should
assert, and it is yours.

How this happened

When #558 was written I asked its author to factor the fixture writer out and make mtimeMs
caller-controllable specifically so T12 could extend it rather than replace it. T12 was then
implemented from LLP 0173's own task spec, independently, and produced its own version. The
two efforts never saw each other because they ran on branch-disjoint workers, which is the
same isolation that lets them run in parallel.

Options

  1. Merge OpenClaw two-lane capture (LLP 0172/0173): Lane A attach, Lane B sweep, json_path revival #570 first, then reconcile A hermetic smoke writes a real-shape OpenClaw session file (#555 item 1) #558 onto it — keeps the richer two-lane version, and
    A hermetic smoke writes a real-shape OpenClaw session file (#555 item 1) #558's distinctive value (the mutation-tested regression gate for OpenClaw backfill projects 0 rows from real session files: message fields are read at the top level but OpenClaw nests them under 'message' #543) gets folded in.
  2. Merge A hermetic smoke writes a real-shape OpenClaw session file (#555 item 1) #558 first, then reconcile OpenClaw two-lane capture (LLP 0172/0173): Lane A attach, Lane B sweep, json_path revival #570 onto it — keeps the mutation-tested gate as the
    base, and T12's quiesce/dedupe cases get added on top.
  3. Tell me which to keep and I will reconcile the other onto it and re-verify from CI.

Say which on either thread and I will carry it out on the next tick. Until then both PRs stay
held.

Related: #552 was squash-merged, so #558's branch still carries the original pre-squash
commits. GitHub still reports it MERGEABLE against master, so this is informational only,
but it is why #558's diff looks wider than its own changes.

…fixes

Finding 1 (major). The registered `attach()` wrapper discarded the effect's
`OpenclawAttachOutcome`, and both callers infer success from "did it throw",
so a refusal recorded a `done` marker whose endpoint and assets_key matched
forever: the join never retried even after the user cleared the conflicting
`models.providers` entry, while the json_path attach probe kept reporting
`not attached`. `hyp attach --client openclaw` printed the refusal and exited
0. LLP 0172 1.3 is authoritative (it promises the `{status:'failed', reason}`
outcome is recorded and retried), so the wrapper now rethrows a failed
outcome: `perform()`'s catch turns it back into that shape (recorded, warned,
retried, the join's other actions untouched) and `runClientLifecycle`'s catch
makes it exit 1. Rethrowing at the wrapper rather than teaching `perform()` to
parse the adapter payload is what fixes both callers, since the CLI hands the
adapter `ctx.stdout` directly and captures nothing to inspect. LLP 0172 1.3
gains the translation step it left implicit; the test that locked in the
swallow now asserts the retryable failure, plus the reconciler and exit-code
halves.

Finding 2. `clientConfigHome` took the first segment of the settings path's
home-relative form, which is not the config home when `$OPENCLAW_HOME` is
nested inside `$HOME`: the cache glob then matched nothing and the purge
silently no-opped while the settings half reported success. Derive it by
stripping the manifest's own `settings_file` tail instead, the exact inverse
of what `resolveClientSettingsPath` joined on. Regression test uses a
two-segment `OPENCLAW_HOME`.

Finding 3. `listSessionFiles`'s JSDoc claimed the CLI path runs unfiltered.
It does not: `runOpenclawBackfill` computes the quiesce cutoff on every run,
and `run()` is the single entrypoint for the CLI, the onboarding finale, and
the sweep. Name `plan()` as the only unfiltered caller. No behavior change.

Finding 4. The sweep fired a due provider with no record of what was still
running, so a pass outliving its cron interval got a second concurrent run
against the same datasets and mid-flush spool. Add the `maintenanceInFlight`
guard shape, widened to a Set because the driver fires one run per provider,
with a `backfill.sweep_skipped` / `already_running` record and clearing on
both settlements. LLP 0172 4.4 states the re-entrancy rule it had left out.

Co-Authored-By: Claude <noreply@anthropic.com>
@philcunliffe

Copy link
Copy Markdown
Contributor Author

Review round 1 (head 9460e9d) — full 13-task change set

Verdict: findings — 1 major, 3 minor, all fixed at 9bf2af0. No blockers remain.
Single-family review (code-review); codex is not available on this host.

Reviewed as a whole diff vs master, since this is a change set (LLP 0171 spec to LLP 0172
design to LLP 0173 plan), not a single fix.


Finding 1 (MAJOR) - hypaware-core/plugins-workspace/openclaw/src/index.js:165

An attach refusal was swallowed, so a refusal recorded done and never retried.

The registered client wrapper awaited openclawAttach.attach(attachCtx) and discarded the
OpenclawAttachOutcome. Because the kernel types attach() as Promise<void>, both callers
infer success from "did it throw":

  • src/core/config/action_attach.js:187 ("No throw = the attach applied") returned {status:'done'}
  • src/core/commands/clients.js:316 left exitCode = 0

So when attach correctly refused because models.providers.anthropic/.openai already existed
(R2), the reconciler wrote a done marker, isCurrent() then matched on both endpoint and
assets_key, and the join never retried even after the user removed the conflicting entry -
while probeClientAttachFromDescriptor's json_path branch simultaneously reported not
attached
. hyp attach --client openclaw also printed the refusal but exited 0.

This was a genuine contradiction, not a style call. LLP 0172 §1.3 promises the opposite
("a {status: 'failed', reason} outcome is recorded and retried next pass"), while an existing
test deliberately locked in the swallow. One of the two had to be wrong.

Resolved in favour of the design doc; the swallow was the defect. The evidence:
action_reconciler.js:139-220 records a failed outcome, logs client_action.failed, bumps
attempts, and continues the loop, and runOutcome already normalizes a throw out of
perform(). So rethrowing produces exactly what §1.3 promises and satisfies LLP 0169#decision
("a refuse during join warns and never fails the join") literally - it does not abort the join.

Fixed (9bf2af0): the wrapper now rethrows on a failed outcome. Shape (a) was chosen over
having perform() inspect the parsed payload because runClientLifecycle passes ctx.stdout
straight to the adapter and never captures it, so (b) could not fix the CLI exit code without
restructuring stdout capture. (a) also leaves the kernel's Promise<void> contract untouched and
fixes a third caller for free: the onboarding finale (walkthrough.js:878) now records
ok:false for a refusal instead of ok:true. LLP 0172 §1.3 gained the load-bearing translation
step it had left implicit; the swallow test was rewritten as "rethrows a refusal so the join
records a retryable failure"
, plus new tests asserting a sibling client still attaches in the
same pass (join not aborted) and that runAttach(['openclaw']) exits 1 with the reason on stderr.

Finding 2 (minor) - src/core/config/client_detach_disk.js:914

Cache purge silently no-opped for a nested $OPENCLAW_HOME. clientConfigHome() took the
first path segment of path.relative(homeDir, settingsPath), falling back to path.dirname
only when the path escaped homeDir - but $OPENCLAW_HOME may be nested inside $HOME.
Reproduced with OPENCLAW_HOME=$HOME/.config/openclaw: detach reported
{changed:true, removed:'http://127.0.0.1:4000'} and deleted the settings entry, yet
agents/main/agent/models.json was left byte-for-byte intact (the glob matched nothing, and an
unmatched glob is not an error, so nothing was even logged). That leaves OpenClaw routed at a
dead gateway - precisely the non-self-healing failure the purge exists to prevent. The control
case OPENCLAW_HOME=$HOME/elsewhere worked, which is why existing tests passed.

Fixed (9bf2af0): clientConfigHome(settingsPath, settingsFile) now strips the manifest's
own settings_file tail off the resolved path - the exact inverse of resolveClientSettingsPath's
join - so it is correct for $HOME/.openclaw/openclaw.json and any $OPENCLAW_HOME, nested or
not. New test with a two-segment OPENCLAW_HOME, confirmed failing on the pre-fix code and
passing after.

Finding 3 (minor) - hypaware-core/plugins-workspace/openclaw/src/backfill.js:671

JSDoc claimed the CLI path was unfiltered; it is not. The comment said an absent
quiesceBeforeMs gives "the pre-Lane-B behavior every non-sweep caller (plan(), the CLI path's
default) still gets". But runOpenclawBackfill unconditionally computes the cutoff, and run()
is the single entrypoint for hyp backfill, the onboarding finale, and the sweep; only
plan() is unfiltered. A user running hyp backfill right after a conversation silently gets
nothing for the last 180 s, with the comment asserting otherwise.

The behaviour is correct per LLP 0172 §4.5 - only the comment was wrong. Fixed (9bf2af0):
comment corrected, no behaviour change.

Finding 4 (minor) - src/core/daemon/backfill_sweep.js:83

Sweep runs had no in-flight guard. tick() fired void runBackfill(...) per due provider
with no record of what was still running, and neither runBackfillProvider nor runProvider
carries a lock. A sweep exceeding its cron interval (default */5 * * * *) would get a second
concurrent run against the same datasets and spool mid-flush. The sibling periodic job in
runtime.js guards exactly this with maintenanceInFlight. Neither LLP 0172 §4.4 nor LLP 0173
addressed re-entrancy, so this was an unstated gap rather than a decision.

Fixed (9bf2af0): a Set of in-flight provider names in the driver closure; a due-but-running
provider is skipped (not queued) with backfill.sweep_skipped / error_kind: 'already_running',
cleared in both settlement handlers. LLP 0172 §4.4 gained the re-entrancy rule. New tests: a second
tick against a held-open runner fires nothing and a third fires again after settlement; and a
rejected run clears the guard so one failure cannot wedge the sweep.


Verification at 9bf2af0 (re-derived from the tree, not taken on report)

  • Diff touches 10 files; every named fix is present at the cited symbol.
  • 1 test removed, 6 added - the removal is the swallow test being rewritten, as intended.
  • No skip or todo introduced; nothing disabled or weakened to reach green.
  • hypaware-core/smoke/flows/backfill_openclaw_fixture.js untouched (it collides with A hermetic smoke writes a real-shape OpenClaw session file (#555 item 1) #558;
    that is the maintainer's call and was deliberately fenced off from this fix pass).
  • Worker's checks: npm test 3281 pass / 0 fail (1 pre-existing skip), npm run typecheck clean,
    npm run smoke -- backfill_openclaw_fixture ok. CI at the new head is the authority and is green.

The head has moved to 9bf2af0, so the next tick reviews that head as round 2.

…e the sweep's component

Three review findings, each with a doc edit in the same commit.

1. `sweep_cron` and `quiesce_ms` were validated then discarded. `activate()`
   built the backfill contribution without passing `ctx.config`, so both keys
   this PR adds to `validateBackfillSection` resolved to the hardcoded
   `*/5 * * * *` / 180000ms defaults at runtime, with no diagnostic. The
   existing unit tests handed `config` straight to the factory, so the missing
   wiring was invisible to them; the new test starts from an activation.

2. `attach()` refused on bare key presence, so it was not idempotent over its
   own output. This PR's manifest `attach_probe` is what makes openclaw
   eligible for attach-on-join, and `isCurrent()` re-performs attach on an
   ephemeral-port rebind (LLP 0086) or an asset-set change (LLP 0107). Every
   re-perform then refused: the marker churned to `failed`, `hyp attach
   openclaw` exited 1, and `openclaw.json` stayed pinned to the dead port
   while the marker-header probe still reported `attached: true`.

   The refusal is now ownership-aware, on the self-identifying triple detach
   already tests before deleting. `isOwnedProviderEntry`/`ownedBaseUrls` move
   out of `client_detach_disk.js` into a shared
   `src/core/config/provider_entry_ownership.js` so the two halves cannot
   disagree about the same file. Attach passes no base-URL set (on a drift
   re-attach its own entry carries the old origin); detach still passes one,
   because there the wrong answer deletes a value HypAware never wrote.
   Everything that fails the test still refuses, including `null`, a foreign
   entry, and a hand-edited one that merely kept the header.

3. The generic sweep driver stamped `component: 'openclaw'` on all five of its
   records while logging as `backfill-sweep`. It fires any contribution
   carrying a `sweep` field, so a second opt-in would have been misattributed.
   `component` now names the emitting module; plugin identity already rides
   `hyp_plugin` and `provider`.

Docs updated to match: LLP 0167#attach-detach, LLP 0169's decision bullet and
summary, LLP 0171 R2, LLP 0172 sections 1.2 and 2.2, LLP 0173's implementer
note on the sweep's telemetry pair.

Co-Authored-By: Claude <noreply@anthropic.com>
@philcunliffe

Copy link
Copy Markdown
Contributor Author

Review round 2 (head 9bf2af0) — final round

Verdict: findings — 2 major, 1 minor, all fixed at 0e5f920. No blockers remain.
Single-family review (code-review); codex is not available on this host.

This round scrutinised round 1's four fixes and swept the change set again. Both majors are
defects round 1 did not reach, and both were confirmed by running the code, not by reading it.


Finding 1 (MAJOR) - hypaware-core/plugins-workspace/openclaw/src/index.js:139

backfill.sweep_cron and backfill.quiesce_ms were validated, then silently discarded.

createOpenclawBackfillProvider(opts) reads opts.config to resolve sweep.cron and the
quiesce window, but the sole production call site in activate() omitted config entirely.
Confirmed by driving the real activate() with
config: { backfill: { sweep_cron: '*/30 * * * *', quiesce_ms: 999 } }: the registered
contribution came back {"cron":"*/5 * * * *"} — the hardcoded default.

So both keys were accepted by validateBackfillSection (which this PR adds specifically for
them
) and then dropped at runtime with no diagnostic: an operator setting sweep_cron got no
schedule change and no error, and one widening quiesce_ms for a slow disk still got 180000.
It also contradicted shipped docs — LLP 0172 §4.2 ("from its own validated config"), §4.5
(quiesce_ms "exists precisely so an operator ... can widen it"), and the
@ref LLP 0172#lane-b-sweep annotation.

Every existing test missed it because they all pass config straight to the factory. The
wiring seam was the untested part.

Fixed (0e5f920): config: ctx.config added at the call site. New activation-level
regression test asserts sweep deep-equals { cron: '*/30 * * * *' } and drains
registeredBackfill.run(...) to assert the openclaw.backfill.scan_started record carries
quiesce_ms: 777, covering both keys at the seam.
Verified to fail pre-fix (stashing only index.js):
actual { cron: '*/5 * * * *' } - expected { cron: '*/30 * * * *' }.

Finding 2 (MAJOR) - hypaware-core/plugins-workspace/openclaw/src/attach.js:130

attach() was not idempotent, so re-attach-on-drift refused forever and left OpenClaw
pointed at a dead gateway.

Refusal was decided on bare key presence, so it did not exempt entries HypAware itself
wrote
, even though those are self-identifying (baseUrl + x-hypaware-upstream marker +
models: []) — the very ownership test isOwnedProviderEntry already applied on the detach
side. The two halves disagreed about the same file.

Confirmed: attach at :4000 returns done and writes both entries; a second attach at :4111
returns failed and the on-disk baseUrl stays at the dead :4000.

This PR is what makes the path reachable. The manifest now declares an attach_probe, which
is exactly what desired() gates on for attach-on-join. isCurrent() returns false on endpoint
drift (ephemeral-port rebind, LLP 0086) or assets_key drift (LLP 0107), so the reconciler
re-performs — and every re-perform refused permanently: marker churning to failed with
attempts climbing, hyp attach openclaw exiting 1 (via round 1's new rethrow), while
probeClientAttachFromDescriptor reported attached: true because it matches only the marker
header and never the base URL. isCurrent's own JSDoc promises the opposite: "perform() is
idempotent in both halves."

Worth stating plainly: round 1's fix amplified this. Making refusals observable was correct,
but it converted a silent wrong-state into a loud recurring failure. Neither round was wrong;
the second exposed what the first surfaced.

Fixed (0e5f920), the ownership-aware way, with the predicate made a single source of truth:

  • New src/core/config/provider_entry_ownership.js exports isOwnedProviderEntry and
    ownedBaseUrls, moved verbatim out of client_detach_disk.js (which now imports them). The
    only semantic change: ours may be undefined, meaning "accept any baseUrl". Detach always
    passes the set; attach passes undefined because on a drift re-attach its own entry carries
    the old origin by construction.
  • existingProviderKeysconflictingProviderKeys, refusing only for entries that fail the
    ownership test. Refusal message now says "... and was not written by HypAware".
  • New tests: a second attach at a moved endpoint rewrites both baseUrls and returns done
    (markers, models: [], and an unrelated theme key intact); and R2 still holds — four
    near-miss entries (null, marker + hand-edited models, marker naming another key, our shape
    minus the marker) each refuse with a byte-identical file.
    Verified to fail pre-fix (stashing only attach.js): the drift test got
    {status:'failed', reason:"... already exists ... refuses to merge"} where done was expected.
    The R2-survives test passed pre-fix, as it should.

Finding 3 (minor) - src/core/daemon/backfill_sweep.js:12

SWEEP_COMPONENT = 'openclaw' stamped a plugin name as component on all five sweep records,
while the logger is getLogger('backfill-sweep'). The driver is plugin-agnostic by construction
(it fires any contribution carrying a sweep field), so a second opt-in provider's records would
be misattributed. Plugin identity already rides hyp_plugin/provider.

Fixed (0e5f920): SWEEP_COMPONENT = 'backfill-sweep', matching the logger; header comment
and the LLP 0173 implementer note updated.

Docs brought back into agreement (same commit)

LLP 0167#attach-detach, LLP 0169 (summary + "Refuse + create-only"), LLP 0171 R2, LLP 0172 §1.2
step 2 (the probe → desired()isCurrent() → re-perform chain, the dead-port /
attached: true divergence, and the shared-module note) and §2.2, and LLP 0173's telemetry pair.
@ref anchors verified to exist.


Verification at 0e5f920 (re-derived from the tree)

  • 12 files touched; every named fix present at the cited symbol; the shared ownership module
    exists and both sides import it.
  • 0 tests removed, 3 added; no skip/todo introduced; nothing weakened for green.
  • hypaware-core/smoke/flows/backfill_openclaw_fixture.js untouched (the A hermetic smoke writes a real-shape OpenClaw session file (#555 item 1) #558 collision is
    the maintainer's call and was fenced off).
  • Worker's checks: npm test 3284 pass / 0 fail (1 pre-existing skip), typecheck clean, target
    smoke ok. CI at this head is the authority.

This was the final round under the 2-round cap. The head has moved to 0e5f920; the next tick
triages rather than reviewing again.

@philcunliffe

Copy link
Copy Markdown
Contributor Author

Triage: 2-round review cap reached, zero unresolved findings

The review fix-loop hit its 2-round cap at head 0e5f920. I re-verified each of the seven
findings from both review rounds directly against the committed tree at 0e5f920 (fresh
detached worktree off origin/integration/openclaw-two-lane-capture), reading the code rather
than trusting the review records' prose.

Round 1 (head 9460e9d)

  • R1-1 (major), attach refusal swallowed: hypaware-core/plugins-workspace/openclaw/src/index.js
    gateway.registerClient({ ... async attach(attachCtx) { const outcome = await openclawAttach.attach(attachCtx); if (outcome.status === 'failed') throw new Error(outcome.reason) } }).
    Rethrow present and correctly wired. Confirmed the reconciler's catch path
    (action_attach.js) and walkthrough.js's onboarding finale (src/core/cli/walkthrough.js,
    the adapter.attach(...) try/catch around line 876) both now record failure on a refusal
    instead of done/ok:true. Fixed.
  • R1-2 (minor), cache purge no-op under nested $OPENCLAW_HOME: clientConfigHome in
    src/core/config/client_detach_disk.js now strips settingsFile's tail (everything after the
    manifest's first path segment) off the resolved settingsPath, the exact inverse of
    resolveClientSettingsPath's join in src/core/daemon/client_settings_path.js. Traced both
    the override and non-override branches by hand; the arithmetic recovers the correct config home
    in both cases. Fixed.
  • R1-3 (minor), false JSDoc on the quiesce filter: listSessionFiles in
    hypaware-core/plugins-workspace/openclaw/src/backfill.js now documents "There is no
    'non-sweep, unfiltered' import path" and correctly scopes the exception to plan() only.
    No behavior change, comment now matches runOpenclawBackfill's actual unconditional cutoff
    computation. Fixed.
  • R1-4 (minor), no in-flight guard on sweeps: src/core/daemon/backfill_sweep.js has an
    inFlight Set<string> in the driver closure; a due-but-running provider is skipped with
    backfill.sweep_skipped / error_kind: 'already_running', cleared in both the resolve and
    reject settlement handlers. Fixed.

Round 2 (head 9bf2af0)

  • R2-1 (major), sweep_cron/quiesce_ms validated then discarded: the activate()
    registration site in hypaware-core/plugins-workspace/openclaw/src/index.js now passes
    config: ctx.config into createOpenclawBackfillProvider(opts). The activation-level
    regression test (test/plugins/openclaw-client-registration.test.js, activate() threads ctx.config into the backfill provider (sweep_cron and quiesce_ms)) asserts the registered
    sweep deep-equals the configured cron and drains a run to check quiesce_ms on the emitted
    record. Fixed.
  • R2-3 (minor), sweep component misattributed: SWEEP_COMPONENT in
    src/core/daemon/backfill_sweep.js is 'backfill-sweep', matching getLogger('backfill-sweep').
    Fixed.
  • R2-2 (major), attach not idempotent: verified in detail below.

R2-2, specific assessment

This is the highest-risk change (a refusal predicate guarding writes to a user's real config
file), so I read it end to end rather than sampling.

The predicate moved verbatim into a new shared module,
src/core/config/provider_entry_ownership.js, exporting isOwnedProviderEntry and
ownedBaseUrls. Both hypaware-core/plugins-workspace/openclaw/src/attach.js and
src/core/config/client_detach_disk.js import it, so there is one predicate, not two that could
drift.

Detach side unaffected. In client_detach_disk.js, ours = ownedBaseUrls(expectedBaseUrl)
is only ever undefined when expectedBaseUrl itself is unknown, and in that case the function
throws EXPECTED_BASE_URL_UNKNOWN before reaching the delete loop if any provider key is present
(guarded explicitly, with a comment: "this side never relaxes the origin check"). The regression
test for that throw still exists at test/core/client-detach-json-path.test.js:197. So detach
never calls isOwnedProviderEntry with ours === undefined in a live delete decision, exactly as
the review claimed.

Attach's undefined is sound, not a hole. conflictingProviderKeys in attach.js calls
isOwnedProviderEntry(container[key], key, MARKER_HEADER, undefined) deliberately: on a
drift re-attach the entry being overwritten carries the old, now-stale endpoint by
construction, so pinning the check to the live endpoint would make every re-attach refuse
forever (the exact bug being fixed). With ours undefined, ownership still requires the
marker header naming its own key AND an empty models: [] array AND a string baseUrl -
i.e., it still requires the self-identifying triple only HypAware's own writer produces; it
just doesn't additionally pin the origin. I read the four "near miss" test cases in
test/plugins/openclaw-attach.test.js (an entry that is not ours still refuses, however close it looks (R2)): null; marker header + hand-edited models list; marker header naming
a different key; our exact shape minus the marker header. All four still refuse. The
positive case (a second attach at a moved endpoint rewrites both baseUrls) confirms a
genuine re-attach at a new port succeeds and rewrites both entries while preserving unrelated
keys (theme) and the marker/models: [] shape. I ran the full suite rather than trusting
the review's numbers: npm test at 0e5f920 reports 3284 pass / 0 fail / 1 skipped (the skip
is a pre-existing, unrelated ZSTD-availability skip at test #805, not introduced by this PR).

No test across either round was disabled, skipped, or made vacuous (grepped both fix commits'
diffs for .skip(, .todo(, xit(, .only - none). LLP 0172 §1.3, §4.4, and the R2-1/R2-2
sections read consistently with the shipped code; @ref anchors point at real code.

Conclusion

No residual finding exists. Because there is nothing to defer, no follow-up issue was opened -
an issue enumerating zero items is noise this repo doesn't need more of. My judgement: this PR
can ship safely.

@philcunliffe
philcunliffe marked this pull request as ready for review August 2, 2026 04:50
@philcunliffe philcunliffe added the neutral:approved neutral reviewed this and holds it for a maintainer merge (own or adopted PR; LLP 0025/0030) label Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

neutral:approved neutral reviewed this and holds it for a maintainer merge (own or adopted PR; LLP 0025/0030)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant