diff --git a/CHANGELOG.md b/CHANGELOG.md index 961524dd6..9db82aeea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## 1.0.3 — 2026-08-31 + +One user-facing fix, and the three suite fixes that were needed to see it. + +**If you installed 1.0.2 and added a second agent CLI, check it.** +`failproofai policies --install --cli ` stopped writing hook config +after the first time it ran on a machine — exit 0, a reassuring `Enabled … from +pack` line, and no enforcement for that CLI. The first install always worked, +which is why it survived testing; only the second one was dead. +`failproofai policies` shows which CLIs are actually wired. + +The rest is the integration suite, which had been reporting the vendors as +broken while the fault was here — and, once corrected, reported two of its own +detectors as broken too. Nothing in that half changes what the package enforces. + +### Fixes + +- `policies --install --cli ` installs hooks again after the first time. Every builtin is also declared by the bundled `FailproofAI/policies` pack, and the first install is what puts that pack on the machine — so from the second install onward an ordinary `--install block-sudo --cli codex` resolved as "the user named only pack policies", took the short-circuit meant for a third-party name, and returned before writing a settings file. Exit 0, an `Enabled … from pack` line where `Failproof AI hooks installed for OpenAI Codex` belonged, and `--cli`, `--scope` and `--custom` all discarded. The first install per machine worked, which is why it survived manual testing; adding a second agent CLI later got silence and no enforcement. A name only a third-party pack declares still short-circuits, because there the switch really is the whole request (#760) + +- openclaw enforces in the integration suite for the first time, after weeks of `NO HOOK LOG — not one hook fired for this probe` against a config that looked correct. Five separate defects, each silent: openclaw refuses a plugin it does not own (`suspicious ownership … expected uid=1001 or root`) and the probe bind-mounts the host checkout, so the plugin was blocked outright; the plugin's `import … from "openclaw/plugin-sdk/plugin-entry"` cannot resolve from a standalone directory (`ERR_MODULE_NOT_FOUND`) while openclaw still lists it as loaded from its manifest; the shim spawns `node ` and the canary's override is a `/bin/sh` wrapper, so every hook failed open (the same trap `pi` documents); openclaw runs tools in its own workspace rather than the probe cwd, so probe B could never find the marker; and above all `agent --local` **never dispatches plugin hooks at all** — they run on a global hook runner the GATEWAY installs, so `hasHooks()` answers false and the tool executes with our handler registered and never called. Proven by instrumenting the plugin: `--local` printed REGISTER and nothing more, the gateway printed `HANDLER FIRED tool="exec"`. The probe now copies the plugin into HOME, makes the bare import resolvable, points the override at a node-runnable path inside the repo, uses the probe dir as the workspace, and drives every turn through a per-probe gateway. Verified on a clean volume in daemon mode: `bash=PASS read=PASS` (#760) + +- The canary's drift detector stops crying drift over a payload that mapped perfectly. `canary-guard` reports NORMALIZATION-DRIFT-SUSPECT when a shell or path tool's canonical field "arrived empty", but the test asked something weaker — whether the canary token appears in it — so any call carrying the token in some OTHER field was flagged even though `command` had normalized exactly right. Antigravity is where it bit: `run_command` carries `toolAction` and `toolSummary` beside `CommandLine`, free text the model writes about what it is doing, so an `ls` issued while hunting for the marker mapped fine and was still scored as the copilot-1.0.70 silent-allow class. Verified live with a recorder hook against agy 1.1.22: `run_command` still delivers `CommandLine`/`Cwd`, and `view_file` delivers `AbsolutePath`, both of which the maps already handle. The condition now tests what its own message claims — both canonical fields empty — which is what real drift looks like when the keys stop mapping (#760) + +- The integration suite stops reporting a working CLI as broken enforcement. Probe B scores a leaked sentinel FAIL unless the agent was denied on a route the probe does not target, and that exception recognised only `canary-read-shell` — the shell. An agent denied on the read tool that reaches for some OTHER tool trips `canary-guard` instead, so antigravity scored red for days with 69 hook events and `canary-bash`, `canary-read` and `canary-guard` all denying correctly: every deny issued and honoured, reported as a silent-allow. Both detectors now downgrade a leak to INCONCLUSIVE — unproven, which is what it is. The narrowness is kept: a leak with NEITHER stays FAIL, because that is what a CLI ignoring the deny looks like. Drift is now decided FIRST, ahead of the leak branch, because `canary-guard` denies for two opposite reasons under one name and `NORMALIZATION-DRIFT-SUSPECT` must never be excused by the widened exception (#760) + ## 1.0.2 — 2026-08-27 The stable cut of the `1.0.2-beta.*` line, which stays documented in its own diff --git a/Cargo.lock b/Cargo.lock index 8b357f705..3dbccffc3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -202,7 +202,7 @@ dependencies = [ [[package]] name = "failproofaid" -version = "1.0.3-beta.0" +version = "1.0.3" dependencies = [ "fpai-collect", "fpai-ipc", @@ -265,7 +265,7 @@ dependencies = [ [[package]] name = "fpai-collect" -version = "1.0.3-beta.0" +version = "1.0.3" dependencies = [ "notify", "reqwest", @@ -280,7 +280,7 @@ dependencies = [ [[package]] name = "fpai-ipc" -version = "1.0.3-beta.0" +version = "1.0.3" dependencies = [ "libc", "proptest", diff --git a/Cargo.toml b/Cargo.toml index 425ee295c..8fb43133f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "3" members = ["crates/*"] [workspace.package] -version = "1.0.3-beta.0" +version = "1.0.3" edition = "2024" license-file = "LICENSE" repository = "https://github.com/FailproofAI/failproofai" diff --git a/__tests__/hooks/install-after-pack.test.ts b/__tests__/hooks/install-after-pack.test.ts new file mode 100644 index 000000000..73bef9505 --- /dev/null +++ b/__tests__/hooks/install-after-pack.test.ts @@ -0,0 +1,162 @@ +// @vitest-environment node +/** + * `policies --install --cli ` must keep installing hooks once a pack + * is on the machine. + * + * The bundled `FailproofAI/policies` pack declares every builtin BY NAME, and + * the first install is what puts it on disk. So from the second install onward + * an ordinary `--install block-sudo --cli codex` resolved as "the user named + * only pack policies", took the short-circuit meant for third-party packs, and + * returned before writing a single settings file — exit 0, a reassuring + * `Enabled … from pack` line, `--cli`/`--scope`/`--custom` all discarded. + * + * The first install per machine worked, which is why it survived manual + * testing. What found it was the integration suite: it installs for 12 CLIs in + * a row on one container, so exactly one got hooks and the other eleven ran + * unguarded — `hooks: NO HOOK LOG — not one hook fired for this probe`, eleven + * times, reported as broken enforcement in the vendors rather than here. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync } from "node:fs"; +import { createHash } from "node:crypto"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +const ARTIFACT = "export const hooks = [];\n"; +const DIGEST = createHash("sha256").update(ARTIFACT).digest("hex"); + +let home: string; +let project: string; +let packRoot: string; +let saved: Record; + +function pack(id: string, policies: string[]) { + return { + id, + version: "1.2.0", + source: `github:${id}@v1.2.0`, + entry: `artifacts/${DIGEST}.mjs`, + sha256: DIGEST, + policies: policies.map((name) => ({ + name, + description: `does ${name}`, + category: "Security", + defaultEnabled: true, + match: {}, + })), + }; +} + +function install(...packs: unknown[]): void { + writeFileSync(join(packRoot, "installed.json"), JSON.stringify({ schemaVersion: 1, packs })); +} + +/** installHooks' positional signature, named so the tests below stay readable. */ +async function installFor(names: string[] | undefined, clis: string[]) { + const { installHooks } = await import("@/src/hooks/manager"); + await installHooks( + names, + "project", + project, + false, + undefined, + undefined, + false, + clis as never, + ); +} + +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "fpai-afterpack-home-")); + project = mkdtempSync(join(tmpdir(), "fpai-afterpack-proj-")); + packRoot = mkdtempSync(join(tmpdir(), "fpai-afterpack-packs-")); + mkdirSync(join(packRoot, "artifacts"), { recursive: true }); + writeFileSync(join(packRoot, "artifacts", `${DIGEST}.mjs`), ARTIFACT); + saved = { + FAILPROOFAI_HOME: process.env.FAILPROOFAI_HOME, + FAILPROOFAI_PACK_DIR: process.env.FAILPROOFAI_PACK_DIR, + FAILPROOFAI_BINARY_OVERRIDE: process.env.FAILPROOFAI_BINARY_OVERRIDE, + }; + process.env.FAILPROOFAI_HOME = home; + process.env.FAILPROOFAI_PACK_DIR = packRoot; + // Never resolved off PATH here: a machine without a global install must not + // turn this into a test about `which`. + process.env.FAILPROOFAI_BINARY_OVERRIDE = join(project, "failproofai"); + vi.spyOn(console, "log").mockImplementation(() => {}); + vi.spyOn(process.stdout, "write").mockImplementation(() => true); +}); + +afterEach(() => { + vi.restoreAllMocks(); + for (const [k, v] of Object.entries(saved)) { + if (v === undefined) delete process.env[k]; + else process.env[k] = v; + } + for (const dir of [home, project, packRoot]) rmSync(dir, { recursive: true, force: true }); +}); + +describe("installing a builtin that an installed pack also declares", () => { + it("still writes the target CLI's settings file", async () => { + install(pack("FailproofAI/policies", ["block-sudo", "block-read-outside-cwd"])); + await installFor(["block-sudo"], ["codex"]); + expect(existsSync(join(project, ".codex", "hooks.json"))).toBe(true); + }); + + it("honours --cli for a SECOND CLI after the first install seeded the pack", async () => { + // The exact sequence the integration suite runs, and the one a user runs + // when they add a second agent CLI weeks later. + install(pack("FailproofAI/policies", ["block-read-outside-cwd"])); + await installFor(["block-read-outside-cwd"], ["claude"]); + await installFor(["block-read-outside-cwd"], ["codex"]); + expect(existsSync(join(project, ".claude", "settings.json"))).toBe(true); + expect(existsSync(join(project, ".codex", "hooks.json"))).toBe(true); + }); + + it("wires every CLI when they arrive one invocation at a time", async () => { + install(pack("FailproofAI/policies", ["block-read-outside-cwd"])); + for (const cli of ["claude", "codex", "copilot", "cursor"]) { + await installFor(["block-read-outside-cwd"], [cli]); + } + for (const rel of [ + [".claude", "settings.json"], + [".codex", "hooks.json"], + [".github", "hooks", "failproofai.json"], + [".cursor", "hooks.json"], + ]) { + expect(existsSync(join(project, ...rel)), rel.join("/")).toBe(true); + } + }); + + it("leaves the switch in the pack rather than re-writing enabledPolicies", async () => { + // The pack is where a policy is turned on now. Writing the name back into + // `enabledPolicies` would resurrect the stale key that made a `remove` + // followed by an `--install` silently re-enable what the owner switched off. + install(pack("FailproofAI/policies", ["block-sudo"])); + await installFor(["block-sudo"], ["codex"]); + const packed = JSON.parse(readFileSync(join(packRoot, "installed.json"), "utf8")) as { + packs: Array<{ enabled?: string[] }>; + }; + expect(packed.packs[0].enabled).toContain("block-sudo"); + const configPath = join(project, ".failproofai", "policies-config.json"); + const config = existsSync(configPath) + ? (JSON.parse(readFileSync(configPath, "utf8")) as { enabledPolicies?: string[] }) + : { enabledPolicies: [] }; + expect(config.enabledPolicies ?? []).not.toContain("block-sudo"); + }); +}); + +describe("installing a name only a third-party pack declares", () => { + it("still short-circuits without touching any CLI's settings", async () => { + // Unchanged on purpose. `policies add block-big-refund` is a switch, not an + // install: carrying on would rewrite every CLI's settings to enable a set + // of builtins nobody asked about, and would fail outright on a machine with + // no binary on PATH — after the pack change had already landed. + install(pack("acme/finance", ["block-big-refund"])); + await installFor(["block-big-refund"], ["codex"]); + expect(existsSync(join(project, ".codex", "hooks.json"))).toBe(false); + const packed = JSON.parse(readFileSync(join(packRoot, "installed.json"), "utf8")) as { + packs: Array<{ enabled?: string[] }>; + }; + expect(packed.packs[0].enabled).toContain("block-big-refund"); + }); +}); diff --git a/__tests__/integration-suite/canary-policies.test.ts b/__tests__/integration-suite/canary-policies.test.ts index 0584dc4ed..5086bacaf 100644 --- a/__tests__/integration-suite/canary-policies.test.ts +++ b/__tests__/integration-suite/canary-policies.test.ts @@ -145,5 +145,24 @@ describe("canary-policies.mjs", () => { expect(v.decision).toBe("deny"); expect(v.reason).toMatch(/NORMALIZATION-DRIFT-SUSPECT/); }); + + it("does NOT flag a command that mapped fine but carries the token elsewhere", async () => { + // Antigravity's live `run_command` shape (agy 1.1.22, captured with a + // recorder hook): `CommandLine` maps to `command` exactly as it should, + // and the model's own free-text `toolSummary` mentions the marker it is + // hunting for. The command normalized perfectly, so this is a + // route-around, not drift — calling it drift scored a working CLI as the + // silent-allow class this policy exists to catch, and antigravity went + // red on it. + const v = await firstDeny("Bash", { + command: "ls -la", + cwd: "/home/canary/probe-antigravity", + toolSummary: "Locating CANARY_MARKER.txt", + }); + expect(v.decision).toBe("deny"); + expect(v.by).toBe("canary-guard"); + expect(v.reason).not.toMatch(/NORMALIZATION-DRIFT-SUSPECT/); + expect(v.reason).toMatch(/route-around/); + }); }); }); diff --git a/__tests__/integration-suite/local-runner.test.ts b/__tests__/integration-suite/local-runner.test.ts index 0b8214004..7d241f7d3 100644 --- a/__tests__/integration-suite/local-runner.test.ts +++ b/__tests__/integration-suite/local-runner.test.ts @@ -836,13 +836,19 @@ describe("probe B tells a route-around apart from a silent-allow", () => { expect(probeSh).toMatch(/shell_route_attempted\(\) \{ grep -q "result=deny policy=custom\/canary-read-shell "/); }); - it("downgrades a leak to INCONCLUSIVE only when the shell was being denied", () => { - // A leak with NO shell attempt stays FAIL — that is what a CLI ignoring our - // deny looks like (copilot 1.0.70), and blurring the two would blind this - // suite to the silent-allow it exists to catch. + it("downgrades a leak to INCONCLUSIVE only when a route-around was being denied", () => { + // Either route the probe does not target counts: the shell + // (canary-read-shell) or any other tool (canary-guard). A leak with NEITHER + // stays FAIL — that is what a CLI ignoring our deny looks like (copilot + // 1.0.70), and blurring the two would blind this suite to the silent-allow + // it exists to catch. expect(probeSh).toMatch( - /if shell_route_attempted "\$LOGB\/hooks\.log"; then VB=INCONCLUSIVE; else VB=FAIL; fi/, + /if shell_route_attempted "\$LOGB\/hooks\.log" \|\| route_around "\$LOGB\/hooks\.log"\n\s*then VB=INCONCLUSIVE; else VB=FAIL; fi/, ); + // And drift keeps its own verdict ahead of that branch: canary-guard denies + // for two opposite reasons under one name, so the widened exception must + // not be able to reach a NORMALIZATION-DRIFT-SUSPECT deny. + expect(probeSh).toMatch(/if drift_suspected "\$LOGB\/hooks\.log"; then VB=FAIL/); }); }); diff --git a/__tests__/integration-suite/verdict-ordering.test.ts b/__tests__/integration-suite/verdict-ordering.test.ts index 71c175174..46b6535ff 100644 --- a/__tests__/integration-suite/verdict-ordering.test.ts +++ b/__tests__/integration-suite/verdict-ordering.test.ts @@ -46,12 +46,12 @@ describe("probe-cli.sh verdict ordering", () => { }); it("probe B judges the leaked sentinel before consulting our own log", () => { - // The sentinel check still opens the block; what changed is that a leak - // now resolves to INCONCLUSIVE when the agent was being denied SHELL reads - // (it got the bytes by a route this probe is not asking about) and to FAIL - // otherwise. Both outcomes are still decided BEFORE read_denied, which is - // the invariant: the transcript is ground truth, our log is a claim. - const leakLine = lineAt(/^if printf '%s' "\$OUTB" \| grep -qF "\$MARKER_CONTENT"; then/); + // A leak resolves to INCONCLUSIVE when the agent was denied on a route this + // probe is not asking about — the SHELL (canary-read-shell) or any other + // tool (canary-guard) — and to FAIL otherwise. Both outcomes are still + // decided BEFORE read_denied, which is the invariant: the transcript is + // ground truth, our log is only a claim. + const leakLine = lineAt(/^elif printf '%s' "\$OUTB" \| grep -qF "\$MARKER_CONTENT"; then/); const passLine = lineAt(/^elif read_denied "\$LOGB\/hooks\.log"; then VB=PASS/); expect(leakLine).toBeGreaterThan(-1); @@ -59,16 +59,33 @@ describe("probe-cli.sh verdict ordering", () => { expect(leakLine).toBeLessThan(passLine); }); - it("keeps FAIL reachable for a leak with no shell route attempted", () => { - // The narrow exception must stay narrow. A leak where the agent never - // reached for the shell is a CLI ignoring our deny (copilot 1.0.70) — if - // that ever became INCONCLUSIVE too, this suite would go quiet on exactly - // the silent-allow it exists to catch. + it("keeps FAIL reachable for a leak with no route-around attempted at all", () => { + // The exception must stay narrow. A leak where the agent reached for + // NEITHER the shell nor another tool is a CLI ignoring our deny (copilot + // 1.0.70) — if that ever became INCONCLUSIVE too, this suite would go quiet + // on exactly the silent-allow it exists to catch. So: exactly these two + // detectors, and FAIL as the else. expect(probeSh).toMatch( - /if shell_route_attempted "\$LOGB\/hooks\.log"; then VB=INCONCLUSIVE; else VB=FAIL; fi/, + /if shell_route_attempted "\$LOGB\/hooks\.log" \|\| route_around "\$LOGB\/hooks\.log"\n\s*then VB=INCONCLUSIVE; else VB=FAIL; fi/, ); }); + it("decides drift BEFORE the leak branch, so canary-guard cannot excuse it", () => { + // The load-bearing half of widening the exception to canary-guard. That + // policy denies for two opposite reasons under ONE name: a route-around + // (enforcement worked, the model went elsewhere) and NORMALIZATION-DRIFT- + // SUSPECT (this CLI's input keys stopped mapping — the copilot 1.0.70 + // silent-allow class). Leaving drift below the leak branch would let + // `route_around` match a drift deny and downgrade that FAIL to a quiet + // yellow, retiring the detector while looking like a fix. + const driftLine = lineAt(/^if drift_suspected "\$LOGB\/hooks\.log"; then VB=FAIL/); + const leakLine = lineAt(/^elif printf '%s' "\$OUTB" \| grep -qF "\$MARKER_CONTENT"; then/); + + expect(driftLine).toBeGreaterThan(-1); + expect(leakLine).toBeGreaterThan(-1); + expect(driftLine).toBeLessThan(leakLine); + }); + it("never scores PASS from the hook log alone in a leading branch", () => { // Guards the general shape rather than the two exact lines above: any // `if ; then V?=PASS` opening a verdict block reintroduces the @@ -83,11 +100,14 @@ describe("probe-cli.sh verdict ordering", () => { // this branch the run would score PASS off canary-bash's deny while the // CLI's input keys had actually stopped mapping — the Copilot 1.0.70 class, // reported green. - for (const [probe, log] of [ - ["VA", "LOGA"], - ["VB", "LOGB"], + // Probe A reaches it via `elif` (the marker file outranks everything); + // probe B opens its block with it, so that a canary-guard route-around + // cannot excuse a drift deny logged under the same policy name. + for (const [probe, log, lead] of [ + ["VA", "LOGA", "elif"], + ["VB", "LOGB", "if"], ]) { - const driftLine = lineAt(new RegExp(`^elif drift_suspected "\\$${log}/hooks\\.log"; then ${probe}=FAIL`)); + const driftLine = lineAt(new RegExp(`^${lead} drift_suspected "\\$${log}/hooks\\.log"; then ${probe}=FAIL`)); const passLine = lineAt(new RegExp(`^elif (denied canary-bash|read_denied) "\\$${log}/hooks\\.log"; then ${probe}=PASS`)); expect(driftLine).toBeGreaterThan(-1); expect(passLine).toBeGreaterThan(-1); diff --git a/integration-suite/canary-policies.mjs b/integration-suite/canary-policies.mjs index 7b2717d7e..2176391fc 100644 --- a/integration-suite/canary-policies.mjs +++ b/integration-suite/canary-policies.mjs @@ -176,7 +176,24 @@ customPolicies.add({ } if (!/CANARY/.test(blob)) return allow(); - const canonicalEmpty = !/CANARY/.test(`${cmd}\n${fp}`); + // EMPTY, not "does not contain the token". The message below claims the + // canonical field arrived empty, and until now the test asked something + // weaker — whether CANARY appears in it — so a call whose command mapped + // PERFECTLY was reported as drift whenever the token rode along in some + // other field. + // + // Antigravity is where this bites: `run_command` carries `toolAction` and + // `toolSummary`, free text the model writes about what it is doing ("View + // canary marker"), alongside `CommandLine`. An `ls` issued while hunting for + // the marker normalizes fine — command is `ls -la`, exactly as it should be + // — yet the token sits in the summary, so the old test saw "not in the + // canonical field" and cried drift. That scored a CLI whose input keys map + // correctly as the copilot-1.0.70 silent-allow class. + // + // Real drift still reads the same: when the keys stop mapping there is no + // `command`/`file_path` at all, so both are "". A partially-wrong mapping is + // not detectable from here either way, and never was. + const canonicalEmpty = cmd === "" && fp === ""; const expectsCanonical = ctx.toolName === "Bash" || PATH_TOOLS.has(String(ctx.toolName)); if (canonicalEmpty && expectsCanonical) { return deny( diff --git a/integration-suite/probe-cli.sh b/integration-suite/probe-cli.sh index 2d1699c66..9c0290dfb 100644 --- a/integration-suite/probe-cli.sh +++ b/integration-suite/probe-cli.sh @@ -140,6 +140,14 @@ fi bun -e 'const m=await import("/repo/src/hooks/fp-config.ts");m.updateConfig({daemon:{configured:false}})' 2>/dev/null || true BASE="$HOME/probe-$CLI" +# openclaw runs every tool inside its OWN workspace, not the cwd it is invoked +# from: `[tools] read failed: File not found: +# /home/canary/.openclaw/workspace/CANARY_MARKER.txt`. So probe B could never +# find the marker and probe A's side effect landed where the verdict never +# looked. Symlinking the workspace at the probe dir is rejected ("workspace path +# alias points to a different current target"), so the probe dir IS the +# workspace — one assignment, and every marker/oracle path below follows. +[ "$CLI" = openclaw ] && BASE="$HOME/.openclaw/workspace" # DEFINITE probes: BENIGN actions (echo/touch a token, read a plain file) the # model never refuses → a tool call is guaranteed, so no INCONCLUSIVE from # self-censorship. A custom canary policy denies exactly those benign markers, @@ -210,10 +218,87 @@ YAML -c "$CUSTOM_POLICIES" >/dev/null 2>&1 # open exec approval (both layers) so the agent issues tool calls headlessly node -e 'const fs=require("fs"),p=process.env.HOME+"/.openclaw/openclaw.json";const c=JSON.parse(fs.readFileSync(p,"utf8"));c.tools=c.tools||{};c.tools.exec=Object.assign({},c.tools.exec,{security:"full",ask:"off",host:"gateway"});fs.writeFileSync(p,JSON.stringify(c,null,2));' - unset FAILPROOFAI_BINARY_OVERRIDE ;; # plugin does `node `; unset → self-resolves to main HEAD + # ── four things openclaw needs that no other CLI does ────────── + # Each was live-diagnosed after openclaw spent weeks reporting + # `NO HOOK LOG — not one hook fired` while its config looked right. + # + # 1. OWNERSHIP. openclaw refuses a plugin it does not own: + # `blocked plugin candidate: suspicious ownership + # (/repo/openclaw-plugin, uid=1000, expected uid=1001 or root)`. + # /repo is the HOST checkout bind-mounted in, so its uid is the + # host user's and never this container's. Copy it into HOME. + # A real install is unaffected — the plugin ships inside the + # user's own npm package. + mkdir -p "$BASE" # the attested workspace must exist before onboard + mkdir -p "$HOME/oc-plugin" + cp -r /repo/openclaw-plugin/. "$HOME/oc-plugin/" + # 2. BARE IMPORT. index.js does `import … from + # "openclaw/plugin-sdk/plugin-entry"`, and from a standalone dir + # node answers ERR_MODULE_NOT_FOUND — openclaw is installed + # under ~/.npm-global, which is on no parent node_modules path. + # openclaw still LISTS the plugin (from its manifest) and + # reports it loaded, so the failure is completely silent. + mkdir -p "$HOME/node_modules" + ln -sfn "$HOME/.npm-global/lib/node_modules/openclaw" "$HOME/node_modules/openclaw" + node -e 'const fs=require("fs"),p=process.env.HOME+"/.openclaw/openclaw.json";const c=JSON.parse(fs.readFileSync(p,"utf8"));c.plugins=c.plugins||{};c.plugins.load=c.plugins.load||{};c.plugins.load.paths=[process.env.HOME+"/oc-plugin"];fs.writeFileSync(p,JSON.stringify(c,null,2));' + # 3. THE BINARY THE SHIM SPAWNS. It runs `node `, and the + # canary's override is a /bin/sh wrapper — `node …/bin/failproofai` + # is a syntax error (the same trap pi hits, see its branch). The + # old fix was to unset it, but that made the shim self-resolve to + # `../dist/cli.mjs` relative to ITSELF, which after (1) is a copy + # in HOME whose bundle cannot resolve its own imports. Point it at + # the real one instead: node-runnable AND still inside /repo, so + # it resolves and stays main HEAD. The shim fails OPEN on a spawn + # error, so every one of these lands as a silent allow. + export FAILPROOFAI_BINARY_OVERRIDE=/repo/dist/cli.mjs + # (4. workspace — handled by BASE at the top; openclaw rejects a + # symlinked workspace with "workspace path alias points to a + # different current target", so the probe dir IS the workspace.) + : ;; esac } +# The gateway openclaw's plugin hooks live in. Restarted per probe for the same +# reason the daemon is: the gateway hosts the plugin, so it must inherit THIS +# probe's FAILPROOFAI_HOOK_LOG_FILE or the oracle lands in the wrong dir. +OPENCLAW_GW_PID="" +openclaw_gateway_stop() { + [ -n "$OPENCLAW_GW_PID" ] || return 0 + kill "$OPENCLAW_GW_PID" 2>/dev/null; wait "$OPENCLAW_GW_PID" 2>/dev/null + OPENCLAW_GW_PID="" +} +OPENCLAW_GW_LOG="" +openclaw_gateway_up() { + # Already serving THIS probe's oracle? Reuse it — drive() is called once per + # retry attempt, and a gateway per attempt would leave three of them bound. + if [ -n "$OPENCLAW_GW_PID" ] && kill -0 "$OPENCLAW_GW_PID" 2>/dev/null \ + && [ "$OPENCLAW_GW_LOG" = "${FAILPROOFAI_HOOK_LOG_FILE:-}" ]; then return 0; fi + openclaw_gateway_stop + OPENCLAW_GW_LOG="${FAILPROOFAI_HOOK_LOG_FILE:-}" + # `gateway run`, and TRUNCATE the log: readiness is a grep for the bound line, + # and openclaw's probe dir persists (it is the attested workspace), so an + # appended log let a PREVIOUS run's line satisfy readiness instantly — the + # turn then went out before the socket existed and got ECONNREFUSED. + # A gateway from a previous probe or a previous RUN still owns the state dir + # ("Another gateway (pid N) already owns this state directory") — the lock + # lives in the persistent volume, so ours never reports ready. Ask openclaw to + # release it; harmless when there is nothing to stop. + openclaw gateway stop >/dev/null 2>&1 || true + : > "$BASE/gateway.log" + openclaw gateway run --allow-unconfigured >> "$BASE/gateway.log" 2>&1 & + OPENCLAW_GW_PID=$! + for _ in $(seq 1 60); do # readiness = the line it prints once bound, not a sleep + grep -q "http server listening" "$BASE/gateway.log" 2>/dev/null && return 0 + kill -0 "$OPENCLAW_GW_PID" 2>/dev/null || break + sleep 1 + done + echo "✗ openclaw gateway did not come up — gateway.log tail:" >&2 + tail -5 "$BASE/gateway.log" >&2 + return 1 +} + +trap openclaw_gateway_stop EXIT + drive() { # $1 = prompt ; run ONE prompt headless, executing tools without approval case "$CLI" in claude) ( cd "$BASE" && claude -p "$1" --model "$CANARY_CLAUDE_MODEL" --dangerously-skip-permissions 2>&1 ) ;; @@ -230,12 +315,30 @@ drive() { # $1 = prompt ; run ONE prompt headless, executing tools without appro devin) ( cd "$BASE" && devin -p "$1" --permission-mode dangerous --respect-workspace-trust false 2>&1 ) ;; antigravity) ( cd "$BASE" && agy -p "$1" --model "${CANARY_ANTIGRAVITY_MODEL:-Gemini 3.5 Flash (Low)}" --dangerously-skip-permissions 2>&1 ) ;; # lightest model → least account-quota use factory) ( cd "$BASE" && droid exec --auto high -m "custom:gw-haiku-0" "$1" 2>&1 ) ;; - openclaw) ( cd "$BASE" && timeout 150 openclaw agent --local --session-key "canary-$RANDOM$RANDOM" --model "gw/$CANARY_LLM_MODEL" -m "$1" 2>&1 ) ;; + # THROUGH THE GATEWAY, never `--local`. openclaw's plugin hooks are + # dispatched by a global hook runner the GATEWAY installs — `hasHooks()` + # answers `?? false` when there is none — so `agent --local` runs the tool + # with our handler registered and never called. Proven by instrumenting the + # plugin itself: `--local` printed REGISTER and nothing else, the gateway + # printed `HANDLER FIRED tool="exec"`. `openclaw agent --help` says as much + # in one line ("Run an agent turn via the Gateway (use --local for …)") and + # the whole integration was probed the other way for weeks. + openclaw) openclaw_gateway_up + ( cd "$BASE" && timeout 150 openclaw agent --session-key "canary-$RANDOM$RANDOM" --model "gw/$CANARY_LLM_MODEL" -m "$1" 2>&1 ) ;; *) echo "drive: $CLI not implemented" >&2; return 3 ;; esac } -rm -rf "$BASE"; mkdir -p "$BASE" +# openclaw ATTESTS its workspace and refuses to run once it disappears +# ("OpenClaw workspace appears to have disappeared after a recent +# initialization … WORKSPACE_VANISHED"), so its probe dir — which IS that +# workspace, see above — is cleared of the probe's own artifacts rather than +# removed. Every other CLI gets the clean slate it always had. +if [ "$CLI" = openclaw ]; then + mkdir -p "$BASE"; rm -f "$BASE"/CANARY_* 2>/dev/null +else + rm -rf "$BASE"; mkdir -p "$BASE" +fi [ "$CLI" = hermes ] && rm -f "$HOME/.hermes/config.yaml" # fresh config each run (append idempotency) # The benign marker file the read-probe asks the agent to read. Its content is a @@ -363,17 +466,31 @@ done # Same ordering rule as probe A: the sentinel leaking into the transcript proves # the read happened, which outranks our own log claiming we denied it. # -# ONE exception, and only one. If the leak arrived while the agent was being -# denied SHELL reads, it got the bytes by a route probe B is not asking about, -# and the honest verdict is "unproven" rather than "broken" — antigravity 1.1.11 -# failed here three runs straight doing exactly that, with every deny correctly -# issued and honoured. The exception is deliberately narrow: a leak with NO -# shell-read attempt is still a FAIL, because that is what a CLI ignoring our -# deny looks like (copilot 1.0.70), and blurring the two would blind this suite -# to the silent-allow it exists to catch. -if printf '%s' "$OUTB" | grep -qF "$MARKER_CONTENT"; then - if shell_route_attempted "$LOGB/hooks.log"; then VB=INCONCLUSIVE; else VB=FAIL; fi -elif drift_suspected "$LOGB/hooks.log"; then VB=FAIL +# TWO exceptions, and only two. If the leak arrived while the agent was being +# denied by canary-read-shell (it reached for the SHELL) or by canary-guard (it +# reached for some OTHER tool), it got the bytes by a route probe B is not +# asking about, and the honest verdict is "unproven" rather than "broken" — +# antigravity failed here for days doing exactly that, with every deny correctly +# issued and honoured (69 hook events, canary-bash + canary-read + canary-guard +# all denying, scored red). The two are one fact wearing two policy names: the +# agent was denied on the tool probe B targets and went looking elsewhere. +# Accepting only the shell half reported the other half as broken enforcement. +# +# Still deliberately narrow: a leak with NEITHER is a FAIL, because that is what +# a CLI ignoring our deny looks like (copilot 1.0.70), and blurring the two +# would blind this suite to the silent-allow it exists to catch. A CLI that +# ignores the deny on the READ tool cannot reach either exception — the probe's +# own payload is exempt from canary-guard by name. +# +# DRIFT IS CHECKED FIRST, and the order is load-bearing. Normalization drift +# denies under `canary-guard` too — same policy, opposite meaning — so leaving +# it below the leak branch would let the widened exception downgrade the very +# silent-allow class (copilot 1.0.70's) this guard exists to catch, from FAIL to +# a quiet yellow. +if drift_suspected "$LOGB/hooks.log"; then VB=FAIL +elif printf '%s' "$OUTB" | grep -qF "$MARKER_CONTENT"; then + if shell_route_attempted "$LOGB/hooks.log" || route_around "$LOGB/hooks.log" + then VB=INCONCLUSIVE; else VB=FAIL; fi elif read_denied "$LOGB/hooks.log"; then VB=PASS elif [ "${CANARY_DAEMON_DEAD:-0}" = 1 ] && daemon_failed_closed "$LOGB/hooks.log"; then VB=PASS elif is_error "$OUTB"; then VB=ERROR diff --git a/package.json b/package.json index 5b8cbd6b7..a97671cdc 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "failproofai", - "version": "1.0.3-beta.0", + "version": "1.0.3", "description": "Observability and enforcement for AI agent harnesses. 39 built-in policies hooked into 12 of them — Claude Code, Codex, Cursor, Hermes, OpenClaw and more — blocking the tool call before it runs. Local dashboard included, no account needed.", "bin": { "failproofai": "./dist/cli.mjs", diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index 4fd4ef929..13dc34e7b 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -394,20 +394,47 @@ async function installHooksImpl( // Check unknown names first (most actionable error for the user). Pack // policies are applied here and taken out of the list: the rest of this // function writes `enabledPolicies`, which is a builtin-only set. + let wireHooksOnly = false; if (nonAllNames.length > 0) { const resolved = resolvePolicyNames(nonAllNames); applyPackPolicies(resolved.packs, true, scope, cwd); if (resolved.packs.length > 0) { policyNames = policyNames.filter((n) => n === "all" || resolved.builtins.includes(n)); - // Named ONLY pack policies: the work is done. Carrying on would resolve - // the failproofai binary and rewrite every CLI's settings to enable a - // set of builtins nobody asked about — and would fail outright on a - // machine where the binary is not on PATH, AFTER the pack change landed. - if (policyNames.length === 0) return; + if (policyNames.length === 0) { + // Named ONLY pack policies. Two different requests hide in that + // shape, and treating them alike wired no hooks for three days + // across nine agent CLIs while reporting success. + // + // A THIRD-PARTY name (`policies add block-big-refund`) implies no + // hook work: the switch IS the request. Carrying on would resolve + // the failproofai binary and rewrite every CLI's settings to enable + // a set of builtins nobody asked about — and would fail outright on + // a machine where the binary is not on PATH, AFTER the pack change + // landed. So that one still stops here. + // + // A BUILTIN name is an ordinary install — `policies --install + // block-sudo --cli codex`, the form 16 CLAUDE.md references and the + // quickstart both name. Every builtin is ALSO declared by the + // bundled pack, so from the first install onward every such command + // looked "pack-only" and returned here: exit 0, a reassuring + // `Enabled … from pack` line, and no settings file for that CLI, + // with --cli/--scope/--custom dropped on the way out. The first + // install per machine worked and every one after it was dead, which + // is why it survived manual testing. + if (!nonAllNames.some((n) => VALID_POLICY_NAMES.has(n))) return; + wireHooksOnly = true; + } } } - // Then check if "all" is mixed with valid specific names - if (policyNames.includes("all") && nonAllNames.length > 0) { + if (wireHooksOnly) { + // Wire the hooks, touch no policy — the same path `--install` with no + // names takes. `applyPackPolicies` above already flipped these on in the + // pack, which is where the switch lives now; re-writing them into + // `enabledPolicies` would resurrect the stale-key problem the pack lane + // exists to end (see the `fromPack` note below). + policyNames = undefined; + } else if (policyNames.includes("all") && nonAllNames.length > 0) { + // Then check if "all" is mixed with valid specific names throw new CliError( `"all" cannot be combined with specific policy names.\n` + `Use either: --install all or --install block-sudo sanitize-jwt ...`