feat(triage): experimental Flue-based issue triage (phase 1 + 2)#1090
feat(triage): experimental Flue-based issue triage (phase 1 + 2)#1090ascorbic wants to merge 2 commits into
Conversation
Adds two complementary triage paths under .flue/: Phase 1: Worker-deployed auto-labeller (agents/triage-label.ts). Receives GitHub issues.opened webhooks, verifies HMAC against raw bytes via Web Crypto, classifies the issue with kimi-k2.6 through our existing Cloudflare AI Gateway (same path /bonk uses), and applies bug/question/documentation + area/* labels with a short summary comment. Filter-then-apply guards against hallucinated label names. DRY_RUN env var skips writes for first deploys. Phase 2: GH-Actions-driven reproduction (agents/repro-issue.ts + .github/workflows/auto-repro.yml). Fires only when a maintainer applies the triage:reproduce label, never on every new issue. Uses local() sandbox so the agent's bash tool gets real pnpm/git/ gh/node access on the runner. Tries to reproduce as a vitest test or repro script and posts a single summary comment with one of reproduced/not-reproduced/repro-skipped labels. Never pushes branches, commits, or attempts fixes. Local prototype runner (scripts/run-local.ts) wraps flue run triage-issue with real issue payloads (5 fixtures from current queue) so the prompt can be iterated without deploying anything. All model traffic routes through CF_AI_GATEWAY_* secrets shared with bonk/review for unified cost tracking. Not deployed yet -- needs a Discussion before any actual rollout.
|
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | b453ae9 | May 23 2026, 03:54 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | b453ae9 | May 23 2026, 03:53 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-perf-coordinator | efd0efc | May 18 2026, 11:23 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | b453ae9 | May 23 2026, 03:54 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-i18n | efd0efc | May 18 2026, 11:23 AM |
Scope checkThis PR changes 6,530 lines across 20 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/blocks
@emdash-cms/cloudflare
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
|
/review |
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
| className: "TriageLabel", | ||
| sqlite: true, | ||
| }), | ||
| GITHUB_WEBHOOK_SECRET: process.env.GITHUB_WEBHOOK_SECRET ?? "dev-secret-rotate-me", |
There was a problem hiding this comment.
HIGH — fail-open default for webhook secret.
When process.env.GITHUB_WEBHOOK_SECRET is unset, the binding is initialised to the literal string "dev-secret-rotate-me". Combined with triage-label.ts:226 which only refuses the request when secret is falsy, this means a deploy that forgets to set the secret will happily verify HMACs against a value now public in this PR. Anyone reading the source can forge issues.opened webhooks against the deployed Worker.
This is the kind of footgun that bites the first time someone runs alchemy deploy against a real stage without the secret bound. The comment two lines above is the only thing stopping it.
Fix: default to "" (or omit the binding) so the request handler's if (!secret) branch refuses the request, instead of a plausible-looking sentinel that passes the existence check.
| lines.push(""); | ||
| lines.push("```"); | ||
| lines.push(repro.notes); | ||
| lines.push("```"); |
There was a problem hiding this comment.
MEDIUM — fenced-code-block escape via model output.
repro.notes is the agent's transcript of bash commands and their captured output. Whenever the agent runs pnpm test or grep against a markdown file (highly likely on this monorepo — docs/, README.md, CONTRIBUTING.md, every plugin's docs), the captured output frequently contains a literal triple-backtick. Wrapping it in ``` ... ``` then breaks out of the fence and the rest of the comment renders as raw markdown / executes any nested formatting.
This isn't theoretical — the very reproduce skill encourages running grep/test commands whose stdout includes the repo's markdown. First time the model captures a doc snippet, the comment is mangled.
Fix: use a four-backtick (or longer) fence, or replace \u0060\u0060\u0060 runs inside repro.notes before interpolation. The simplest: find the longest backtick run in notes, fence with one more.
| // surface. Override with FLUE_REPRO_MODEL for experiments. | ||
| model: process.env.FLUE_REPRO_MODEL ?? "cloudflare-ai-gateway/claude-opus-4-7", | ||
| }); | ||
| const reproSession = await reproHarness.session(); |
There was a problem hiding this comment.
MEDIUM — prompt injection from issue body into shell-equipped agent.
The repro session uses local() so its bash tool has real pnpm / gh / git / node against the runner's filesystem, and GH_TOKEN is explicitly injected on line 132. Then reproSession.skill("reproduce", { args: { ..., issueTitle, issueBody, triage } }) feeds the entirely-attacker-controlled issue body into the prompt context.
The "strict guardrails" called out in the PR description and in SKILL.md ("no commits, no fix attempts, no curl") are prompt-level — they are advisory text the model can be argued out of by a sufficiently clever issue body. With write-scoped GH_TOKEN in env, a successful jailbreak can call gh to comment on any issue/PR in the repo, apply or remove arbitrary labels, close issues, etc. (The token can't push to branches because the workflow grants contents: read, which is the one real guardrail.)
The trigger is gated on triage:reproduce being applied by a maintainer, which is the main mitigation. But the threat model is: open a malicious issue, wait for a triager to mistakenly request reproduction. Worth being explicit about this in the README / Discussion before enabling, and considering whether the runner should run with a token that only has issues:write on the target issue (e.g. a finer-grained PAT or App installation scoped via headers).
| ? "reproduced" | ||
| : "not-reproduced"; | ||
| try { | ||
| await addLabels(githubToken, owner, repo, issueNumber, [resultLabel]); |
There was a problem hiding this comment.
LOW — comment claims the workflow creates labels lazily; nothing does.
Lines 165-169 say "these labels don't exist yet; the workflow creates them lazily" but neither auto-repro.yml nor this file calls POST /repos/{owner}/{repo}/labels to create them. addLabels calls POST /repos/{owner}/{repo}/issues/{n}/labels, which returns 422 if the label doesn't exist on the repo. The try/catch on line 175 swallows this so it's not fatal, but the first N runs will silently fail to apply any result label until someone manually creates reproduced, not-reproduced, repro-skipped in the repo's Labels settings.
Fix options: (a) precreate the three labels as part of the workflow's setup step using gh label create --force, or (b) update the comment to say "these labels must be precreated" so the next person reading the code knows.
| // Set to "true" to skip HMAC verification. DEV ONLY — never enable in | ||
| // production. Used for local `flue run` prototyping where we can't | ||
| // easily sign the synthetic payload. | ||
| SKIP_HMAC?: string; |
There was a problem hiding this comment.
LOW — SKIP_HMAC is read directly from env with no compile-time gate.
SKIP_HMAC === "true" bypasses HMAC verification entirely. The comment says "DEV ONLY — never enable in production" and alchemy.run.ts doesn't currently expose SKIP_HMAC as a binding, so it can't be turned on via the standard deploy path. But a future contributor copy-pasting a binding row or a wrangler secret put SKIP_HMAC true would silently disable auth on the live Worker, with no log telemetry beyond a single warn line that gets buried under request volume.
Fix: gate this on a build-time signal that can't be flipped after deploy — e.g. only consult the flag when globalThis.process?.env?.NODE_ENV === "development" (or equivalent for the Workers runtime), or just delete it. The local prototype path (scripts/run-local.ts) doesn't hit this code path anyway — it calls triage-issue.ts directly without the webhook agent.
| } | ||
| lines.push(""); | ||
| lines.push(`> ${triage.summary}`); | ||
| return lines.join("\n"); |
There was a problem hiding this comment.
LOW — triage.summary interpolated into a blockquote line with no newline-stripping.
The schema allows up to 200 chars and \n is a valid character in a v.string(). If the model returns a multi-line summary, only the first line is in the blockquote — the rest renders as bare body text, breaking the formatted comment.
Not a security issue (GitHub comments don't render raw HTML), but the auto-comment loses its structured look the moment the model decides to return two sentences. Strip \r?\n from triage.summary before rendering, or replace with a space.
| async function loadFixture(arg: string, live: boolean): Promise<Fixture> { | ||
| if (live) { | ||
| const raw = execSync( | ||
| `gh issue view ${arg} --repo emdash-cms/emdash --json number,title,body,labels`, |
There was a problem hiding this comment.
LOW — unvalidated arg interpolated into gh command.
gh issue view ${arg} where arg is args.filter((a) => !a.startsWith("--"))[i]. Nothing enforces that arg is a number — pnpm prototype '1 && rm -rf /' (or anything with shell metachars) is passed through execSync as-is. This is a local dev tool with no security boundary, but the same arg is then used to construct issue-${arg}.json and would happily traverse with ../.
Fix: assert /^\d+$/.test(arg) before either call, fail with a clear message otherwise.
|
Review posted successfully with 7 line-anchored comments. Summary of findings:
|
HIGH: - alchemy.run.ts: default GITHUB_WEBHOOK_SECRET to "" (was a sentinel string that would have fail-open authenticated forged webhooks). The empty default makes the agent's `if (!secret)` guard fail closed. MEDIUM: - repro-issue.ts: dynamic-length backtick fence for `repro.notes` in the rendered comment. Captured bash output frequently contains triple backticks (markdown docs, README snippets, etc.) which would otherwise break out of the code fence. - README + SKILL.md: document the threat model for the Phase 2 repro agent. The maintainer label gate is the real security boundary, not the prompt-level guardrails. Add explicit hard prohibitions to the skill (no git push, no PR writes, no label management, no curl, etc.) and call out the worst-case capabilities of a successful jailbreak. LOW: - auto-repro.yml: precreate the three result labels (reproduced / not-reproduced / repro-skipped) idempotently with gh label create --force. Update the misleading comment in repro-issue.ts that implied the workflow did this already. - triage-label.ts: remove SKIP_HMAC entirely. The local prototype path uses triage-issue.ts (no webhook) so the bypass was never reachable in dev, and a wrangler secret put SKIP_HMAC true on a live deploy would silently disable verification. - triage-label.ts: collapse newlines in triage.summary before rendering into the blockquote, so a multi-line model summary doesn't leak text outside the > prefix. - run-local.ts: validate issueNumber matches /^\d+$/ before interpolating into the gh command and the fixture path.
1 participant
What does this PR do?
Adds an experimental Flue-based issue triage system. Two complementary paths, both opt-in, neither deployed. Tooling-only -- nothing published, no behaviour change in any package.
Phase 1: Worker-deployed auto-labeller (
.flue/agents/triage-label.ts)Receives
issues.openedwebhooks, verifies HMAC against raw bytes via Web Crypto, classifies the issue withkimi-k2.6through our existing Cloudflare AI Gateway (same path/bonkand/reviewuse), and applies labels + posts a structured summary comment. Filter-then-apply guard against hallucinated label names.DRY_RUNenv var skips writes for first deploys.Schema enforces:
kind(bug/enhancement/documentation/question),severity(low/medium/high/critical), 0-3area/*labels,reproduciblebool,dataLossRiskbool, one-sentencesummary.Phase 2: GH-Actions-driven reproduction (
.flue/agents/repro-issue.ts+.github/workflows/auto-repro.yml)Fires only when a maintainer applies the
triage:reproducelabel, never on every new issue. Uses Flue'slocal()sandbox so the agent's bash tool gets realpnpm/git/gh/nodeaccess on the runner. Tries to reproduce as a vitest test or repro script, posts a single summary comment, appliesreproduced/not-reproduced/repro-skipped. Strict guardrails: no branch pushes, no commits, no fix attempts.Local prototype runner (
.flue/scripts/run-local.ts)Wraps
flue run triage-issuewith real issue payloads. 5 fixture issues from the current queue (#1021, #1042, #1046, #1049, #1080) so the prompt can be iterated without deploying anything.All model traffic routes through the
CF_AI_GATEWAY_*secrets shared withbonk.ymlandreview.ymlfor unified cost tracking and request logs in the gateway dashboard. No new auth surface, no new provider keys.Why two phases: Phase 1 is cheap (~$0/issue on Workers AI), fast (~5s), and conservative (label + comment only) -- safe to run on every new issue. Phase 2 is expensive (Opus on a 30-min runner) and powerful (real shell, can write tests) -- runs only on explicit maintainer opt-in.
Not deployed. This PR adds the scaffolding only. Before turning either on, we should agree on the design in a Discussion -- particularly the wording of the auto-comment, whether to skip
enhancementauto-labelling (currently yes, since features need a prior Discussion per CONTRIBUTING.md), and how/whether the Phase 2 reproduction comment should be allowed to suggest a next step.Closes #
Type of change
Checklist
pnpm typecheckpassespnpm lintpasses (0 diagnostics in the new files; 79 pre-existing inmainare untouched)pnpm testpasses (or targeted tests for my change) -- no test surface changedpnpm formathas been runmessages.pochanges except in translation PRs -- a workflow extracts catalogs on merge tomain. (N/A -- no admin UI changes.)AI-generated code disclosure
Screenshots / test output
pnpm typecheckin.flue/:Both build targets work:
Lint:
Review feedback
The first review pass surfaced one HIGH (fail-open webhook secret default), two MEDIUM (fence-escape in repro comment, prompt-injection threat model), and four LOW issues. All addressed in
b453ae95. See the resolved review threads for details.