Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,8 @@ After writing the ADR, number it by finding the highest existing `docs/adr/NNNN-

## Troubleshooting

For Sentry `buildwithfern` / `cli` false-positive triage, see [`automation/sentry-triage/AGENT.md`](automation/sentry-triage/AGENT.md), [`automation/sentry-triage/DESIGN_CHOICES.md`](automation/sentry-triage/DESIGN_CHOICES.md), and [`automation/sentry-triage/ledger.json`](automation/sentry-triage/ledger.json).

### Quick Fixes by Issue Type
- **Generator failures**: Check `docker ps` → Rebuild image → Check container logs
- **IR compilation**: `pnpm fern check` → Check circular refs → Review migrations
Expand Down
108 changes: 108 additions & 0 deletions automation/sentry-triage/AGENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Sentry CLI false-positive triage — agent instructions

Read **`DESIGN_CHOICES.md`** before proposing code changes; it holds short, accumulated fix patterns. After human review changes the accepted approach, update **`DESIGN_CHOICES.md`** with **1–3 new bullets** (short phrases only) when the lesson is reusable.

## Scope

- Sentry org: `buildwithfern`
- Sentry project: `cli`
- Use the repo-baked Sentry CLI: `pnpm exec sentry-cli` (do not rely on a global `sentry` binary).
- Default: unresolved only — `pnpm exec sentry-cli issue list … --query "is:unresolved"` unless the task says to re-validate resolved issues.

## Run shape

- **One PR per solution group.** Group Sentry issues by the code change that fixes them: if one change solves multiple `shortId`s, ship those together in one PR; unrelated fixes must be separate PRs.
- **Do not create one catch-all PR for a whole run.** A run may produce multiple PRs, but each PR must correspond to exactly one solution group.
- **Defer** opening PRs until grouping and investigation are far enough along that you are not guessing which `shortId`s share a fix.

## Fetch (Sentry CLI)

```bash
pnpm exec sentry-cli issue list buildwithfern/cli \
--json --fields shortId,title,status,lastSeen \
--limit 100 --query "is:unresolved"

pnpm exec sentry-cli issue view buildwithfern/cli/<SHORT_ID> --json
```

Prefer `--json` and `--fields` to keep payloads small.

## Ledger-first and query-first

1. Collect `shortId` from list output.
2. Do **not** load all of `automation/sentry-triage/ledger.json` into context as it grows.
3. Targeted lookup with `jq`:

```bash
jq --argjson ids '["CLI-2W","CLI-2V"]' '
.issues
| to_entries
| map(select(.key as $id | $ids | index($id)))
| map({
shortId: .key,
disposition: .value.disposition,
duplicateOf: .value.duplicateOf,
problemSignature: .value.problemSignature,
prOrIssue: .value.prOrIssue
})
' automation/sentry-triage/ledger.json
```

4. **Same `shortId` only:** if this Sentry issue’s `shortId` already exists in the ledger with disposition `shipped`, `ignored`, `duplicate`, or `keep_sentry`, **skip** re-triaging that row (`keep_sentry` means do not suppress—do not reclassify as a false positive). This is **not** “similar title so skip”; it is exact `shortId` match.
5. **Re-process** `pending_review` until set to a terminal disposition.
6. `pnpm exec sentry-cli issue view` in full only for unknown or `pending_review` rows (and tight clustering follow-ups).
7. **Similar past cases (inspiration only):** for a `shortId` **not** in the ledger (or still in play), you may stream `problemSignature` + `shortId` (e.g. `jq -r '.issues | to_entries[] | "\(.key)\t\(.value.problemSignature)"' automation/sentry-triage/ledger.json`) and keyword-search using the **new** issue’s title/exception (MDX, YAML, errno, container, `generators.yml`, etc.) to find **prior fixes to learn from**. **Do not** treat a text match as proof the new issue is already solved: always confirm with current Sentry payload, stack, and code paths. A recurrence or regression still needs a normal investigation and disposition.

## Ledger shape (`issues`)

| Field | Required | Meaning |
|-------|----------|---------|
| `title` | yes | Short Sentry title. |
| `problemSignature` | yes | **1–3 short sentences:** symptom + feature surface so **future** triage can grep the ledger for **similar** past cases as **hints** (how we fixed before)—not as proof the new issue is closed. For `duplicate`, copy the canonical row’s `problemSignature` so search still hits this `shortId`. |
| `disposition` | yes | See table below. |
| `rationale` | yes | Why this disposition. |
| `fixSummary` | yes | What changed, or `—` for `duplicate` / `ignored` when not applicable. |
| `prOrIssue` | yes | PR URL, issue link, or note. |
| `lastAnalyzed` | yes | ISO date. |
| `duplicateOf` | if `duplicate` | Canonical ledger `shortId` whose `fixSummary` / PR is the source of truth. |

## Ledger disposition meanings

| Value | Meaning |
|-------|---------|
| `shipped` | Fix merged; events should stop after release or issue resolved in Sentry. |
| `ignored` | No code change; rationale recorded. |
| `keep_sentry` | Real bug; stays reportable until a product fix. |
| `duplicate` | Same **Sentry** issue family as another ledger row: `duplicateOf` points at the canonical `shortId` (one fix narrative). **Skip** re-triage for **that** `shortId` only—same as `shipped` for workload. This is unrelated to “looks like” similarity for **new** `shortId`s; those still get full analysis. |
| `pending_review` | Needs decision; resolve on next pass. |

**Why keep `duplicate`?** Sentry often opens **multiple issues** for one underlying fix. One row stays `shipped` with the full story; the rest become `duplicate` + `duplicateOf` so the next run does not re-investigate **those same ledger shortIds** and the ledger stays small. If you prefer, you can mark every follower `shipped` instead—`duplicate` is only ergonomics.

## Code and PR rules

- Follow **`DESIGN_CHOICES.md`** and nearby throw sites in this repo.
- Branch: use `FedeZara/fix/sentry-<short-id-summary>-<kebab-solution>` for each solution group (for example, `FedeZara/fix/sentry-cli-30-cli-3g-docs-yaml-parse`). Include at most three `shortId`s in the branch name.
- PR title: include the solved `shortId`s in the title. If there are more than three, list only the first three and summarize the family (for example, `fix(cli): classify CLI-30 CLI-3G CLI-Q docs YAML parse errors`).
- Changelog: `packages/cli/cli/changes/unreleased/` when the CLI behavior or reporting changes.
- PR description: explicitly list **each Sentry error solved** by the PR, including its `shortId` (for example, `CLI-2W`) and a short fix note.

## Phases (single parent agent)

Optional read-only explore subagents for search only; parent owns edits and the PR.

1. **Parent** — read this file + `DESIGN_CHOICES.md`; `jq` ledger for current `shortId`s; no PR until grouping is sound.
2. **Fetch** — `pnpm exec sentry-cli issue list` / `pnpm exec sentry-cli issue view` as above.
3. **Ledger filter** — for each **current** `shortId`, if the ledger row exists and disposition is terminal (`shipped`, `ignored`, `duplicate`, `keep_sentry`), skip that issue; keep unknown `shortId`s + `pending_review`. Similar `problemSignature` matches elsewhere do **not** skip an unknown `shortId`.
4. **Group** — by the concrete solution, not just by similar titles. Same code path / same root cause / same patch → one solution-group PR. Different patches → different PRs.
5. **Investigate** — nearest catch/throw; align with `DESIGN_CHOICES.md`.
6. **Fix** — open one PR per solution group; each PR description lists every solved Sentry error with `shortId`; changelog when needed; `keep_sentry` for confirmed real bugs.
7. **Record** — update `ledger.json` in the **same** PR as the code (and `DESIGN_CHOICES.md` when a reusable pattern is confirmed).

## Record updates after human review

When the task is to align records with **final** human decisions (no product code unless asked):

- Load `ledger.json` only (or the specific `issues.<shortId>` entries you are changing).
- Update ledger fields consistently (including `duplicateOf` if you change which row is canonical, and **`problemSignature`** if humans clarify how this issue should be found next time).
- **Update `DESIGN_CHOICES.md`** when the correction is a reusable pattern: add or replace **short** bullets only—no long narratives.
- Do **not** re-fetch Sentry unless the task requires verification.
20 changes: 20 additions & 0 deletions automation/sentry-triage/DESIGN_CHOICES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Triage design choices (compact)

One line per item. Append or edit **short** bullets when human review confirms a pattern; do not paste long prose. Principles here; **implementation lives in code and PRs**, not in this file.

## Choices

- User/environment syscalls (errno-style): classify at the **central** error-mapping layer **only** when the signal is a clear, code-level errno—not guessed from message text.
- Do **not** route “looks like X” string heuristics through that central layer; keep heuristics out or extremely narrow.
- Do **not** add central `isXError` classifiers except existing schema-validation / Node-version checks; parse/config/user errors should be explicit `CliError` codes at the boundary.
- User-authored content (docs, specs, config files): surface failures as **user-facing** parse/config/validation errors **at the boundary** where input enters the CLI—not as generic internal errors.
- Conversion or ingest steps on user input (format upgrades, parsers): failures should read as **user input problems**, with useful context when available.
- User configuration (versions, paths, generator references): wrong or invalid values → **config or validation** semantics, not internal errors.
- True product defects: **keep_sentry** in the ledger; do not hide them behind reclassification.
- Long-running or multi-step work: avoid **double reporting** (one failure should not become both a task failure and a top-level internal error).
- External service/job failures: surface as **non-reportable** service/network failures unless the CLI itself caused the defect.
- Subprocess / tool re-exec failures: prefer **abort or controlled failure** over a misleading internal error.
- Local tool failures from user worktrees/config (git, buf, sed): classify at the **tool boundary** as user/environment errors.
- Auth/AI/provider setup failures: classify at the **auth/config boundary**, not as generic internal errors.
- Pipe/TTY noise when users chain Unix tools: swallow or classify at the **I/O boundary** so it never looks like an app bug.
- If runtime minification or bundling can break **name-based** error checks, make classification rely on something **stable** (e.g. explicit names or codes), not fragile `constructor.name` alone.
3 changes: 3 additions & 0 deletions automation/sentry-triage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# automation/sentry-triage

Agent triage instructions: **[`AGENT.md`](AGENT.md)**. Fix patterns (short bullets, updated after review): **[`DESIGN_CHOICES.md`](DESIGN_CHOICES.md)**. State: **`ledger.json`**.
Loading
Loading