Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions docs/code/sentry-integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
title: "Sentry Auto-fixes"
sidebarLabel: "Sentry Auto-fixes"
description: "Turn Sentry error groups into repo-routed fixes from the workspace worker"
section: "Automation"
order: 4
dateModified: 2026-09-04
---

# Sentry Auto-fixes

The workspace worker can poll one or more Sentry projects for unresolved error
groups and run actionable errors through the normal fix pipeline: isolated
worktree, coding agent, tests, commit, and pull request.

## Configure projects in `workspace.toml`

Add one `[[error_monitors]]` entry per Sentry project. Every entry maps to the
repository that owns the code, so a multi-repo worker never has to guess where
an error should be fixed.

```toml
[[error_monitors]]
id = "api-production"
provider = "sentry"
repo = "backend"
team = "platform" # optional; must match a [[teams]] name
organization = "acme"
project = "api"
query = "environment:production level:error"
poll_interval = 60
min_occurrences = 5
max_per_tick = 3
env_file = "env/sentry-api.env"

[[error_monitors]]
id = "web-production"
provider = "sentry"
repo = "frontend"
organization = "acme"
project = "web"
env_file = "env/sentry-web.env"
```

`repo` may be omitted only when the workspace has exactly one `[[repos]]`
entry. In a multi-repo workspace it is required. `team` is optional and lets
the fix run inherit that team's environment and identity in addition to the
repository environment.

Each source is independent: use a separate `env_file` or
`[error_monitors.env]` table when projects need different credentials.

```bash
# env/sentry-api.env
SENTRY_AUTH_TOKEN=sntrys_...
```

```toml
[[error_monitors]]
id = "internal-api"
provider = "sentry"
repo = "backend"
organization = "acme"
project = "api"
base_url = "https://sentry.internal.example"
[error_monitors.env]
SENTRY_AUTH_TOKEN = "sntrys_..."
```

Do not put a Sentry DSN here. A DSN sends events into Sentry; polling issues
requires an auth token plus the organization and project slugs. Create an auth
token with `project:read` and `event:read` access.

Credential precedence, from lowest to highest, is: process environment,
workspace `.env`, repo `env_file`, `[repos.env]`, team credentials, the error
monitor's `env_file`, then `[error_monitors.env]`. This allows one worker to
serve teams and projects whose tokens differ.

## Behavior

An error is eligible when it meets `min_occurrences` (default `5`) and includes
a title plus a culprit, exception type, or filename. At most `max_per_tick`
(default `3`) errors are dispatched per poll. `poll_interval` defaults to
`[defaults].poll_interval`.

Handled issue IDs are stored in the workspace database under a source key that
includes the provider and configured source `id`. That prevents collisions
between Sentry projects. A failed fix is not automatically repeated; a run
deferred because the repo or agent capacity is busy is released and retried on
a later poll.

The provider contract is shared by all error monitors. Sentry is the first
adapter; adding Datadog support does not require another polling, deduplication,
or workspace-routing implementation.

`[[error_monitors]]` changes are validated by live reload but require a worker
restart because clients and credentials are startup-scoped.
8 changes: 8 additions & 0 deletions docs/code/worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ devintern webhook serve

Set `AGENT_HARNESS=codex,grok` (comma-separated, priority first) in the workspace `.env` so the worker keeps going when one agent hits a usage limit. Failover applies to every worker job: tracker tasks, PR review addressing, `@mention` runs, conflict resolution, scheduled automations, estimations, dashboard retries, and relay-driven work. Details: [Failover across multiple harnesses](./configuration.md#failover-across-multiple-harnesses).

## Error-monitor auto-fixes

`[[error_monitors]]` entries let the worker turn unresolved production errors
into normal repo-scoped fix runs. Each Sentry project maps explicitly to one
`[[repos]]` entry and can inherit an optional `[[teams]]` environment, so one
worker can safely serve multiple teams, repositories, and credentials. See
[Sentry Auto-fixes](./sentry-integration.md) for the schema and setup.

## Recurring automations

Put recurring work in `workspace.toml`. Set `repo` when the workspace has multiple repositories; it is optional for a one-repo workspace:
Expand Down
4 changes: 3 additions & 1 deletion docs/code/workspaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ prompt = "Review the frontend and clean up one source of recurring noise."
- `[worker.schedule]` gates only new-task pickup: multiple windows union, windows may cross midnight, `blocked` wins on overlap, and a missed whole window triggers one catch-up drain at startup. Timezone/DST semantics and `devintern worker run-now` are covered in [Running the Worker Unattended: Working windows](./automated-task-processing.md#working-windows-quiet-hours).
- `[[automations]]` uses the same schema as single-repo `.devintern-code/automations.toml`. An entry must name `repo` when the workspace has more than one repository. See [Worker Daemon → Recurring automations](./worker.md#recurring-automations) for prompt-writing guidance and schedule semantics.
- `[[estimations]]` schedules unattended story-point sweeps (tracker query + cron/interval, no `prompt`, no `repo`). The workspace tracker must support estimation. See [Worker Daemon → Scheduled story-point estimation](./worker.md#scheduled-story-point-estimation).
- `[[error_monitors]]` maps each Sentry project to one repo and an optional team, with per-source credential layers for multi-project setups. See [Sentry Auto-fixes](./sentry-integration.md).

### Multiple teams and tracker boards

Expand Down Expand Up @@ -229,7 +230,7 @@ devintern worker # auto-detects ~/.devintern/workspace.toml
devintern worker --workspace /path/to/workspace.toml
```

The single-source fleet query comes from `[defaults].task_query`; multi-team workspaces use each team's `task_query`. A workspace with automations or estimations can omit the defaults query and run as a schedules-only worker. Poll interval, per-task flags, and the embedded dashboard are also set in `workspace.toml` (`poll_interval`, `worker_task_args`, `[worker.schedule]` quiet hours, `[workspace].dashboard` / `dashboard_port`). Direct webhooks are an advanced repo-local service: run `devintern webhook serve` from that repository as a separate process. Automation and estimation schedule state and leases, plus the task-polling timestamp used for missed-window catch-up, live in the central workspace database.
The single-source fleet query comes from `[defaults].task_query`; multi-team workspaces use each team's `task_query`. A workspace with automations, estimations, or an enabled error monitor can omit the defaults query. Poll interval, per-task flags, and the embedded dashboard are also set in `workspace.toml` (`poll_interval`, `worker_task_args`, `[worker.schedule]` quiet hours, `[workspace].dashboard` / `dashboard_port`). Direct webhooks are an advanced repo-local service: run `devintern webhook serve` from that repository as a separate process. Automation and estimation schedule state and leases, plus task-polling and error-monitor deduplication state, live in the central workspace database.

While the daemon is running you can request one immediate drain (for example while quiet hours are closed) with `devintern worker run-now`; see [Working windows](./automated-task-processing.md#working-windows-quiet-hours).

Expand All @@ -239,6 +240,7 @@ The worker watches `workspace.toml` and reloads it automatically a moment after

- **Routing rules, repos, defaults/team `task_query`, team `repo`, `[[automations]]`, `[[estimations]]`, `worker_task_args`, `poll_interval`, `worktrees_ttl_days`, and conflict-resolution mode/schedules apply to subsequent work.** Runs already in progress finish under the configuration they started with; everything picked up afterwards uses the new one. Changing a repo's `remote` updates its managed bare clone the next time that repo is prepared.
- **Team identity and credentials are startup-only.** Restart after changing a team's name, tracker, `env_file`, or inline `[teams.env]` values.
- **Error monitor clients are startup-only.** Restart after changing `[[error_monitors]]`, including project routing or source credentials.
- **A broken edit never takes the daemon down.** The reload validates the file first; parse or schema errors are logged (naming the offending entries) and the last valid configuration keeps serving until you fix it. Rewriting identical content is ignored.
- **Manual fallback:** send SIGHUP (`kill -HUP <pid>`) to force an immediate reload if file watching is unavailable on your system.
- **Startup-only settings** still require a restart: tracker credentials in the workspace `.env` and `[defaults].tracker` (the tracker client and its detector are built once), `[worker.schedule]` quiet hours (the working-window gate is built once at startup), plus `[workspace].dashboard` / `dashboard_port`. A reload that changes one of these settings is rejected in full, so the active config remains internally consistent.
Expand Down
161 changes: 161 additions & 0 deletions packages/code/src/lib/error-monitor.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
/**
* Provider-neutral error-monitor polling.
*
* Provider adapters normalize their native issue shape through
* {@link ErrorMonitorProvider}. The acquirer owns lifecycle, deduplication,
* thresholds, and dispatch so future providers (for example Datadog) do not
* need to duplicate worker behavior.
*/

import type { TaskExecutionResult } from "./task-polling-acquirer";
import type { WebhookQueue } from "./webhook-queue";
import type { Acquirer } from "../worker";
import type { ErrorMonitorConfig } from "./workspace/config";
import { SentryClient } from "./sentry-client";

/** Minimum normalized issue data needed by the shared acquirer. */
export interface ErrorMonitorIssue {
/** Provider-stable identifier used for durable deduplication. */
externalId: string;
/** Human-readable identifier used in logs and task filenames. */
displayId: string;
title: string;
occurrenceCount: number;
}

export interface IssueValidity {
valid: boolean;
reason?: string;
}

/** Adapter contract implemented once per error-monitoring vendor. */
export interface ErrorMonitorProvider<TIssue extends ErrorMonitorIssue> {
readonly providerName: string;
fetchIssues(): Promise<TIssue[]>;
validateIssue(issue: TIssue): IssueValidity;
buildTaskMarkdown(issue: TIssue): string;
}

export interface ErrorMonitorAcquirerOptions<TIssue extends ErrorMonitorIssue> {
sourceId: string;
intervalSeconds: number;
queue: WebhookQueue;
provider: ErrorMonitorProvider<TIssue>;
executeTask: (issue: TIssue, markdown: string) => Promise<TaskExecutionResult>;
minOccurrences?: number;
maxIssuesPerTick?: number;
verbose?: boolean;
}

/** Build the configured vendor adapter without leaking vendor logic into the worker. */
export function createErrorMonitorProvider(
config: ErrorMonitorConfig,
env: Record<string, string | undefined>,
): ErrorMonitorProvider<ErrorMonitorIssue> {
switch (config.provider) {
case "sentry": {
const authToken = env.SENTRY_AUTH_TOKEN;
if (!authToken) {
throw new Error(
`Error monitor "${config.id}" is missing SENTRY_AUTH_TOKEN. ` +
"Add it to the workspace, repo, team, or source env_file/env layer.",
);
}
return new SentryClient({
authToken,
organization: config.organization,
project: config.project,
baseUrl: config.baseUrl,
query: config.query,
});
}
}
}

/** Shared polling acquirer for Sentry, Datadog, and future adapters. */
export class ErrorMonitorAcquirer<TIssue extends ErrorMonitorIssue> implements Acquirer {
readonly name: string;
private readonly options: ErrorMonitorAcquirerOptions<TIssue> & {
minOccurrences: number;
maxIssuesPerTick: number;
};
private timer: ReturnType<typeof setInterval> | null = null;
private busy = false;

constructor(options: ErrorMonitorAcquirerOptions<TIssue>) {
this.options = { minOccurrences: 5, maxIssuesPerTick: 3, ...options };
this.name = `errors:${options.provider.providerName}:${options.sourceId}`;
}

async start(): Promise<void> {
if (this.timer) return;
console.log(
`🔎 [${this.name}] polling every ${this.options.intervalSeconds}s ` +
`(min occurrences: ${this.options.minOccurrences}, max per tick: ${this.options.maxIssuesPerTick})`,
);
await this.tick();
this.timer = setInterval(() => void this.tick(), this.options.intervalSeconds * 1000);
}

stop(): void {
if (this.timer) {
clearInterval(this.timer);
this.timer = null;
}
}

async tick(): Promise<void> {
if (this.busy) return;
this.busy = true;

const { queue, provider, minOccurrences, maxIssuesPerTick, verbose } = this.options;
const dedupeSource = this.name;
try {
const issues = await provider.fetchIssues();
let handled = 0;
for (const issue of issues) {
if (handled >= maxIssuesPerTick) break;

if (!Number.isFinite(issue.occurrenceCount) || issue.occurrenceCount < minOccurrences) {
if (verbose) {
console.log(
` [${this.name}] skipping ${issue.displayId}: only ${issue.occurrenceCount} occurrence(s); need ${minOccurrences}`,
);
}
continue;
}
const validity = provider.validateIssue(issue);
if (!validity.valid) {
if (verbose) {
console.log(
` [${this.name}] skipping ${issue.displayId}: ${validity.reason ?? "not actionable"}`,
);
}
continue;
}

// Claim atomically before execution so competing watcher processes
// cannot both submit the same issue. A capacity deferral is explicitly
// unclaimed so it can run on a later tick.
if (!queue.tryMarkProcessed(dedupeSource, issue.externalId)) continue;
console.log(`\n📌 [${this.name}] ${issue.displayId}: ${issue.title}`);
const result = await this.options.executeTask(issue, provider.buildTaskMarkdown(issue));
if (result === "deferred") {
queue.unmarkProcessed(dedupeSource, issue.externalId);
console.log(`⏳ [${this.name}] ${issue.displayId} deferred; it will be retried`);
break;
}
handled++;
console.log(
result
? `✅ [${this.name}] fix for ${issue.displayId} completed`
: `⚠️ [${this.name}] fix for ${issue.displayId} did not complete cleanly`,
);
}
} catch (error) {
console.warn(`⚠️ [${this.name}] polling tick failed: ${(error as Error).message}`);
} finally {
this.busy = false;
}
}
}
Loading
Loading