Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/code/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Observability Dashboard"
description: "A local web dashboard for worker run history: per-task timelines, stage-by-stage outcomes, aggregate stats, run retries, and worker logs"
section: "Server Automation"
order: 2
dateModified: 2026-08-27
dateModified: 2026-08-29
---

# Observability Dashboard
Expand All @@ -28,11 +28,14 @@ The standalone command reads the database in read-only mode, so it is safe to ru

## What it shows

- **Run list**: every run with its status, task key or automation id, origin (tracker task, PR mention, scheduled, or estimate), agent harness, PR link, and duration. The task key links straight to the tracker ticket when the tracker's URL can be derived from your configuration; filter by status or origin (`origin=scheduled` isolates automation runs; `origin=estimate` isolates story-point sweeps).
- **Run list**: every run with its status, task key or automation id, origin (tracker task, PR mention, scheduled, or estimate), agent harness, git branch, PR link, and duration. The task key links straight to the tracker ticket when the tracker's URL can be derived from your configuration; filter by status or origin (`origin=scheduled` isolates automation runs; `origin=estimate` isolates story-point sweeps). The harness and branch are recorded when the run starts, so runs from before they were recorded show `–`.
- **Run detail**: the task key header (linked to its tracker ticket when possible) plus a snapshot of the original task description — captured when the run started and rendered as markdown — followed by a stage-by-stage timeline: the feasibility verdict, the implementation summary, each self-review iteration, each human change request and how it was handled, and the final outcome.
- **Agent PRs**: every pull request the worker created that is still open — repo, PR number, branch, linked ticket key, and age — with a direct link to each PR on GitHub. The worker reconciles this list with GitHub on every poll cycle, so PRs merged or closed outside the worker drop out automatically.
- **Stats**: runs per week, success and escalation rates, median run duration, and a per-harness breakdown over a selectable window (7, 30, or 90 days, or all time).
- **Logs**: the most recent worker log lines (timestamp, severity, message), filterable by level (`everything` / `warnings` / `errors`) with a search box over the loaded window. Lines that mention a task key link straight to that task's latest run in the Runs view.
- **Worker status**: whether the daemon is running, queued and failed events, open agent PRs, and per-source poll cursors.
- **Worker status**: whether the daemon is running, queued and failed events, the open agent PR count (linked to the Agent PRs view), and per-source poll cursors.

Worker liveness is read from the daemon's lock file, in the project's `.devintern-code/` directory and in the workspace home (`~/.devintern`), so the header is accurate whether the worker runs in the foreground or as a launchd/systemd service. When no lock file is found in either location, the header says "worker status unknown" instead of "stopped" — the dashboard may simply be pointed at a different directory than the worker.

Success and escalation rates are computed over finished runs only. Run duration is measured from pickup to PR creation and is a proxy for ticket-to-PR time. Merge rate is not shown yet: the worker records PRs as open or closed but does not track merges separately.

Expand Down Expand Up @@ -106,9 +109,10 @@ The dashboard is backed by a small read-only JSON API you can use directly, for
| --------------------------- | --------------------------------------------------------------------- |
| `GET /api/runs` | Paginated run list (`limit`, `offset`, `status`, `origin`, `taskKey`); `origin=scheduled` and `origin=estimate` are supported |
| `GET /api/runs/:id` | One run with its stage timeline and retry metadata |
| `GET /api/agent-prs` | Open agent-created PRs with GitHub links, branches, and ticket keys |
| `POST /api/runs/:id/retry` | Schedule a re-run of the task behind a failed/escalated/abandoned run (requires sign-in) |
| `GET /api/stats?window=30d` | Aggregate stats (`7d`, `30d`, `90d`, or `all`) |
| `GET /api/worker` | Worker liveness, queue counts, agent PRs, poll cursors |
| `GET /api/worker` | Worker liveness (`running`, `stopped`, or `unknown`), queue counts, agent PR counts, poll cursors |
| `GET /api/logs` | Recent worker log entries (`limit` 1–1000, default 500; `level` all/info/warn/error) |
| `GET /api/health` | Health check |

Expand Down
2 changes: 1 addition & 1 deletion docs/code/worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ Unattended automation is exactly where sandboxing the agent matters most: set `A

## Review feedback on the agent's PRs

In polling mode the worker also watches the pull requests it created (no webhook needed). When a human requests changes or leaves new inline review comments on one of the agent's own PRs, the worker addresses the feedback automatically; no mention is required on its own PRs. Closed and merged PRs leave the watch list on their own.
In polling mode the worker also watches the pull requests it created (no webhook needed). When a human requests changes or leaves new inline review comments on one of the agent's own PRs, the worker addresses the feedback automatically; no mention is required on its own PRs. Closed and merged PRs leave the watch list on their own: the watch list is reconciled with GitHub on every poll cycle, so PRs merged or closed outside the worker (and PRs that disappear because a repository was renamed, transferred, or deleted) drop out of the open count within one poll.

The watch list is scoped to repos listed in `workspace.toml`. Registry entries for any other repo — typically left behind when a repository is renamed or transferred — are unwatched automatically at startup instead of being polled (and failing auth) forever.

Expand Down
4 changes: 4 additions & 0 deletions packages/code/src/dashboard-server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { join, normalize, resolve } from "path";

import {
DashboardData,
handleAgentPrs,
handleLogs,
handleRetryRun,
handleRuns,
Expand Down Expand Up @@ -149,6 +150,9 @@ export function startDashboardServer(
if (pathname === "/api/worker") {
return json(handleWorkerStatus(data));
}
if (pathname === "/api/agent-prs") {
return json(handleAgentPrs(data));
}
if (pathname === "/api/logs") {
return json(handleLogs(data, url.searchParams));
}
Expand Down
6 changes: 6 additions & 0 deletions packages/code/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ import {
RunStore,
beginRun,
endRun,
recordRunBranch,
recordRunPr,
recordRunStage,
recordRunTicket,
Expand Down Expand Up @@ -1441,6 +1442,8 @@ async function processSingleTask(taskKey: string, taskIndex = 0, totalTasks = 1)
origin: scheduledAutomationId ? "scheduled" : "task",
taskKey: workflowKey,
tracker: trackerName,
// The harness that will implement this run (resolved at startup).
harness: resolvedAgent.harness.name,
...(scheduledAutomationId ? { automationId: scheduledAutomationId } : {}),
// Ticket link for remote trackers only: markdown-file inputs and
// materialized automation prompts have no tracker page, and deriving
Expand Down Expand Up @@ -1682,6 +1685,9 @@ async function processSingleTask(taskKey: string, taskIndex = 0, totalTasks = 1)

if (branchResult.success) {
console.log(`✅ ${branchResult.message}`);
// Record the actual branch (it can gain an attempt suffix) so the
// dashboard shows which branch a run worked on.
recordRunBranch(branchResult.branchName);
} else {
// Branch creation failed - this is critical for safety
console.error(`\n❌ Failed to create feature branch: ${branchResult.message}`);
Expand Down
1 change: 1 addition & 0 deletions packages/code/src/lib/address-review.ts
Original file line number Diff line number Diff line change
Expand Up @@ -598,6 +598,7 @@ export async function addressReview(
repo: `${owner}/${repo}`,
prNumber,
branch: pr.head.ref,
harness: resolveHarness({ warnDeprecated: false }).harness.name,
});
recordRunStage("change_request", {
status: "succeeded",
Expand Down
151 changes: 151 additions & 0 deletions packages/code/src/lib/agent-pr-reconciler.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
/**
* Agent PR Reconciler
*
* Keeps the `agent_prs` registry (worker-state.ts) truthful against GitHub.
* The dashboard's "N agent PRs open" count reads this registry, so a PR
* closed or deleted outside the worker (merged by a human, closed from the
* GitHub UI, repo renamed or transferred) must leave the registry within one
* poll cycle instead of being counted as open forever.
*
* The review poller fetches every watched PR's state on each tick anyway, so
* `applyAgentPrFetch` folds reconciliation into that fetch (no extra API
* call). `reconcileOpenAgentPrs` is the batch form: one conditional GET per
* watched PR — 304s are rate-limit-free, so the steady-state sync costs
* nothing against GitHub's limits (App installs or PATs alike) — and it
* shares its fetch results with the caller's per-PR poll loop via `fresh`.
*/

import type { AgentPr, WorkerState } from "./worker-state";

/** The subset of a GitHub PR payload the poller and reconciler rely on. */
export interface PolledPr {
state: string;
/** GitHub's computed merge state; `"dirty"` means merge conflicts. */
mergeable_state?: string;
head?: { sha: string; ref?: string; repo?: { full_name: string } | null };
base?: { sha: string; ref?: string };
}

/** Result of a conditional (ETag-cached) GitHub GET. */
export interface ConditionalResult<T> {
data: T | null;
etag?: string;
notModified: boolean;
/**
* GitHub answered 404: the PR or repo is gone (renamed, transferred,
* deleted, or the credential has no access). Such rows can never be
* fetched again and must not stay open in the registry.
*/
gone?: boolean;
}

/** GitHub access the reconciler needs (satisfied by the review poller's client). */
export interface AgentPrReconcileGitHub {
fetchPr(repo: string, prNumber: number, etag?: string): Promise<ConditionalResult<PolledPr>>;
}

/** One registry row closed by reconciliation. */
export interface AgentPrClosure {
repo: string;
prNumber: number;
/** Why the row was closed: the PR's GitHub state, or "gone from GitHub". */
reason: string;
}

export interface AgentPrReconcileSummary {
/** Open registry rows examined (foreign repos excluded). */
checked: number;
/** Rows left open because GitHub could not be reached this pass. */
failed: number;
closed: AgentPrClosure[];
}

/** Cursor source holding the PR-state ETag shared by poller and reconciler. */
export function agentPrStateCursorSource(repo: string, prNumber: number): string {
return `github:pr:${repo}#${prNumber}`;
}

/** Stable map key for a PR (same shape as the poller's in-memory cache). */
export function agentPrKey(repo: string, prNumber: number): string {
return `${repo.toLowerCase()}#${prNumber}`;
}

/**
* Apply one fetched PR state to the registry: persist the ETag cursor and
* close the row when the PR is no longer open on GitHub or is gone.
*
* @returns The closure record, or `null` when the PR stays watched.
*/
export function applyAgentPrFetch(
workerState: WorkerState,
pr: { repo: string; prNumber: number },
result: ConditionalResult<PolledPr>,
): AgentPrClosure | null {
if (!result.notModified && result.etag) {
workerState.setCursor(agentPrStateCursorSource(pr.repo, pr.prNumber), "state", result.etag);
}
if (result.gone || (result.data && result.data.state !== "open")) {
const reason = result.gone ? "gone from GitHub" : (result.data?.state ?? "closed");
workerState.markAgentPrClosed(pr.repo, pr.prNumber);
return { repo: pr.repo, prNumber: pr.prNumber, reason };
}
return null;
}

/**
* Reconcile the open-PR registry with GitHub.
*
* Every watched row that belongs to an allowed repo is checked once; rows
* whose PR is closed/merged or gone are closed. Failures (network errors,
* rate limits) leave the row open — it is retried next pass, never closed
* on missing information.
*
* @param options.workerState - Registry store
* @param options.github - GitHub client (the review poller's, so App auth applies)
* @param options.watched - Open registry rows to verify
* @param options.allowedRepos - Repos this worker manages; foreign rows are skipped
* @param options.etagFor - Stored ETag per PR; defaults to the shared `github:pr:` cursor
* @param options.fresh - Collect fetch results here (keyed by {@link agentPrKey})
* so the caller can poll each PR without a second request
*/
export async function reconcileOpenAgentPrs(options: {
workerState: WorkerState;
github: AgentPrReconcileGitHub;
watched: AgentPr[];
allowedRepos?: string[];
etagFor?: (repo: string, prNumber: number) => string | undefined;
fresh?: Map<string, ConditionalResult<PolledPr>>;
}): Promise<AgentPrReconcileSummary> {
const { workerState, github, watched, allowedRepos, fresh } = options;
const etagFor =
options.etagFor ??
((repo: string, prNumber: number) =>
workerState.getCursor(agentPrStateCursorSource(repo, prNumber))?.etag);

const summary: AgentPrReconcileSummary = { checked: 0, failed: 0, closed: [] };
for (const pr of watched) {
if (allowedRepos && allowedRepos.length > 0 && !allowedRepos.includes(pr.repo)) {
continue;
}
summary.checked += 1;
let result: ConditionalResult<PolledPr>;
try {
result = await github.fetchPr(pr.repo, pr.prNumber, etagFor(pr.repo, pr.prNumber));
} catch {
// Transient (network, rate limit): leave the row open and retry next
// pass — reconciliation must never close a row on missing information.
summary.failed += 1;
continue;
}
const closure = applyAgentPrFetch(workerState, pr, result);
if (closure) {
summary.closed.push(closure);
fresh?.delete(agentPrKey(pr.repo, pr.prNumber));
continue;
}
if (result.data && fresh) {
fresh.set(agentPrKey(pr.repo, pr.prNumber), result);
}
}
return summary;
}
94 changes: 91 additions & 3 deletions packages/code/src/lib/dashboard-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
*/

import { LockManager } from "./lock-manager";
import type { LockStatus } from "./lock-manager";
import { RunStore } from "./run-recorder";
import type { RunOrigin, RunRecord, RunStageRecord, RunStats, RunStatus } from "./run-recorder";
import {
Expand Down Expand Up @@ -111,6 +112,29 @@ export interface EnrichedLogEntry extends LogEntry {
runStatus?: RunStatus;
}

/** Worker liveness as reported by `GET /api/worker`. */
export type WorkerLiveness = "running" | "stopped" | "unknown";

export interface WorkerStatusView {
status: WorkerLiveness;
pid?: number;
startedAt?: string;
/** Lock file the status was read from; absent when undeterminable. */
lockFile?: string;
}

/** One open agent PR as served by `GET /api/agent-prs`. */
export interface OpenAgentPrView {
repo: string;
prNumber: number;
prUrl: string;
branch?: string;
taskKey?: string;
ticketUrl?: string;
createdAt: number;
updatedAt: number;
}

/**
* Lazily opened read-only view over the worker's SQLite database.
*
Expand Down Expand Up @@ -288,6 +312,28 @@ export class DashboardData {
return this.read({ open: 0, closed: 0 }, (stores) => stores.state.countAgentPrs());
}

/**
* Open agent-created PRs with their GitHub URLs (`GET /api/agent-prs`).
* Ticket links are frozen in the registry by the worker at PR-creation
* time (from the tracker configured then), so they survive tracker
* switches and stay correct even when this dashboard process runs
* without tracker configuration.
*/
getOpenAgentPrs(): OpenAgentPrView[] {
return this.read([], (stores) =>
stores.state.listOpenAgentPrs().map((pr) => ({
repo: pr.repo,
prNumber: pr.prNumber,
prUrl: `https://github.com/${pr.repo}/pull/${pr.prNumber}`,
branch: pr.branch,
taskKey: pr.taskKey,
ticketUrl: pr.ticketUrl,
createdAt: pr.createdAt,
updatedAt: pr.updatedAt,
})),
);
}

getCursors(): Cursor[] {
return this.read([], (stores) => stores.state.listCursors());
}
Expand Down Expand Up @@ -618,18 +664,48 @@ export function handleStats(data: DashboardData, params: URLSearchParams): ApiRe
return { status: 200, body: { window, stats } };
}

/**
* Resolve worker liveness from the daemon's lock file.
*
* The worker writes `.worker.lock` under `.devintern-code/` of its working
* directory (simple mode) or directly into the workspace home (fleet mode),
* so both locations are consulted. A readable lock with a dead pid (a stale
* lock left behind by a crashed worker) must not shadow a live lock in the
* other location: the live lock wins, and `stopped` is only reported when
* every readable lock belongs to a dead process. When no readable lock file
* exists in either location, liveness is `unknown`: the worker may be
* running elsewhere, and claiming "stopped" would be wrong.
*
* @param workingDir - Project root the dashboard was started from
*/
function resolveWorkerStatus(workingDir: string): WorkerStatusView {
const locks = [
LockManager.readLockStatus(workingDir, WORKER_LOCK_FILE),
LockManager.readLockStatus(resolveWorkspaceDir(), WORKER_LOCK_FILE, { plainDir: true }),
].filter((lock): lock is LockStatus => lock !== null);
if (locks.length === 0) {
return { status: "unknown" };
}
// A live lock beats a stale one regardless of which location it sits in.
const lock = locks.find((candidate) => candidate.running) ?? locks[0];
return {
status: lock.running ? "running" : "stopped",
pid: lock.pid,
startedAt: lock.startedAt,
lockFile: lock.path,
};
}

/**
* `GET /api/worker` — worker liveness, queue counts, agent PRs, poll cursors.
*
* @param data - Dashboard data source
*/
export function handleWorkerStatus(data: DashboardData): ApiResponse {
const lock = LockManager.readLockStatus(data.workingDir, WORKER_LOCK_FILE);
return {
status: 200,
body: {
worker:
lock === null ? null : { running: lock.running, pid: lock.pid, startedAt: lock.startedAt },
worker: resolveWorkerStatus(data.workingDir),
queue: data.getQueueStats(),
agentPrs: data.getAgentPrCounts(),
cursors: data.getCursors().map((cursor) => ({
Expand All @@ -643,6 +719,18 @@ export function handleWorkerStatus(data: DashboardData): ApiResponse {
};
}

/**
* `GET /api/agent-prs` — open agent-created PRs with GitHub links.
*
* The registry is reconciled with GitHub by the worker's review polling, so
* PRs merged or closed outside the worker drop out within one poll cycle.
*
* @param data - Dashboard data source
*/
export function handleAgentPrs(data: DashboardData): ApiResponse {
return { status: 200, body: { prs: data.getOpenAgentPrs() } };
}

/**
* `GET /api/logs` — the most recent worker log entries, tailed from the
* capture files with an entry-count bound.
Expand Down
Loading
Loading