fix(framework): remove the harness system prompt for CLI agents (AI-1034) by seanoliver · Pull Request #241 · supabase/evals

seanoliver · 2026-08-25T22:17:43Z

This is stacked on #240. Review that first; this diff is against it. No results are touched.

Problem

We wrote our own system prompt into every eval run, for all four agents:

You are an agent solving a Supabase eval task in a Linux workspace. Use the provided tools to inspect and modify the workspace and run commands. When you are done, end your turn with a short summary of what you did.

Claude Code, Codex and OpenCode ship their own system prompt. Ours was layered on top, and Codex and OpenCode have no system-prompt flag, so for them it went onto the user prompt.
It described bash and files_read, which only the in-process ai-sdk agent has.
"End your turn with a short summary" shapes agentReport, which the LLM judge reads. That is a scoring change.
It says Supabase, and "project". An agent that would otherwise pick a database has been handed the answer (Base prompt #164).

Changes

Agent	Before	After
`ai-sdk`	framing + tool list + skills list	one neutral line + tool list + skills list
`claude-code`	ours appended to its own	nothing
`codex`	ours prepended to the user prompt	nothing
`opencode`	ours prepended to the user prompt	nothing

CLI agents take no system prompt at all. createCliAgent no longer stages a system-prompt file or passes a path to the runner; --append-system-prompt-file and the Codex/OpenCode prepend are gone. A non-empty systemPrompt for a CLI agent throws.
The ai-sdk base prompt is now You are an agent. Use the provided tools to complete the task. It is the same in both modes. No Supabase, no project, no stopping instructions.
buildToolSurfaceAddendum and buildSkillsPrompt return '' for every CLI agent. The bare sandbox no longer takes an agent or returns a promptAddendum.
Prompt assembly moves out of run-eval.ts into system-prompt.ts so it can be unit tested.
Each run records systemPrompt in its raw result.json. export-results drops it, so nothing reaches the site.

Since the first review: the system-prompt-file plumbing for CLI agents is deleted rather than made optional, and the ai-sdk base prompt is rewritten for #164. The branch is re-applied on the current #240 (the retry loop is gone there), so the commits are new. #240 now has main merged in, which brings the usage and duration fields from #270.

Testing

pnpm --filter @supabase-evals/core test, pnpm --filter @supabase-evals/sandbox test, pnpm --filter @supabase-evals/framework test, pnpm --filter @supabase-evals/web test: all pass locally.
typecheck for framework, sandbox and web passes. biome check . is clean.
Unverified against real CLIs since the deletion. The earlier sandbox runs (Claude Code 32785480269, Codex 32880822811, OpenCode 32880839623) recorded an empty prompt and still loaded the Supabase skills, and the path they exercised (no flag, task alone) is the one that remains.

Score movement

Full benchmark on both branches before this revision, artifact only, nothing published. Both used runs=2.

	#240, no prompt change	#241, prompt removed
all pairs	86.0%	84.7%	21 flips, 12 down 9 up
`claude-code`	92.3%	92.3%	no change
`codex`	81.5%	75.0%	down 6.5 points
`opencode`	82.6%	89.1%	up 6.5 points

This looks like run-to-run variability. Once #229 lands and scores average across runs, the numbers should settle.

Follow-ups

Re-run the benchmark comparison once feat: average eval scores across independent runs #229 lands and the numbers can carry attribution.
Drop the "end your turn with a short summary" line from evals/resolve-performance-001-slow-query-cpu-spike/PROMPT.md:15, the last one left. It sits in the task text so it hits every agent equally, and removing it invalidates that eval's recorded results, so it should be a separate change.

Ref AI-1034

vercel · 2026-08-25T22:17:51Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 8, 2026 8:51pm UTC

mattrossman · 2026-09-07T16:57:53Z

Tested running pnpm eval -- --eval build-cli-002-declarative-schema --experiment codex-gpt-5.4-mini and I see Codex reading its natively supported .agents/skills files now instead of Claude's.

mattrossman · 2026-09-07T17:08:33Z

+  if (agent !== 'ai-sdk') return '';
+  if (mode === 'local-stack') {
+    return (
+      'You are an agent solving a Supabase eval task in a Linux workspace. ' +


Could we also tie in the fix for #164 here? I.e. we shouldn't mention that this is a Supabase eval or that there is a "project" needing to be modified, as that could give an unfair advantage to the non terminal-agent experiments.

I recognize it's a minor issue at this point, since only the AI SDK harness is affected here, which is no longer used in our benchmarks / regression runs. Might end up taking it out entirely. But also easy to clean up while we're here.

Done. The base prompt is now one line, the same in both modes: You are an agent. Use the provided tools to complete the task. No Supabase, project, or eval in it, and a test asserts that.

mattrossman · 2026-09-07T17:23:27Z

+    process.env[API_KEY_ENV_VAR] = 'k';
+  });
+
+  it('stages a system prompt and hands its path to the runner', async () => {


Do we still need to support system prompt overrides for terminal agents?

Seems like that's now legacy behavior we won't want to support going forward, I think there's a cleanup / code deletion opportunity here. It's a bit convoluted to support the plumbing of writing a system prompt to a file that's read from inside the container.

The only use case I can think of is the executor example cited in the PR, but (a) the only experiments that use executor are AI SDK ones which don't need this system prompt file workaround (b) we don't use the executor experiments, they're just POC placeholders (c) we could achieve the same thing with an addendum on the user prompt.

Agreed. It's deleted: createCliAgent no longer stages a system prompt file or passes a path to the runner, and the three runners lost their flag/prepend branches. A non-empty systemPrompt for a CLI agent now throws instead of being dropped, so pairing the executor MCP with a CLI agent fails loudly. If anyone wants that pairing later, the protocol text goes on the user prompt as you suggested.

mattrossman · 2026-09-07T17:34:29Z

    agentReport: string;
    stoppedReason: string;
+    /**
+     * The exact system prompt handed to the agent (`''` when it got none). CLI


If we end up keeping this for terminal agents, I'd clarify here that it's more of an optional thing appended to the system prompt, not replacing it with this exact string alone.

This is moot now that the override is gone. What remains here is that each run records the systemPrompt it sent, '' for CLI agents.

createCliAgent no longer stages a system prompt file or hands a path to the runner. Claude Code drops --append-system-prompt-file; Codex and OpenCode send the task alone instead of prepending a block to the user prompt. A non-empty systemPrompt now throws rather than being dropped.

…034) buildToolSurfaceAddendum and buildSkillsPrompt return '' for every CLI harness: those describe in-process tools only aiSdkAgent has, and each CLI discovers the installed skills itself. The bare sandbox no longer takes an agent or returns a promptAddendum.

…s (AI-1034) Prompt assembly moves to system-prompt.ts so it can be unit tested. The ai-sdk base prompt no longer names Supabase, a project, or how to end the turn (#164), and is the same in both modes. CLI agents get an empty prompt. Each run records the system prompt it sent.

…arness-native-prompts # Conflicts: # apps/framework/harness/run-eval.ts # packages/core/src/agents/codex/runner.test.ts

This was referenced Aug 25, 2026

fix(sandbox): install skills natively per harness (AI-1034) #240

Open

fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034) #180

Closed

seanoliver marked this pull request as ready for review August 25, 2026 23:38

seanoliver requested review from a team and mattrossman and removed request for mattrossman August 25, 2026 23:49

mattrossman reviewed Sep 7, 2026

View reviewed changes

seanoliver added 3 commits September 8, 2026 13:50

seanoliver force-pushed the sean/ai-1034-harness-native-prompts branch from 5bed078 to 3cefa60 Compare September 8, 2026 20:51

seanoliver added 2 commits September 8, 2026 14:00

Merge branch 'sean/ai-1034-native-skills-install' into sean/ai-1034-h…

b737d6d

…arness-native-prompts # Conflicts: # apps/framework/harness/run-eval.ts # packages/core/src/agents/codex/runner.test.ts

test(core): restore the usage tests after the merge

17c738c

seanoliver requested a review from mattrossman September 8, 2026 21:16

Navigation Menu

Uh oh!

fix(framework): remove the harness system prompt for CLI agents (AI-1034) - #241

fix(framework): remove the harness system prompt for CLI agents (AI-1034)#241
seanoliver wants to merge 5 commits into
sean/ai-1034-native-skills-installfrom
sean/ai-1034-harness-native-prompts

seanoliver commented Aug 25, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

mattrossman commented Sep 7, 2026

Uh oh!

mattrossman Sep 7, 2026

Uh oh!

seanoliver Sep 8, 2026

Uh oh!

mattrossman Sep 7, 2026

Uh oh!

seanoliver Sep 8, 2026

Uh oh!

mattrossman Sep 7, 2026

Uh oh!

seanoliver Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

seanoliver commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Testing

Score movement

Follow-ups

Uh oh!

vercel Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattrossman commented Sep 7, 2026

Uh oh!

mattrossman Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

seanoliver Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

mattrossman Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

seanoliver Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

mattrossman Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

seanoliver Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

seanoliver commented Aug 25, 2026 •

edited

Loading

vercel Bot commented Aug 25, 2026 •

edited

Loading