Skip to content

feat(prompt): keep the system prompt stable across a resume - #907

Closed
breedx wants to merge 1 commit into
mpfaffenberger:mainfrom
breedx:feat/stable-prompt-prefix-on-resume
Closed

breedx wants to merge 1 commit into
mpfaffenberger:mainfrom
breedx:feat/stable-prompt-prefix-on-resume

Conversation

@breedx

@breedx breedx commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

The problem

pydantic-ai stamps the assembled system prompt into instructions on every request message, and instructions is the provider's cache prefix. Two things in the current assembly change between turns, so a long conversation misses that cache on every turn — precisely when it's worth having.

Measured on one long-running thread before this change: three turns under three different identities, with the instructions block moving 5571 → 6129 → 6175 characters. Every turn re-sent, and re-paid for, context that hadn't changed.

Three changes

1. load_prompt fragments are gathered once per conversation.

A fragment that grows between turns invalidates the prefix every turn. A memory/recall plugin is the worst case — it recalls what the previous turn wrote, so it grows exactly as the conversation gets long. The system prompt is the CONTRACT and shouldn't change mid-conversation; recall is CONTEXT and belongs in the message stream. Turn one still gets it.

Gathered on the first call rather than keyed on "is the history empty": the setup path calls the assembly more than once before any history exists (_estimate_context_overhead is one such caller), so a history-keyed cache would still re-poll and still drift, only earlier. None means "not gathered yet"; [] is a real answer meaning no plugin contributed. clear_message_history is the single reset.

2. Identity belongs to the conversation, not the process.

__init__ mints self.id as a fresh uuid4 and renders it into the prompt, so any front door running one turn per process changes identity mid-conversation. Besides rewriting the cache prefix, the prompt tells the agent to use that id "for claiming task ownership or coordination with other agents" — and an id that doesn't outlive the turn can't own anything.

It now travels in the session metadata, on both the lifecycle and autosave paths (quick-resume reads what autosave writes). Backward compatible both ways: a session written before this has no agent_id and the resuming agent keeps its own. The id is type-checked before being recorded, because the sidecar is JSON written after the history — an unserialisable id must not turn a save into a half-write and lose the user's conversation to a cosmetic field.

3. set_message_history is the one resume door.

Every front end that continues a conversation arrives there, so the behaviour lives there rather than in restore_named_session. Hanging it off one caller fixes exactly that caller — an earlier attempt did precisely that, passed its own tests, and left every other front end unchanged.

The prompt is inherited, not recomputed: the restored history already knows what it was built with, and that string is what the provider cached. Identity is passed in rather than parsed back out of the prompt text, so there's one representation instead of two that can drift — the rendered line shows only six characters of a uuid, so the round trip is lossy, and recovering it would make user-facing wording load-bearing.

Ordering of scoped additions

temporary_system_prompt_addition text is appended after the identity line, where the strip that produces the adoptable body already discards it. Ephemeral text is unadoptable by construction, and body + identity stay a maximal stable prefix.

This matters because the headless loop calls set_message_history after the scoped block exits — the list is empty by then, so nothing on the object can say which bytes were ephemeral. Position is the only signal that survives the trip through storage into another process. Without it, the headless autonomy instruction was dropped on a resumed -p run and frozen permanently into the prefix in the opposite order.

Additions are also stripped from an adopted body on the way in, not only kept out on the way out: the prompt is persisted, so a session written by an older build would otherwise re-apply the instruction to every later turn — including interactive ones, telling a user's own session never to ask them for confirmation. The exact known string is removed and nothing looser.

_strip_identity_prompt uses rpartition deliberately: _builder and subagent invocation both append after the identity line, so the last marker is the correct split point. A test pins that, since swapping it for partition otherwise passes the whole suite.

Verification

  • 7720 passed, 33 skipped
  • tests/command_line/test_autosave_menu.py::TestDisplayResumedHistory::test_displays_last_n_messages fails on a clean checkout of main too — pre-existing, unrelated, untouched here
  • ruff check + ruff format --check clean
  • Regression tests verified to fail without the fix; the rpartition test verified to fail under exactly that mutation
  • Exercised against real ModelRequest/ModelResponse objects rather than stand-ins

Happy to split this into smaller PRs if you'd prefer to take the pieces separately.

pydantic-ai stamps the assembled system prompt into `instructions` on every
request message, and `instructions` is the provider's cache prefix. Two
things in the current assembly change between turns, so a long conversation
misses that cache on every turn -- precisely when it is worth having.

Measured on one long-running thread before this change: three turns under
three different identities, with the instructions block moving 5571 -> 6129
-> 6175 characters. Every turn re-sent, and re-paid for, context that had
not changed.

`load_prompt` fragments are gathered once per conversation
------------------------------------------------------------------
A fragment that grows between turns invalidates the prefix every turn. A
memory/recall plugin is the worst case: it recalls what the previous turn
wrote, so it grows exactly as the conversation gets long. The system prompt
is the CONTRACT and should not change mid-conversation; recall is CONTEXT
and belongs in the message stream. Turn one still gets it; a test pins that.

Gathered on the FIRST call rather than keyed on "is the history empty": the
setup path calls the assembly more than once before any history exists
(`_estimate_context_overhead` is one such caller), so a history-keyed cache
would still re-poll and still drift, only earlier. `None` means "not
gathered yet" and `[]` is a real answer meaning no plugin contributed;
conflating them re-polls forever for conversations that have none.
`clear_message_history` is the single reset, so the prefix changes exactly
when the conversation does.

Identity belongs to the conversation, not the process
------------------------------------------------------------------
`__init__` mints `self.id` as a fresh uuid4 and renders it into the prompt,
so any front door running one turn per process changes identity
mid-conversation. Besides rewriting the cache prefix, the prompt tells the
agent to use that id "for claiming task ownership or coordination with
other agents" -- and an id that does not outlive the turn cannot own
anything. It now travels in the session metadata, on both the lifecycle and
autosave paths, since quick-resume reads what autosave writes.

Backward compatible in both directions: a session written before this has
no `agent_id` and the resuming agent keeps its own. The id is type-checked
before being recorded because the sidecar is JSON written AFTER the history
-- an unserialisable id must not turn a save into a half-write and lose the
user's conversation to a cosmetic field.

`set_message_history` is the one resume door
------------------------------------------------------------------
Every front end that continues a conversation arrives there, so the
behaviour lives there rather than in `restore_named_session`. Hanging it off
one caller fixes exactly that caller: an earlier attempt did precisely that,
passed its own tests, and left every embedding front end unchanged.

The prompt is INHERITED, not recomputed: the restored history already knows
what it was built with, and that string is what the provider cached.
Identity is passed in rather than parsed back out of the prompt text, so
there is one representation instead of two that can drift -- the rendered
line shows only six characters of a uuid, so the round trip is lossy, and
recovering it would make user-facing wording load-bearing.

Scoped runtime additions (`temporary_system_prompt_addition`) are appended
after the identity line, where the strip that produces the adoptable body
already discards them. Ephemeral text is therefore unadoptable by
construction, and body + identity remain a maximal stable prefix. Without
this the headless autonomy instruction was dropped on a resumed `-p` run and
frozen permanently into the prefix in the opposite order.

Scoped additions are removed from an adopted body on the way in, not only
kept out of it on the way out. The prompt is persisted, so a session
written by a build that stored the addition inside the durable part would
otherwise re-apply it to every later turn -- including interactive ones,
telling a user's own session never to ask them for confirmation, with
/clear as the only escape. The exact known string is removed and nothing
looser: a heuristic over prompt prose could eat authored text that merely
resembles it.

`_strip_identity_prompt` uses rpartition deliberately. Downstream code
(`_builder`, subagent invocation) appends after the identity line, so the
LAST marker is the correct split point; a test pins that, because swapping
it for partition otherwise passes the whole suite.

Verified against real ModelRequest/ModelResponse objects, not stand-ins.
@breedx
breedx marked this pull request as draft September 3, 2026 15:26
@breedx

breedx commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Closing this outdated implementation. Follow-up work will separate durable conversation identity from stable prompt-prefix persistence, restore through the common history entry point, and carry explicit structured state rather than infer boundaries from rendered prompt text. Replacement proposals will include real fresh-process resume and compaction regressions; this closure does not mean the underlying issue is resolved.

@breedx

breedx commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

The corrected replacements are now available: #925 preserves agent identity across fresh-process resume, and #926 preserves the structural prompt prefix without freezing temporary runtime instructions. They split the two concerns so identity can be reviewed and accepted first.

This supersedes the proposal in this closed PR; it does not reopen its old implementation. No measured provider cache-hit or cost-saving claim is made.

@breedx

breedx commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Update to the replacement links: #926 is now the single combined identity-and-structural-prefix proposal, ready for review. It includes all commits from #925, which was closed as superseded, without changing the reviewed code or rewriting history. #926 explains the related #832/#839 preparation and delivery work and retains the recorded validation and CI limitations. Please use #926 for further review; this original PR remains closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant