Conversation
pydantic-ai stamps the assembled system prompt into `instructions` on every request message, and `instructions` is the provider's cache prefix. Two things in the current assembly change between turns, so a long conversation misses that cache on every turn -- precisely when it is worth having. Measured on one long-running thread before this change: three turns under three different identities, with the instructions block moving 5571 -> 6129 -> 6175 characters. Every turn re-sent, and re-paid for, context that had not changed. `load_prompt` fragments are gathered once per conversation ------------------------------------------------------------------ A fragment that grows between turns invalidates the prefix every turn. A memory/recall plugin is the worst case: it recalls what the previous turn wrote, so it grows exactly as the conversation gets long. The system prompt is the CONTRACT and should not change mid-conversation; recall is CONTEXT and belongs in the message stream. Turn one still gets it; a test pins that. Gathered on the FIRST call rather than keyed on "is the history empty": the setup path calls the assembly more than once before any history exists (`_estimate_context_overhead` is one such caller), so a history-keyed cache would still re-poll and still drift, only earlier. `None` means "not gathered yet" and `[]` is a real answer meaning no plugin contributed; conflating them re-polls forever for conversations that have none. `clear_message_history` is the single reset, so the prefix changes exactly when the conversation does. Identity belongs to the conversation, not the process ------------------------------------------------------------------ `__init__` mints `self.id` as a fresh uuid4 and renders it into the prompt, so any front door running one turn per process changes identity mid-conversation. Besides rewriting the cache prefix, the prompt tells the agent to use that id "for claiming task ownership or coordination with other agents" -- and an id that does not outlive the turn cannot own anything. It now travels in the session metadata, on both the lifecycle and autosave paths, since quick-resume reads what autosave writes. Backward compatible in both directions: a session written before this has no `agent_id` and the resuming agent keeps its own. The id is type-checked before being recorded because the sidecar is JSON written AFTER the history -- an unserialisable id must not turn a save into a half-write and lose the user's conversation to a cosmetic field. `set_message_history` is the one resume door ------------------------------------------------------------------ Every front end that continues a conversation arrives there, so the behaviour lives there rather than in `restore_named_session`. Hanging it off one caller fixes exactly that caller: an earlier attempt did precisely that, passed its own tests, and left every embedding front end unchanged. The prompt is INHERITED, not recomputed: the restored history already knows what it was built with, and that string is what the provider cached. Identity is passed in rather than parsed back out of the prompt text, so there is one representation instead of two that can drift -- the rendered line shows only six characters of a uuid, so the round trip is lossy, and recovering it would make user-facing wording load-bearing. Scoped runtime additions (`temporary_system_prompt_addition`) are appended after the identity line, where the strip that produces the adoptable body already discards them. Ephemeral text is therefore unadoptable by construction, and body + identity remain a maximal stable prefix. Without this the headless autonomy instruction was dropped on a resumed `-p` run and frozen permanently into the prefix in the opposite order. Scoped additions are removed from an adopted body on the way in, not only kept out of it on the way out. The prompt is persisted, so a session written by a build that stored the addition inside the durable part would otherwise re-apply it to every later turn -- including interactive ones, telling a user's own session never to ask them for confirmation, with /clear as the only escape. The exact known string is removed and nothing looser: a heuristic over prompt prose could eat authored text that merely resembles it. `_strip_identity_prompt` uses rpartition deliberately. Downstream code (`_builder`, subagent invocation) appends after the identity line, so the LAST marker is the correct split point; a test pins that, because swapping it for partition otherwise passes the whole suite. Verified against real ModelRequest/ModelResponse objects, not stand-ins.
|
Closing this outdated implementation. Follow-up work will separate durable conversation identity from stable prompt-prefix persistence, restore through the common history entry point, and carry explicit structured state rather than infer boundaries from rendered prompt text. Replacement proposals will include real fresh-process resume and compaction regressions; this closure does not mean the underlying issue is resolved. |
|
The corrected replacements are now available: #925 preserves agent identity across fresh-process resume, and #926 preserves the structural prompt prefix without freezing temporary runtime instructions. They split the two concerns so identity can be reviewed and accepted first.
This supersedes the proposal in this closed PR; it does not reopen its old implementation. No measured provider cache-hit or cost-saving claim is made. |
|
Update to the replacement links: #926 is now the single combined identity-and-structural-prefix proposal, ready for review. It includes all commits from #925, which was closed as superseded, without changing the reviewed code or rewriting history. #926 explains the related #832/#839 preparation and delivery work and retains the recorded validation and CI limitations. Please use #926 for further review; this original PR remains closed. |
The problem
pydantic-ai stamps the assembled system prompt into
instructionson every request message, andinstructionsis the provider's cache prefix. Two things in the current assembly change between turns, so a long conversation misses that cache on every turn — precisely when it's worth having.Measured on one long-running thread before this change: three turns under three different identities, with the instructions block moving 5571 → 6129 → 6175 characters. Every turn re-sent, and re-paid for, context that hadn't changed.
Three changes
1.
load_promptfragments are gathered once per conversation.A fragment that grows between turns invalidates the prefix every turn. A memory/recall plugin is the worst case — it recalls what the previous turn wrote, so it grows exactly as the conversation gets long. The system prompt is the CONTRACT and shouldn't change mid-conversation; recall is CONTEXT and belongs in the message stream. Turn one still gets it.
Gathered on the first call rather than keyed on "is the history empty": the setup path calls the assembly more than once before any history exists (
_estimate_context_overheadis one such caller), so a history-keyed cache would still re-poll and still drift, only earlier.Nonemeans "not gathered yet";[]is a real answer meaning no plugin contributed.clear_message_historyis the single reset.2. Identity belongs to the conversation, not the process.
__init__mintsself.idas a fresh uuid4 and renders it into the prompt, so any front door running one turn per process changes identity mid-conversation. Besides rewriting the cache prefix, the prompt tells the agent to use that id "for claiming task ownership or coordination with other agents" — and an id that doesn't outlive the turn can't own anything.It now travels in the session metadata, on both the lifecycle and autosave paths (quick-resume reads what autosave writes). Backward compatible both ways: a session written before this has no
agent_idand the resuming agent keeps its own. The id is type-checked before being recorded, because the sidecar is JSON written after the history — an unserialisable id must not turn a save into a half-write and lose the user's conversation to a cosmetic field.3.
set_message_historyis the one resume door.Every front end that continues a conversation arrives there, so the behaviour lives there rather than in
restore_named_session. Hanging it off one caller fixes exactly that caller — an earlier attempt did precisely that, passed its own tests, and left every other front end unchanged.The prompt is inherited, not recomputed: the restored history already knows what it was built with, and that string is what the provider cached. Identity is passed in rather than parsed back out of the prompt text, so there's one representation instead of two that can drift — the rendered line shows only six characters of a uuid, so the round trip is lossy, and recovering it would make user-facing wording load-bearing.
Ordering of scoped additions
temporary_system_prompt_additiontext is appended after the identity line, where the strip that produces the adoptable body already discards it. Ephemeral text is unadoptable by construction, and body + identity stay a maximal stable prefix.This matters because the headless loop calls
set_message_historyafter the scoped block exits — the list is empty by then, so nothing on the object can say which bytes were ephemeral. Position is the only signal that survives the trip through storage into another process. Without it, the headless autonomy instruction was dropped on a resumed-prun and frozen permanently into the prefix in the opposite order.Additions are also stripped from an adopted body on the way in, not only kept out on the way out: the prompt is persisted, so a session written by an older build would otherwise re-apply the instruction to every later turn — including interactive ones, telling a user's own session never to ask them for confirmation. The exact known string is removed and nothing looser.
_strip_identity_promptusesrpartitiondeliberately:_builderand subagent invocation both append after the identity line, so the last marker is the correct split point. A test pins that, since swapping it forpartitionotherwise passes the whole suite.Verification
tests/command_line/test_autosave_menu.py::TestDisplayResumedHistory::test_displays_last_n_messagesfails on a clean checkout ofmaintoo — pre-existing, unrelated, untouched hereruff check+ruff format --checkcleanrpartitiontest verified to fail under exactly that mutationModelRequest/ModelResponseobjects rather than stand-insHappy to split this into smaller PRs if you'd prefer to take the pieces separately.