fix: size-aware emergency truncation (billion-context-pi#12) - #133
Open
ranxianglei wants to merge 4 commits into
Open
fix: size-aware emergency truncation (billion-context-pi#12)#133ranxianglei wants to merge 4 commits into
ranxianglei wants to merge 4 commits into
Conversation
The emergency truncation hard-protected the last N messages (protectRecentMessages: 3), which was counterproductive when the most recent tool-results were the largest — e.g. decompress inline results. The 6 inlined blocks in the 2026-08-22 overflow were the newest messages, so the last 3 (~100K tokens) fell in the protected zone and were never truncated; only the older 3 were, which was insufficient. Now ALL tool-results are candidates (including recent ones), sorted largest-first. The size-based sort naturally preserves small recent messages and truncates large ones regardless of recency. protectRecentMessages is kept in the API for backward compatibility but no longer hard-excludes recent messages from candidacy.
…-08-23) A single 50,358-char bash toolResult (~31,475 tokens, ~24% of the effective 131,072-token window) entered context whole: it sat under every byte cap, and the usage-gated valves never fired because the kernel's own estimate read 51.8% while real usage was >100%. No percentage-gated valve can catch a single-message spike. Add truncate.maxToolResultTokens (null = auto: min(10% of modelContextLimit, 16384); 0 disables) and a tool-result-cap pipeline node that runs on EVERY turn and ignores recency protection: any tool-result estimated above the cap (CJK-aware tokenizer, same basis as the rest of the kernel) is rewritten head+tail with a `[acp: tool-result truncated, original ~N tokens]` marker. Placed after prune and before recommend/nudge so downstream token estimates reflect the capped sent view. Emergency truncation now skips already-capped texts so the two valves never stack markers.
…mergency re-truncation Review findings from del_mt5xrwcl_sb29 (2026-08-23): 1. [major] A legitimate tool-result merely QUOTING the cap-marker string escaped the hard cap entirely. Skip on marker only while the text still fits (<= cap * 1.25 margin for tokenizer drift). 2. [major] Non-finite maxToolResultTokens / modelContextLimit produced a NaN cap that replaced EVERY tool-result with the bare marker. Sanitized to fall back to auto. 3. [major] Auto cap min(10% x limit, 16384) moved with the learned limit -> prefix-cache break per change. Quantized down to powers of two (>=1024 step), so the truncation point moves only across power-of-two boundaries. 4. [major->minor] Emergency truncation skipped cap-marked messages, making them permanently untouchable at >=95% usage. Now re-truncates them (stacked markers acceptable: the prefix is broken by design at emergency). 5. toolResultCappedCount now surfaces on ProcessTurnResult for host observability; char prefilter skips re-tokenizing small tool-results. Fractional explicit caps clamp to >=1 (0.5 no longer silently disables). 482/482 tests (18 in toolresult-token-cap, +5 regression tests).
fix: hard per-tool-result token cap (billion-context-pi incident 2026-08-23)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The emergency truncation (src/truncate-tools.ts) hard-protected the last N messages (
protectRecentMessages: 3). This was counterproductive when the most recent tool-results were the largest — e.g.decompress inline:trueresults.In the 2026-08-22 overflow (billion-context-pi#12), the model inlined 6 blocks (574 msgs) in one turn at 21% usage. The 6 inlined tool-results were the newest messages, so the last 3 (~100K tokens) fell in the protected zone and were never truncated; only the older 3 were, which was insufficient to bring the context under the limit.
Fix
All tool-results are now candidates for truncation (including recent ones), sorted largest-first. The size-based sort naturally preserves small recent messages and truncates large ones regardless of recency.
protectRecentMessagesis kept in the API for backward compatibility but no longer hard-excludes recent messages from candidacy.Tests
tests/truncate-keep-filter.test.ts: the "protects recent messages" test now verifies a large recent message IS truncated (size-aware).Related