Fix prompt truncation logic #838
Conversation
|
/gcbrun exp -n dg -ag |
|
/gcbrun exp -n dg -ag |
|
Truncation looking good now: I reckon we can be even more aggressive on the amount of text to truncate. |
|
/gcbrun exp -n dg1 -ag |
|
Report looking alright: |
|
/gcbrun skip |
| total_tokens = self.estimate_token_num(raw_prompt_text) | ||
|
|
||
| # Allow buffer space for potential prompts that will be appended later. | ||
| allowed_tokens = self.MAX_INPUT_TOKEN // 10 - extra_tokens |
There was a problem hiding this comment.
why // 10 ? can you add a comment to explain?
There was a problem hiding this comment.
Done.
A bit more context:
This is mainly used when sending the stdout/stderr of agent's bash commands or compilation requests to LLM.
Empirically, my observation is that each LLM response contains up to 10 such commands/requests from LLM. We allocate at most 1/10 of MAX_INPUT_TOKEN to each item to ensure balanced token distribution.
|
/gcbrun skip |
Given an overlong prompt, we want to truncate it to:
Where
...(truncated due to exceeding input token limit)...replaces sufficient prompt text so that the final prompt is within token limit.