Skip to content

fix(streaming): skip duplicate delta.tool_calls already emitted during streaming #9286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
walcz-de wants to merge 1 commit into from
+9 −0
Closed

fix(streaming): skip duplicate delta.tool_calls already emitted during streaming #9286

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions core/http/endpoints/openai/chat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,15 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator

default:
for i, ss := range functionResults {
// Skip tool calls that were already emitted incrementally by the
// streaming JSON/XML parser during ComputeChoices (tracked by
// lastEmittedCount). Re-emitting them would produce duplicate
// delta.tool_calls events, causing clients to accumulate
// "{args}{args}" which is invalid JSON (cogito streaming bug).
if i < lastEmittedCount {
Copy link
Copy Markdown
Owner

@mudler mudler Apr 9, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real problem is the Go-side incremental JSON parser at lines 270-316. It runs on every streaming token, re-parses the full accumulated text, and re-emits the same tool call every time the JSON is valid. This produces more duplicate emissions, while the default: case only adds 1 more on top.

But you should issue this only when backends don't emit chatdeltas. That tells me you are running an old version of the llama.cpp backend - try to upgrade it at least - and should fix this issue.

While the problem is indeed there, the PR is partially fixing it and needs adding some robust testing first to fix it properly.

Copy link
Copy Markdown
Contributor Author

@walcz-de walcz-de Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a glitch in my local build setup to build localai with rocm-7.12-preview which did not use the proper llama.cpp version. Now it works. So I will close this pr

continue
}

name, args := ss.Name, ss.Arguments
toolCallID := ss.ID
if toolCallID == "" {
Expand Down
Loading