feat: adopt new StreamingChunk in Anthropic
#2096
Conversation
github-actions
bot
added
integration:anthropic
type:documentation
...s/anthropic/src/haystack_integrations/components/generators/anthropic/chat/chat_generator.py
Show resolved
Hide resolved
...s/anthropic/src/haystack_integrations/components/generators/anthropic/chat/chat_generator.py
Outdated
Show resolved
Hide resolved
...s/anthropic/src/haystack_integrations/components/generators/anthropic/chat/chat_generator.py
Show resolved
Hide resolved
...s/anthropic/src/haystack_integrations/components/generators/anthropic/chat/chat_generator.py
Outdated
Show resolved
Hide resolved
...s/anthropic/src/haystack_integrations/components/generators/anthropic/chat/chat_generator.py
Outdated
Show resolved
Hide resolved
| @@ -455,6 +570,12 @@ def test_convert_streaming_chunks_to_chat_message(self): | |||
| "index": 1, | |||
| "content_block": {"type": "tool_use", "id": "toolu_123", "name": "weather", "input": {}}, | |||
| }, | |||
| component_info=ComponentInfo.from_component(self), | |||
| index=1, | |||
| tool_calls=[ToolCallDelta(index=1, id="toolu_123", tool_name="weather", arguments=None)], | |||
There was a problem hiding this comment.
I think we talked about this, but this should be index=0 since it's the first and only tool call in this chunk.
| tool_calls=[ToolCallDelta(index=1, id="toolu_123", tool_name="weather", arguments=None)], | |
| tool_calls=[ToolCallDelta(index=0, id="toolu_123", tool_name="weather", arguments=None)], |
There was a problem hiding this comment.
Same goes for the rest of the chunks in this test.
There was a problem hiding this comment.
Sorry if its already discussed.
Anthropic response chunks iteratively increase the index for each content block regardless of text and tool calls. So here index=1 is because its the second content block in stream that has tool call.
See the raw response here
To achieve what you suggest we will have to handle this on our end by separately keeping track of tool calls indexes.
WDYT?
There was a problem hiding this comment.
So here index=1 is because its the second content block in stream that has tool call.
Reference
Right! So the overall index of the StreamingChunk should be 1 which you have already. So StreamingChunk.index == 1, but you're right we should then separately keep track of the tool call indexes. Remember though the tool call index is only relevant for the number of ToolCallDeltas present in a single StreamingChunk. So if there is only ever one ToolCallDelta in a StreamingChunk then the tool call index will always be 0.
There was a problem hiding this comment.
Right, so a single StreamingChunk always has a single ToolCallDelta as each InputJSONDelta is passed as a separate RawContentBlockDeltaEvent.
So maybe we can always assign tool call index = 0.
There was a problem hiding this comment.
I couldn't trigger multiple ToolCalls, as anthropic api kept producing malformed JSON args, like you mentioned.
|
|
||
| def test_check_duplicate_tool_names(self, tools): | ||
| """Test that the AnthropicChatGenerator component fails to initialize with duplicate tool names.""" | ||
| with pytest.raises(ValueError): | ||
| AnthropicChatGenerator(tools=tools + tools) | ||
|
|
||
| def test_convert_anthropic_chunk_to_streaming_chunk(self): | ||
| def test_convert_anthropic_completion_chunks_with_tools_to_streaming_chunks(self): |
There was a problem hiding this comment.
@sjrl I updated this test to be more through in checking the conversion of each chunk to streaming chunk. Additionally I removed the old test as it should be covered here.
The mock_anthropic_completion_chunks are also not needed anymore, so I'll remove them. But they were helpful in understanding the flow of response chunks from anthropic.
Related Issues
AnthropicChatGeneratorto use theStreamingChunkfields #2073Proposed Changes:
adopt the new
StreamingChunkand stream also tool callspass component info to
StreamingChunkHow did you test it?
Updated the tests
Added new tests for streaming chunks
Notes for the reviewer
Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:.