Skip to content

Conversation

@mpangrazzi
Copy link
Contributor

Ideally, an AsyncPipeline should have only async-compatible components with async streaming_callback support.

However, there are cases where a pipeline might include a mix of components with both sync-only and async streaming support.

For example, you might have a pipeline that includes an older generator such as HuggingFaceLocalGenerator (or any other component lacking async streaming support) alongside an async-compatible one like OpenAIChatGenerator.

With this, I am introducing a parameter to async_streaming_generator called allow_sync_streaming_callbacks which accepts the following values:

  • False (default): normal behaviour (strict) - accepts only async-streaming-compatible components on AsyncPipeline
  • "auto": Automatically detect and enable hybrid mode if needed - Hybrid mode allows components with sync-only streaming callback support to work in async pipelines.

Using "auto" mode will allow async streaming to work with (old) generators like HuggingFaceLocalGenerator.

Running synchronous callbacks within an async streaming context is still not recommended, as it introduces minor overhead - though typically negligible (~1–2 μs per call).

@mpangrazzi mpangrazzi requested a review from sjrl October 28, 2025 14:11
@mpangrazzi mpangrazzi marked this pull request as ready for review October 28, 2025 14:11
…ync method signature rather than components attributes)
@mpangrazzi
Copy link
Contributor Author

@sjrl I know it's a bit off-topic, but on this PR I've also improved the check for streaming capable components.

Indeed, before this we were still looking for streaming_callback in component attributes. But this is somewhat wrong, since for making concurrent streaming to work we need to pass streaming callback in run/run_async methods.

So now the behaviour is the following:

  • Check if streaming_callback is a parameter in run() or run_async() method signatures (just implemented)
  • Pass streaming_callback as a method parameter via pipeline_run_args (present from v1.2.0)

This will make stream to work also with components like FallbackChatGenerator which doesn't have streaming_callback as a component attribute.

mpangrazzi and others added 3 commits October 29, 2025 09:52
@mpangrazzi mpangrazzi requested a review from sjrl October 29, 2025 14:26
Copy link
Contributor

@sjrl sjrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@mpangrazzi mpangrazzi merged commit 42dc75e into main Oct 29, 2025
5 checks passed
@mpangrazzi mpangrazzi deleted the auto-hybrid-streaming branch October 29, 2025 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants