You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Users would like to use vLLM or other model providers for text routing but currently the only supported option is to load a model from huggingface and run it.
Describe the solution you'd like
Similar to the vLLM integration for chat generators, it would be great if there was a text router that could be used with api_base_url or a similar parameter.
Describe alternatives you've considered
Write a custom component.
Additional context
TransformersZeroShotTextRouter could be extended in a similar way.
The text was updated successfully, but these errors were encountered:
I don't think this is feasible using something like api_base_url. I'm also unsure of any provider supporting a classification endpoint (no support on llama.cpp, vllm, ollama, openai). Since there is no standard OpenAI compatible API endpoint, there would need to be a custom component for each provider's implementation (if there are any).
An alternative (not a good one) would be to use the chat endpoints with logits restrictions via the API to control the output. This is heavily dependent on the prompt and would be similar to the implementations used for the LLM based evaluators. This would allow for a somewhat standardized component that utilize the OpenAI compatible API endpoint (aka OpenAITextRouter with the api_base_url)
Is your feature request related to a problem? Please describe.
Users would like to use vLLM or other model providers for text routing but currently the only supported option is to load a model from huggingface and run it.
Describe the solution you'd like
Similar to the vLLM integration for chat generators, it would be great if there was a text router that could be used with
api_base_url
or a similar parameter.Describe alternatives you've considered
Write a custom component.
Additional context
TransformersZeroShotTextRouter could be extended in a similar way.
The text was updated successfully, but these errors were encountered: