Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend TransformersTextRouter to use other model providers #8747

Open
julian-risch opened this issue Jan 19, 2025 · 1 comment
Open

Extend TransformersTextRouter to use other model providers #8747

julian-risch opened this issue Jan 19, 2025 · 1 comment
Labels
P3 Low priority, leave it in the backlog type:feature New feature or request

Comments

@julian-risch
Copy link
Member

Is your feature request related to a problem? Please describe.
Users would like to use vLLM or other model providers for text routing but currently the only supported option is to load a model from huggingface and run it.

Describe the solution you'd like
Similar to the vLLM integration for chat generators, it would be great if there was a text router that could be used with api_base_url or a similar parameter.

Describe alternatives you've considered
Write a custom component.

Additional context
TransformersZeroShotTextRouter could be extended in a similar way.

@julian-risch julian-risch added P3 Low priority, leave it in the backlog type:feature New feature or request labels Jan 19, 2025
@lbux
Copy link
Contributor

lbux commented Jan 24, 2025

I don't think this is feasible using something like api_base_url. I'm also unsure of any provider supporting a classification endpoint (no support on llama.cpp, vllm, ollama, openai). Since there is no standard OpenAI compatible API endpoint, there would need to be a custom component for each provider's implementation (if there are any).

An alternative (not a good one) would be to use the chat endpoints with logits restrictions via the API to control the output. This is heavily dependent on the prompt and would be similar to the implementations used for the LLM based evaluators. This would allow for a somewhat standardized component that utilize the OpenAI compatible API endpoint (aka OpenAITextRouter with the api_base_url)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 Low priority, leave it in the backlog type:feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants