LLM Provider Fallback LLM Proxy/Gateway

[mmabrouk]

Current Situation

The current LLM proxy allows a deployment configuration to contain a prompt and the provider configuration for it. However, it does not support a failover mechanism.

Problem

If the configured LLM provider starts getting errors or becomes unavailable (for example, Gemini returns errors or another provider is too busy), the request fails. There is no fallback to ensure the service continues to operate.

Proposed Solution

We should add a failover capability to the deployment, so that the user has the ability to deploy a revision as a fallback to a deployment. This would allow a deployment to be configured with a primary provider/prompt and a secondary (fallback) provider/prompt.

If the user fetches the deployment configuration (prompt management) they would receive the primary and fallback (optional) configuration.

If the user call the LLM proxy, it would handle the logic itself:

1. Try the request with the primary provider and its configured prompt.
2. If the first attempt fails, the proxy automatically retries the request using the configured fallback provider and its specific prompt.

Original Request by Faizan

Can LLM Proxy feature also failover to a different prompt + LLM provider automatically?

Well, I have noticed recently that Gemini suddenly starts giving errors. Anthropic from time to time can also give too busy return code.
The use case is simple to just fallback to another provider with a prompt configured for that provider.