-
Notifications
You must be signed in to change notification settings - Fork 503
Guardrails on the Gateway Framework
About a year ago, we started Portkey to solve a problem we'd faced ourselves — lack of adequate tooling to confidently put LLM apps in production. This showed up in many ways:
- There was no way to debug LLM requests
- You couldn't see request-level cost data
- You couldn't iterate fast on prompts without involving devs
- When you wanted to try out a new model, it took forever to integrate
- And finally, the LLM output itself was often not reliable enough to use in production
We started building Portkey to solve these "ops" challenges, and we did it in our own way, with our opinionated, open source AI Gateway. (that reminds - have you starred this repo? 😬)
With Portkey Gateway, we now process billions of LLM tokens every day, and help hundreds of companies take their AI apps to production.
Teams use Portkey to debug fast, iterate fast, handle API failures, and much more!
We built the Gateway to avoid affecting core LLM behavior. Portkey performs request/response transformations across 200+ different LLMs today, making them more robust. However, the core LLM behavior remains unpredictable:
- Outputs can be complete hallucinations or factually inaccurate
- Results may show bias, violate privacy norms, or potentially harm the organization
When it comes to taking AI apps to production, we believe this unpredictability is THE BIGGEST missing component. Chip Huyen recently wrote a definitive guide on "Building a Gen AI Platform" and highlighted this issue:
What happens, when you bring,
Guardrails are systems that help control and guide LLM outputs. By integrating them into the Gateway, we can create a powerful solution:
With this integration, you can orchestrate your LLM requests based on the Guardrail's verdict, handling LLM behavior EXACTLY as you want. This combination brings together Interoperability, Routing, and Guardrails on the Gateway.
We believe Gateway is critical tooling every AI developer will need. However, we recognize that steering and evaluating LLM behavior requires specialized expertise.
That's why we're partnering with some of the world's best AI guardrails platforms to make them available on Portkey Gateway.
Guardrails are now available on our open source repo for you to try out, as well as free on Portkey's hosted app.
Check out the Plugins folder and take a deep dive into all that's possible! You can also explore our Guardrail docs here.
And while you do that, please don't forget to star this repo!
This development is a significant step towards closing a crucial production gap that many companies face — and it's just the beginning!
As we move forward, continuous learning, adaptation, and collaboration will be key to addressing the complex challenges that lie ahead.
We're excited to see how the community adopts and adapts these concepts. We look forward to continuing this important conversation about the future of AI APIs!