Skip to content

Guardrails on the Gateway Framework

vrushankportkey edited this page Aug 14, 2024 · 3 revisions

wiki-3

Introduction

About a year ago, we started Portkey to solve a problem we'd faced ourselves — lack of adequate tooling to confidently put LLM apps in production. This showed up in many ways:

  • There was no way to debug LLM requests
  • You couldn't see request-level cost data
  • You couldn't iterate fast on prompts without involving devs
  • When you wanted to try out a new model, it took forever to integrate
  • And finally, the LLM output itself was often not reliable enough to use in production

We started building Portkey to solve these "ops" challenges, and we did it in our own way, with our opinionated, open source AI Gateway. (that reminds - have you starred this repo? 😬)

The Gateway Today Moves The Needle

With Portkey Gateway, we now process billions of LLM tokens every day, and help hundreds of companies take their AI apps to production.

Teams use Portkey to debug fast, iterate fast, handle API failures, and much more!

But, The Problem Remains

We built the Gateway to avoid affecting core LLM behavior. Portkey performs request/response transformations across 200+ different LLMs today, making them more robust. However, the core LLM behavior remains unpredictable:

  • Outputs can be complete hallucinations or factually inaccurate
  • Results may show bias, violate privacy norms, or potentially harm the organization

This is the Biggest Missing Component

When it comes to taking AI apps to production, we believe this unpredictability is THE BIGGEST missing component. Chip Huyen recently wrote a definitive guide on "Building a Gen AI Platform" and highlighted this issue:

image

No, look closely:

image

What happens, when you bring,

What Happens When You Bring Guardrails Inside the Gateway?

Guardrails are systems that help control and guide LLM outputs. By integrating them into the Gateway, we can create a powerful solution:

wiki

With this integration, you can orchestrate your LLM requests based on the Guardrail's verdict, handling LLM behavior EXACTLY as you want. This combination brings together Interoperability, Routing, and Guardrails on the Gateway.

But, we are no experts in Guardrails

We believe Gateway is critical tooling every AI developer will need. However, we recognize that steering and evaluating LLM behavior requires specialized expertise.

That's why we're partnering with some of the world's best AI guardrails platforms to make them available on Portkey Gateway.

CleanShot 2024-08-14 at 19 40 48@2x

Available on Portkey App & on Open Source

Guardrails are now available on our open source repo for you to try out, as well as free on Portkey's hosted app.

Check out the Plugins folder and take a deep dive into all that's possible! You can also explore our Guardrail docs here.

And while you do that, please don't forget to star this repo!

This development is a significant step towards closing a crucial production gap that many companies face — and it's just the beginning!

As we move forward, continuous learning, adaptation, and collaboration will be key to addressing the complex challenges that lie ahead.

We're excited to see how the community adopts and adapts these concepts. We look forward to continuing this important conversation about the future of AI APIs!