📣 Pezzo Prompt A/B Testing Support - Share your thoughts!

[arielweinberger]

Our Mission

Pezzo is the first truly open-source and cloud-native LLMOps platform. We are committed to this vision, and for this vision to succeed, it's important that the platform continues to evolve to meet the needs of AI adopters.

A/B Testing and Targeting

A/B Testing is probably one of the most common feature requests I get when talking to the community. This can be interpreted in many ways, but in a nutshell, it could be summarized as serving different versions of a prompt with varying settings using various targeting/bucketing strategies. The goal is to test multiple versions at the same time and use feedback/data to decide which one works best. Another use case (not quite A/B testing) is targeting - targeting a particular audience with a particular prompt version.

Some examples:

• A/B Testing: Serve 50% of version A and 50% of version B, where the temperature differs.
• Targeting: Serve prompt version A to users from Brazil (which uses a Brazil-trained fine-tuned model) and version B to the rest. Another example is using a whole different LLM for particular markets, due to regulatory reasons.

Share your thoughts - how can Pezzo help here? It could be anything from UIs for humans to manually rate results, APIs to close the trace loop, integrations with analytics systems to track converting AI operations, and even Pezzo automatically validating prompt results against a "validator" prompt.

📣 Please provide your feedback! Any feedback, even unstructured, or random ideas, is most welcome.

---

This is a living document that will be used to document ideas, community needs and challenges with regards to this feature.

I will occasionally update the the main post with information based on community feedback, and the goal is to deliver a first functioning version as soon as possible.