How to use Azure OpenAI with FaithfulnessEvaluator

[joshdawson]

Does anybody know how to use an Azure OpenAI deployment with the FaithfulnessEvaluator?

I've tried setting the api_params as detailed in the example to evaluate using a local model here: https://docs.haystack.deepset.ai/docs/model-based-evaluation#using-local-llms.

Here's how my evaluator is setup

evaluator = FaithfulnessEvaluator(
    api_key=Secret.from_token({AZURE_OPENAI_API_KEY}),
    api_params={"api_base_url": {AZURE_OPENAI_ENDPOINT}, "model": {AZURE_OPENAI_MODEL_DEPLOYMENT}}
)

However, I get a 404 error. I've also tried various different values for the api_base_url and model with the same 404 error

[Amnah199]

Hi @joshdawson, the FaithfulnessEvaluator currently supports only the OpenAI API, so Azure OpenAI cannot be used with this component at the moment. However, you can configure a different model if that suits your needs.

For more details, please refer to the API reference.

[joshdawson]

Thanks for that! Do you know if the RagasEvaluator supports Azure OpenAI? It doesn't look like it does from the documentation

[Amnah199]

Yes, the RagasEvaluator can work with Azure OpenAI. Since RagasEvaluator accepts any LangChain-compatible LLM through the evaluator_llm parameter, you can use Azure OpenAI by wrapping it with LangChain's Azure OpenAI client.

Hope this helps!