Unable to invoke eval APIs with gpt-5x models

[Vaccarini-Lorenzo]

• Package Name: azure-ai-evaluation
• Package Version: 1.15
• Operating System: macOS 26.0.1 / Windows 11 Pro
• Python Version: 3.12.10

Describe the bug
The evaluation SDK APIs that require a LLM-as-a-judge do not work with gpt-5x models: The prompty file hard-wires the param "max_tokens" which is not supported anymore in newer models.

To Reproduce
Steps to reproduce the behavior:

1. Invoke any LLM-assisted evaluator (CoherenceEvaluator, FluencyEvaluator ...) providing a gpt-5x model.
2. The response to such invokation will be something like:
   OpenAI API hits BadRequestError: Error code: 400 - {'error': {'message': "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}} [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors]

[kashifkhan]

Thank you for the feedback @Vaccarini-Lorenzo . Ill route this over to the right folks who can help you further.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.

[Vaccarini-Lorenzo]

I just realized that internally PromptyEvaluatorBase handles the max_tokens issue checking a propagated kwarg: is_reasoning_model.

e.g.
eval = CoherenceEvaluator(model_config=model_config, is_reasoning_model=True)