Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

o3-mini is not working with G-Eval due to wrong temperature and logprobs parameter #1358

Open
RicardoBochnia opened this issue Feb 12, 2025 · 0 comments

Comments

@RicardoBochnia
Copy link

RicardoBochnia commented Feb 12, 2025

Describe the bug
Using a custom G-Eval metric with o3-mini does not work at moment because the OpenAI API is called with temperature = 0.7 and logprobs = True.

To Reproduce
Using a custom G-Eval metric with o3-mini and try to measure it. The API call to OpenAI will fail.

Expected behavior
The temperature should be 1.0 and logprobs = False (or omitted entirely).

Current workaround:
g_eval.py
Not setting logprobs and top_logprobs self.model.a_generate_raw_response and self.model.generate_raw_response (top_logprobs is configurable already but not logprobs)

gpt_model.py
Setting the temperature to 1

        else:
           # check necessary, if model_name = o3-mini set temperature=1
            return ChatOpenAI(
                model_name=self.model_name,
                openai_api_key=self._openai_api_key,
                base_url=self.base_url,
                *self.args,
                **self.kwargs
            )

With these changes o3-mini (and also o1-mini if manually added to the list of supported models) works with deepeval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant