You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Using a custom G-Eval metric with o3-mini does not work at moment because the OpenAI API is called with temperature = 0.7 and logprobs = True.
To Reproduce
Using a custom G-Eval metric with o3-mini and try to measure it. The API call to OpenAI will fail.
Expected behavior
The temperature should be 1.0 and logprobs = False (or omitted entirely).
Current workaround:
g_eval.py
Not setting logprobs and top_logprobs self.model.a_generate_raw_response and self.model.generate_raw_response (top_logprobs is configurable already but not logprobs)
gpt_model.py
Setting the temperature to 1
else:
# check necessary, if model_name = o3-mini set temperature=1
return ChatOpenAI(
model_name=self.model_name,
openai_api_key=self._openai_api_key,
base_url=self.base_url,
*self.args,
**self.kwargs
)
With these changes o3-mini (and also o1-mini if manually added to the list of supported models) works with deepeval.
The text was updated successfully, but these errors were encountered:
Describe the bug
Using a custom G-Eval metric with o3-mini does not work at moment because the OpenAI API is called with temperature = 0.7 and logprobs = True.
To Reproduce
Using a custom G-Eval metric with o3-mini and try to measure it. The API call to OpenAI will fail.
Expected behavior
The temperature should be 1.0 and logprobs = False (or omitted entirely).
Current workaround:
g_eval.py
Not setting logprobs and top_logprobs self.model.a_generate_raw_response and self.model.generate_raw_response (top_logprobs is configurable already but not logprobs)
gpt_model.py
Setting the temperature to 1
With these changes o3-mini (and also o1-mini if manually added to the list of supported models) works with deepeval.
The text was updated successfully, but these errors were encountered: