Skip to content

per_turn_user_simulator_quality_v1 crashes with pydantic ValidationError when judge_model_config is not set #5677

@christiannpierre-afk

Description

@christiannpierre-afk

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient
triaging. Requests without complete information may be rejected / deprioritized.
If an item is not applicable to you - please mark it as N/A

Describe the Bug:
A clear and concise description of what the bug is.

per_turn_user_simulator_quality_v1 always crashes with a pydantic ValidationError when configured via a JSON eval config file. The evaluator unconditionally passes judge_model_config as LlmRequest.config, but judge_model_config is always None because it is annotated with SkipJsonSchema and cannot be set through a JSON file. LlmRequest does not accept None for its config field, causing every eval case using this metric to fail.

Steps to Reproduce:

  1. Install google-adk==1.33.0
  2. Create a conversation eval config JSON file that includes per_turn_user_simulator_quality_v1:
    {
    "criteria": {
    "per_turn_user_simulator_quality_v1": {
    "threshold": 0.7,
    "judge_model_options": {
    "judge_model": "gemini-2.5-flash",
    "num_samples": 1
    }
    }
    },
    "user_simulator_config": {
    "model": "gemini-2.5-flash",
    "max_allowed_invocations": 10
    }
    }
  3. Run adk eval <agent_module> <eval_set_id> --config_file_path <path_to_config>
  4. Observe error for every eval case that uses this metric

Expected Behavior:
The evaluator should treat a None judge_model_config as "use model defaults" — either by omitting config from the LlmRequest entirely or by constructing a default GenerateContentConfig.

Observed Behavior:
Every eval case evaluated with this metric raises a pydantic ValidationError and is skipped:
ERROR - local_eval_service.py:357 - Metric evaluation failed for metric
per_turn_user_simulator_quality_v1 for eval case id '' with following error:
1 validation error for LlmRequest
config
Input should be a valid dictionary or object to extract fields from
[type=model_attributes_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.12/v/model_attributes_type

Environment Details:

  • ADK Library Version: google-adk 1.33.0
  • Desktop OS: Linux
  • Python Version: 3.12

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-2.5-flash

🟡 Optional Information

Providing this information greatly speeds up the resolution process.

Regression:
Did this work in a previous version of ADK? If so, which one?

Logs:

  ERROR - local_eval_service.py:357 - Metric evaluation failed for metric
  `per_turn_user_simulator_quality_v1` for eval case id 'b7427af2' with following error
  `1 validation error for LlmRequest
  config
    Input should be a valid dictionary or object to extract fields from
    [type=model_attributes_type, input_value=None, input_type=NoneType]
      For further information visit https://errors.pydantic.dev/2.12/v/model_attributes_type`
  Traceback (most recent call last):
    File ".../google/adk/evaluation/local_eval_service.py", line 348, in _evaluate_metric_for_eval_case
      evaluation_result = await self._evaluate_metric(...)
    File ".../google/adk/evaluation/local_eval_service.py", line 432, in _evaluate_metric
      return await metric_evaluator.evaluate_invocations(...)
    File ".../google/adk/evaluation/simulation/per_turn_user_simulator_quality_v1.py", line 188, in evaluate_invocations
      stop_signal_evaluation = await self._evaluate_stop_signal_turn(...)
    File ".../google/adk/evaluation/simulation/per_turn_user_simulator_quality_v1.py", line 365, in _evaluate_stop_signal_turn
      return await self._evaluate_intermediate_turn(...)
    File ".../google/adk/evaluation/simulation/per_turn_user_simulator_quality_v1.py", line 328, in _evaluate_intermediate_turn
      llm_request = LlmRequest(
  pydantic_core._pydantic_core.ValidationError: 1 validation error for LlmRequest
  config
    Input should be a valid dictionary or object to extract fields from
    [type=model_attributes_type, input_value=None, input_type=NoneType]

Screenshots / Video:
If applicable, add screenshots or screen recordings to help explain
your problem.

Additional Context:
Add any other context about the problem here.

Minimal Reproduction Code:
Please provide a code snippet or a link to a Gist/repo that isolates the issue.

// Code snippet here

How often has this issue occurred?:

  • Always (100%)

Metadata

Metadata

Labels

core[Component] This issue is related to the core interface and implementationrequest clarification[Status] The maintainer need clarification or more information from the author

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions