Skip to content

[v1 backport] LocalEvalService still crashes on failed inference with inferences=None #6553

Description

@ftnext

🔴 Required Information

Describe the Bug:

The current v1 branch (v1.37.0, commit 8b2feb5) still crashes in LocalEvalService._evaluate_single_inference_result() when inference fails before producing invocations.

This is the same bug reported in #5876 and fixed on main by #5878 / commit 9a6cf60.
The fix has not been backported to v1.

Steps to Reproduce:

  1. Create an InferenceResult with:
    • status=InferenceStatus.FAILURE
    • inferences=None
  2. Pass it to LocalEvalService.evaluate().
  3. _evaluate_single_inference_result() reaches:

if eval_case.conversation_scenario is None and len(
inference_result.inferences
) != len(eval_case.conversation):

  1. Evaluation crashes with:
TypeError: object of type 'NoneType' has no len()

Expected Behavior:

As on main, a failed inference without invocations should produce an EvalCaseResult with final_eval_status=EvalStatus.FAILED, without aborting the rest of the evaluation run.

Observed Behavior:

The entire evaluation fails while processing len(None), masking the original inference error.

Environment Details:

  • ADK Library Version: 1.37.0 (v1 branch)
  • Desktop OS: macOS
  • Python Version: 3.13.12
  • Model Information: N/A; the failure path is model-independent

🟡 Optional Information

Proposed Fix / Backport:

Please cherry-pick the existing fix from main:

9a6cf60fa8d54523e95943ebdb49d4f35341aed0
fix(eval): handle failed inference results without invocations

I verified that the complete commit applies cleanly to the current v1 branch (8b2feb52) without conflicts.

I also ran the focused tests after applying it to v1:

4 passed, 15 deselected

Command:

pytest tests/unittests/evaluation/test_local_eval_service.py -k "failed_without_inferences or evaluate_single_inference_result" -q

Regression History:

A previous fix for #2729 was added by #3805 / commit 7d4326c, but the evaluation changes and regression tests were removed in the immediately following commit 81eaeb5.
The current main fix in 9a6cf60 is a smaller backport and already matches the desired v2 behavior.

Metadata

Metadata

Labels

eval[Component] This issue is related to evaluation

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions