🔴 Required Information
Describe the Bug:
The current v1 branch (v1.37.0, commit 8b2feb5) still crashes in LocalEvalService._evaluate_single_inference_result() when inference fails before producing invocations.
This is the same bug reported in #5876 and fixed on main by #5878 / commit 9a6cf60.
The fix has not been backported to v1.
Steps to Reproduce:
- Create an
InferenceResult with:
status=InferenceStatus.FAILURE
inferences=None
- Pass it to
LocalEvalService.evaluate().
_evaluate_single_inference_result() reaches:
|
if eval_case.conversation_scenario is None and len( |
|
inference_result.inferences |
|
) != len(eval_case.conversation): |
- Evaluation crashes with:
TypeError: object of type 'NoneType' has no len()
Expected Behavior:
As on main, a failed inference without invocations should produce an EvalCaseResult with final_eval_status=EvalStatus.FAILED, without aborting the rest of the evaluation run.
Observed Behavior:
The entire evaluation fails while processing len(None), masking the original inference error.
Environment Details:
- ADK Library Version:
1.37.0 (v1 branch)
- Desktop OS: macOS
- Python Version:
3.13.12
- Model Information: N/A; the failure path is model-independent
🟡 Optional Information
Proposed Fix / Backport:
Please cherry-pick the existing fix from main:
9a6cf60fa8d54523e95943ebdb49d4f35341aed0
fix(eval): handle failed inference results without invocations
I verified that the complete commit applies cleanly to the current v1 branch (8b2feb52) without conflicts.
I also ran the focused tests after applying it to v1:
Command:
pytest tests/unittests/evaluation/test_local_eval_service.py -k "failed_without_inferences or evaluate_single_inference_result" -q
Regression History:
A previous fix for #2729 was added by #3805 / commit 7d4326c, but the evaluation changes and regression tests were removed in the immediately following commit 81eaeb5.
The current main fix in 9a6cf60 is a smaller backport and already matches the desired v2 behavior.
🔴 Required Information
Describe the Bug:
The current
v1branch (v1.37.0, commit 8b2feb5) still crashes inLocalEvalService._evaluate_single_inference_result()when inference fails before producing invocations.This is the same bug reported in #5876 and fixed on
mainby #5878 / commit 9a6cf60.The fix has not been backported to
v1.Steps to Reproduce:
InferenceResultwith:status=InferenceStatus.FAILUREinferences=NoneLocalEvalService.evaluate()._evaluate_single_inference_result()reaches:adk-python/src/google/adk/evaluation/local_eval_service.py
Lines 273 to 275 in 8b2feb5
Expected Behavior:
As on
main, a failed inference without invocations should produce anEvalCaseResultwithfinal_eval_status=EvalStatus.FAILED, without aborting the rest of the evaluation run.Observed Behavior:
The entire evaluation fails while processing
len(None), masking the original inference error.Environment Details:
1.37.0(v1branch)3.13.12🟡 Optional Information
Proposed Fix / Backport:
Please cherry-pick the existing fix from
main:I verified that the complete commit applies cleanly to the current
v1branch (8b2feb52) without conflicts.I also ran the focused tests after applying it to
v1:Command:
pytest tests/unittests/evaluation/test_local_eval_service.py -k "failed_without_inferences or evaluate_single_inference_result" -qRegression History:
A previous fix for #2729 was added by #3805 / commit 7d4326c, but the evaluation changes and regression tests were removed in the immediately following commit 81eaeb5.
The current
mainfix in 9a6cf60 is a smaller backport and already matches the desired v2 behavior.