feat(cli): add adk eval --num_runs N with per-eval-case aggregation - #6840
Open
copybara-service[bot] wants to merge 1 commit into
Open
feat(cli): add adk eval --num_runs N with per-eval-case aggregation#6840copybara-service[bot] wants to merge 1 commit into
adk eval --num_runs N with per-eval-case aggregation#6840copybara-service[bot] wants to merge 1 commit into
Conversation
**Problem:** `adk eval` did not support repeated runs, so users had to script external loops to reduce nondeterminism. **Solution:** - Add `--num_runs` to `adk eval` (default `1`, min `1`). The value is carried on `InferenceConfig.num_runs`, so the eval service repeats each eval case through its existing parallelism pool instead of the CLI multiplying the requests. - Aggregate the per-run results for each eval case (`eval_set_id` + `eval_id`) with a mean-over-invocations strategy that matches `AgentEvaluator` (the pytest entrypoint), so multi-run summaries are consistent across entrypoints. The strategy is an input via `EvaluateConfig.aggregation_strategy` (`AggregationStrategy`); only `MEAN_OVER_INVOCATIONS` is supported today. - `--print_detailed_results` continues to show per-run details. Tests cover the aggregator (mean-over-invocations, grouping, threshold, and runs that failed before producing metric results), the service-level `num_runs` repetition, and the CLI validation/summary behavior. Co-authored-by: ftnext <takuyafjp+develop@gmail.com> Co-authored-by: Haran Rajkumar <haranrk@google.com> PiperOrigin-RevId: 959841339
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(cli): add
adk eval --num_runs Nwith per-eval-case aggregationProblem:
adk evaldid not support repeated runs, so users had to script external loopsto reduce nondeterminism.
Solution:
--num_runstoadk eval(default1, min1). The value is carried onInferenceConfig.num_runs, so the eval service repeats each eval case throughits existing parallelism pool instead of the CLI multiplying the requests.
eval_set_id+eval_id)with a mean-over-invocations strategy that matches
AgentEvaluator(the pytestentrypoint), so multi-run summaries are consistent across entrypoints. The
strategy is an input via
EvaluateConfig.aggregation_strategy(
AggregationStrategy); onlyMEAN_OVER_INVOCATIONSis supported today.--print_detailed_resultscontinues to show per-run details.Tests cover the aggregator (mean-over-invocations, grouping, threshold, and
runs that failed before producing metric results), the service-level
num_runsrepetition, and the CLI validation/summary behavior.
Co-authored-by: ftnext takuyafjp+develop@gmail.com
Co-authored-by: Haran Rajkumar haranrk@google.com