-
Notifications
You must be signed in to change notification settings - Fork 0
feat: support repeatable --test-id flag in eval run #924
Copy link
Copy link
Closed
Description
Summary
agentv eval run --test-id currently accepts a single glob pattern. It would be useful to support repeatable --test-id flags (like --target already does) to filter by multiple specific test IDs in a single run.
Use case
When running evals in CI via GitHub Actions, we expose test_id as a workflow input. Users sometimes want to run a handful of specific tests across suites without writing a glob that matches them all.
Example:
agentv eval run evals/**/*.eval.yaml --test-id "fsa-shipment-mode" --test-id "schema-qualification-*"Current workaround
Glob patterns can cover some cases (--test-id "fsa-*"), but there's no way to specify multiple unrelated test IDs without running separate commands.
Proposal
Make --test-id repeatable (OR logic — run test if it matches any pattern), consistent with how --target already works.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels