Skip to content

feat: support repeatable --test-id flag in eval run #924

@christso

Description

@christso

Summary

agentv eval run --test-id currently accepts a single glob pattern. It would be useful to support repeatable --test-id flags (like --target already does) to filter by multiple specific test IDs in a single run.

Use case

When running evals in CI via GitHub Actions, we expose test_id as a workflow input. Users sometimes want to run a handful of specific tests across suites without writing a glob that matches them all.

Example:

agentv eval run evals/**/*.eval.yaml --test-id "fsa-shipment-mode" --test-id "schema-qualification-*"

Current workaround

Glob patterns can cover some cases (--test-id "fsa-*"), but there's no way to specify multiple unrelated test IDs without running separate commands.

Proposal

Make --test-id repeatable (OR logic — run test if it matches any pattern), consistent with how --target already works.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions