Skip to content

[Evals] Built-in or integrated evaluation metrics #2813

Open
@ssbushi

Description

@ssbushi

Is your feature request related to a problem? Please describe.
Evaluation metrics existing as a separate plugin that needs to be explicitly installed may be a hurdle for user.

Describe the solution you'd like
Some considerations:

  • Provide genkitEvals plugin as part of the core SDK. (no need to do a separate npm i)
  • Stretch, make some metrics integrated into the CLI to leverage a common platform. No need to port every metric to every runtime (instant parity). Considerations needed here are how to provide genkit primitives in CLI, or perhaps a workaround that references a CLI-metric from the runtime.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions