Update Measure in AI eng #465
Conversation
| } | ||
| ); | ||
|
|
||
| Eval('text-match-eval', { |
There was a problem hiding this comment.
Should we encourage users to add capability name to metadata?
| Create an eval file `src/evals/ticket-classification.eval.ts`: | ||
|
|
||
| ```typescript | ||
| import { experimental_Eval as Eval, Scorer } from 'axiom/ai/evals'; |
There was a problem hiding this comment.
| import { experimental_Eval as Eval, Scorer } from 'axiom/ai/evals'; | |
| import { Eval, Scorer } from 'axiom/ai/evals'; |
There was a problem hiding this comment.
@c-ehrlich we need to write something about cli auth: axiom login, maybe @manototh can help us structure this in a good way. There would be 2 ways to authenticate the SDK:
- Either using CLI, and then you only need to specify a dataset in
axiom.config.ts - Specifying an API token and dataset in axiom.config.ts
There was a problem hiding this comment.
@thesollyz Thanks for this. I would specify API token only because that's what we use in the rest of the AI eng docs anyway.
There was a problem hiding this comment.
@manototh not sure what do you mean? CLI auth is a new feature that we want the users to use.
| title: "Measure" | ||
| description: "Learn how to measure the quality of your AI capabilities by running evaluations against ground truth data." | ||
| keywords: ["ai engineering", "AI engineering", "measure", "evals", "evaluation", "scoring", "graders"] | ||
| keywords: ["ai engineering", "AI engineering", "measure", "evals", "evaluation", "scoring", "scorers"] |
There was a problem hiding this comment.
I think it would be also good to have graders as a keyword here for users who would search for graders instead of scorers. Maybe we can add "scores" as well.
3 participants
No description provided.