Commit 92db1fe
feat(gooddata-eval): add KDA-skill agentic evaluator
Adds kda_skill.py, evaluating the chatbot's create_key_driver_analysis /
execute_key_driver_analysis tool calls against the agent_kda_skill Langfuse
dataset (QA-28800).
Current scope is completion, not field correctness: strict_pass requires
kda_triggered + executed + success + turn_completed. Per-field checks
(Measure/Date Attribute/Periods/Filters/Summary) are computed and logged to
Langfuse for visibility, but intentionally excluded from strict_pass --
that verification is scoped to a follow-up ticket.
Includes a bounded (max 2 turns) disambiguation safety net, mirroring
alert_skill's/metric_skill's simulated-user pattern: if the agent asks a
clarifying question (a metric-title collision, or a choice between the
metric-id and ad-hoc fact+SUM forms of the same measure) instead of
triggering KDA, a simulated reply picks an acceptable candidate so the
disambiguation turn doesn't block measuring whether KDA itself completes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>1 parent acfcc1a commit 92db1fe
2 files changed
Lines changed: 437 additions & 0 deletions
Lines changed: 14 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
33 | 41 | | |
34 | 42 | | |
35 | 43 | | |
| |||
56 | 64 | | |
57 | 65 | | |
58 | 66 | | |
| 67 | + | |
59 | 68 | | |
60 | 69 | | |
61 | 70 | | |
| |||
69 | 78 | | |
70 | 79 | | |
71 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
72 | 84 | | |
73 | 85 | | |
74 | 86 | | |
| |||
81 | 93 | | |
82 | 94 | | |
83 | 95 | | |
| 96 | + | |
84 | 97 | | |
85 | 98 | | |
86 | 99 | | |
87 | 100 | | |
88 | 101 | | |
89 | 102 | | |
90 | 103 | | |
| 104 | + | |
91 | 105 | | |
92 | 106 | | |
93 | 107 | | |
| |||
0 commit comments