Skip to content

Commit 92db1fe

Browse files
FrankHuynhclaude
andcommitted
feat(gooddata-eval): add KDA-skill agentic evaluator
Adds kda_skill.py, evaluating the chatbot's create_key_driver_analysis / execute_key_driver_analysis tool calls against the agent_kda_skill Langfuse dataset (QA-28800). Current scope is completion, not field correctness: strict_pass requires kda_triggered + executed + success + turn_completed. Per-field checks (Measure/Date Attribute/Periods/Filters/Summary) are computed and logged to Langfuse for visibility, but intentionally excluded from strict_pass -- that verification is scoped to a follow-up ticket. Includes a bounded (max 2 turns) disambiguation safety net, mirroring alert_skill's/metric_skill's simulated-user pattern: if the agent asks a clarifying question (a metric-title collision, or a choice between the metric-id and ad-hoc fact+SUM forms of the same measure) instead of triggering KDA, a simulated reply picks an acceptable candidate so the disambiguation turn doesn't block measuring whether KDA itself completes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent acfcc1a commit 92db1fe

2 files changed

Lines changed: 437 additions & 0 deletions

File tree

packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@
3030
evaluate_agentic_guardrail,
3131
run_agentic_guardrail,
3232
)
33+
from gooddata_eval.core.agentic.kda_skill import (
34+
AgenticKdaSummary,
35+
KdaEvaluation,
36+
KdaRunResult,
37+
KdaSkillAssertionError,
38+
evaluate_agentic_kda_skill,
39+
run_agentic_kda_skill,
40+
)
3341
from gooddata_eval.core.agentic.metric_skill import (
3442
AgenticMetricSummary,
3543
MetricRunResult,
@@ -56,6 +64,7 @@
5664
"AgenticAlertSummary",
5765
"AgenticGeneralQuestionSummary",
5866
"AgenticGuardrailSummary",
67+
"AgenticKdaSummary",
5968
"AgenticMetricSummary",
6069
"AgenticSearchSummary",
6170
"AgenticRunSummary",
@@ -69,6 +78,9 @@
6978
"GeneralQuestionResult",
7079
"GuardrailAssertionError",
7180
"GuardrailResult",
81+
"KdaEvaluation",
82+
"KdaRunResult",
83+
"KdaSkillAssertionError",
7284
"MetricRunResult",
7385
"MetricSkillAssertionError",
7486
"RunResult",
@@ -81,13 +93,15 @@
8193
"evaluate_agentic_conversation",
8294
"evaluate_agentic_general_question",
8395
"evaluate_agentic_guardrail",
96+
"evaluate_agentic_kda_skill",
8497
"evaluate_agentic_metric_skill",
8598
"evaluate_agentic_search_tool",
8699
"evaluate_agentic_visualization",
87100
"run_agentic_alert_skill",
88101
"run_agentic_conversation",
89102
"run_agentic_general_question",
90103
"run_agentic_guardrail",
104+
"run_agentic_kda_skill",
91105
"run_agentic_metric_skill",
92106
"run_agentic_search_tool",
93107
"run_agentic_visualization",

0 commit comments

Comments
 (0)