feat(gooddata-eval): support targeting a specific AI Hub agent - #1703
feat(gooddata-eval): support targeting a specific AI Hub agent#1703Tomkess wants to merge 3 commits into
Conversation
|
Warning Review limit reached
Next review available in: 5 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (15)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1703 +/- ##
==========================================
+ Coverage 79.50% 79.66% +0.15%
==========================================
Files 272 272
Lines 19019 19024 +5
==========================================
+ Hits 15121 15155 +34
+ Misses 3898 3869 -29 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
GoodData has no admin-settable "default agent": when a conversation
doesn't name one, the platform picks whichever agent was last used or
last edited in that workspace. Orgs with multiple AI Hub agents (e.g.
one scoped to visualization only) can end up silently evaluating the
wrong one -- a metric_skill/alert_skill item run against a
visualization-only agent never passes, regardless of question quality.
ChatClient never sent an agentId at all, so there was no way to pick.
- ChatClient gains an `agent_id` param; `create_conversation()` sends
`{"agentId": ...}` only when set -- omitted entirely when None, so
existing behavior (platform's own default-agent resolution) is
unchanged unless the caller opts in.
- Threaded through all 7 run_agentic_*/evaluate_agentic_* pairs, the
agentic-dispatch layer (_dispatch_agentic/run_agentic_items), and the
non-agentic ChatClient construction in cli/main.py.
- New `gd-eval run --agent-id ID` flag (or `GD_EVAL_AGENT_ID` env var,
same precedence convention as --token/GOODDATA_TOKEN).
- README: new flags-table row + a "Targeting a specific AI Hub agent"
section with real usage examples.
- Tests: ChatClient POST-body shape (with/without agent_id), CLI arg
parsing, flag/env-var/unset precedence into the constructed
ChatClient, and _dispatch_agentic threading it to evaluate_agentic_*.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sse_client.py: reformat create_conversation() to ruff's line-length rule (the PR's own change). - test_cli.py: drop an unused `original_chat_client` local -- pre-existing on master (same line, unrelated to this PR), but ruff check runs whole-file and blocks this PR's lint-and-format-check job since this test function lives in a file the PR also touches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
codecov flagged one uncovered line in _dispatch_agentic's agent_id threading -- the earlier tests only exercised 2 of 7 kind branches. Parametrized test now covers all 7 (vis_agentic, agentic_visualization, agentic_search, agentic_general_question, agentic_guardrail, agentic_conversation, plus the two already covered). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
b55f608 to
f2764ac
Compare
1 participant
Summary
GoodData has no admin-settable "default agent": when a conversation doesn't name one, the platform picks whichever agent was last used or last edited in that workspace. Orgs with multiple AI Hub agents configured (e.g. one scoped to visualization only, another with every skill enabled) can end up silently evaluating the wrong one — a
metric_skill/alert_skillitem run against a visualization-only agent never passes, no matter how well-formed the question is, because that agent can't call the tools those items need.ChatClientnever sent anagentIdat all, so there was no way to target a specific one.Changes
ChatClientgains anagent_id: str | None = Noneparam;create_conversation()sends{"agentId": ...}only when set. When unset, behavior is byte-for-byte unchanged (noagentIdin the body at all) — this is purely additive, opt-in.run_agentic_*/evaluate_agentic_*pairs (metric_skill,alert_skill,visualization,search_tool,general_question,guardrail,conversation), the agentic-dispatch layer (_dispatch_agentic/run_agentic_items), and the non-agenticChatClientconstruction incli/main.py.gd-eval run --agent-id IDflag, orGD_EVAL_AGENT_IDenv var — same precedence convention already used for--token/GOODDATA_TOKEN.Test plan
ChatClientPOST body:agentIdsent when set, omitted when not (backward-compat regression guard).--agent-idpresent/absent.ChatClientconstruction via a spy._dispatch_agenticthreadsagent_idthrough toevaluate_agentic_*(and omits it by default).gooddata-evalsuite: 251 passed, same 9 pre-existing failures onmastertoo (missingopenaiextra in this env, unrelated) — no regressions from this change.Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com