Skip to content

Commit ecf8718

Browse files
Tomkessclaude
andcommitted
test(gooddata-eval): assert exact MAQL string appears in simulated-user prompt
Addresses CodeRabbit review comment on #1718: the regression test only checked for generic instruction words ("verbatim", "every clause"), not that expected_output["maql"] itself made it into the prompt -- a regression that stripped the metric/label reference or filter value entirely could still pass. Assert the exact MAQL string is present. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 4e1cbda commit ecf8718

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

packages/gooddata-eval/tests/test_agentic_metric_skill.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ def test_generate_simulated_response_prompt_preserves_maql_fidelity(monkeypatch)
4242
# `openai` is an optional [llm-judge] extra, not installed in this test env --
4343
# inject a fake module rather than patching a real one (mirrors how the source
4444
# itself does `from openai import OpenAI` as a local, guarded import).
45-
fake_openai_module = types.SimpleNamespace(OpenAI=MagicMock(return_value=mock_client))
45+
fake_openai_module = types.SimpleNamespace(OpenAI=MagicMock(return_value=mock_client), OpenAIError=Exception)
4646
monkeypatch.setitem(sys.modules, "openai", fake_openai_module)
4747

4848
expected_output = {"maql": 'SELECT {metric/spend_amount_-_cutcgco} WHERE {label/ecommerce_indicator_code} = "1"'}
@@ -51,6 +51,7 @@ def test_generate_simulated_response_prompt_preserves_maql_fidelity(monkeypatch)
5151
call_kwargs = mock_client.chat.completions.create.call_args.kwargs
5252
sent_prompt = call_kwargs["messages"][0]["content"]
5353

54+
assert expected_output["maql"] in sent_prompt
5455
assert "verbatim" in sent_prompt
5556
assert "every clause" in sent_prompt
5657
assert "WHERE" in sent_prompt or "filter" in sent_prompt.lower()

0 commit comments

Comments
 (0)