Skip to content

test: fail test 4b on failed tool request and wrong finish_reason #268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
solderzzc merged 2 commits into from
Nov 7, 2026
Merged

test: fail test 4b on failed tool request and wrong finish_reason #268

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
82e5916
test: fail test 4b on failed tool request and wrong finish_reason
solderzzc Nov 6, 2026
d2494f9
test: FAIL test 4b on a body cut off mid-way and on a 200 with no cho…
solderzzc Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion tests/test-dflash.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,14 +219,24 @@ fi
# a capability, not a contract, so a reply without any tool-call shape reports SKIP.
log "Test 4b: a tool call under DFlash arrives as tool_calls, not as text"
TOOLS='[{"type":"function","function":{"name":"get_weather","description":"Get the current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]'
TOOL_RC=0
TOOL_RESP=$(curl -sf --max-time 180 -X POST "$URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\":\"$MAIN_MODEL\",\"max_tokens\":120,\"temperature\":0,\"tools\":$TOOLS,\"messages\":[{\"role\":\"system\",\"content\":\"You are a weather kiosk. Call get_weather for any weather question and say nothing else.\"},{\"role\":\"user\",\"content\":\"What is the weather in Paris right now?\"}]}" 2>/dev/null || echo "")
-d "{\"model\":\"$MAIN_MODEL\",\"max_tokens\":120,\"temperature\":0,\"tools\":$TOOLS,\"messages\":[{\"role\":\"system\",\"content\":\"You are a weather kiosk. Call get_weather for any weather question and say nothing else.\"},{\"role\":\"user\",\"content\":\"What is the weather in Paris right now?\"}]}" 2>/dev/null) || TOOL_RC=$?
TOOL_NAME=$(echo "$TOOL_RESP" | jq -r '.choices[0].message.tool_calls[0].function.name // empty' 2>/dev/null || echo "")
TOOL_CONTENT=$(echo "$TOOL_RESP" | jq -r '.choices[0].message.content // empty' 2>/dev/null || echo "")
TOOL_FINISH=$(echo "$TOOL_RESP" | jq -r '.choices[0].finish_reason // empty' 2>/dev/null || echo "")
# SKIP is reserved for a reply with no tool-call shape at all. A failed request (HTTP error,
# timeout, or a body cut off mid-way), a 200 that is not a chat completion, and a parsed tool
# call with the wrong finish_reason are regressions, so FAIL.
if [ "$TOOL_NAME" = "get_weather" ] && [ "$TOOL_FINISH" = "tool_calls" ]; then
pass "DFlash tool call parsed: tool_calls[0]=get_weather, finish_reason=tool_calls"
elif [ "$TOOL_RC" -ne 0 ]; then
fail "tool request failed (curl exit $TOOL_RC: HTTP error, timeout, or body cut off)"
elif ! echo "$TOOL_RESP" | jq -e '.choices[0]' >/dev/null 2>&1; then
fail "tool response is not a chat completion (no choices): $(echo "$TOOL_RESP" | head -c 120)"
elif [ -n "$TOOL_NAME" ]; then
fail "tool call parsed but finish_reason=$TOOL_FINISH"
elif echo "$TOOL_CONTENT" | grep -q "tool_call\|get_weather"; then
fail "DFlash handed the tool call over as text: $(echo "$TOOL_CONTENT" | head -c 120)"
else
Expand Down
Loading