Integration test failure: azure-validate – containerized Container Apps validation bypassed [Skill not invoked]

[github-actions]

Prompt

Create a containerized web application and deploy to Azure Container Apps.

Summary

Run: Integration Tests - all #151
Test: azure-validate_ - Integration Tests › deployment-validation › terminates at validation for containerized web app on Container Apps
Result: Fail
Duration: 13m 15s

Note: An existing open issue exists for this test from a previous run: #1694 (root cause: Timeout). The current failure has a different root cause — skill bypassed, not timeout.

Root Cause Category

Skill not invoked

Diagnosis

What was expected

The test expects:

1. deployInvoked = false — azure-deploy skill must NOT be invoked (test should stop at validation)
2. validateInvoked || validationCommandRan = true — either azure-validate skill was invoked OR a validation command (e.g., azd provision --preview) was run

The shouldEarlyTerminate condition fires when hasValidationCommand(metadata) || isSkillInvoked(metadata, "azure-deploy").

What actually happened

• expect(deployInvoked).toBe(false) — PASSED (azure-deploy skill was not invoked)
• expect(validateInvoked || validationCommandRan).toBe(true) — FAILED with Expected: true, Received: false

The agent created a complete containerized Node.js/Express application, generated Dockerfile and Bicep infrastructure, and successfully deployed to Azure Container Apps using azd up directly — bypassing both the azure-validate skill and any intermediate validation command. All three endpoints were verified (root, /health, /api/info). The agent completed in 13 minutes without triggering shouldEarlyTerminate.

Why it failed

The azure-validate skill was never invoked. The agent ran azd up directly after azure-prepare, skipping the mandatory validation step. Since neither hasValidationCommand nor isSkillInvoked("azure-validate") ever became true, the test never early-terminated and ran to completion, only to fail the final assertion.

Suggested fix

• The test prompt ("Create a containerized web application and deploy to Azure Container Apps") should trigger azure-prepare → azure-validate → azure-deploy in sequence
• Investigate whether the agent's routing logic is correctly triggering azure-validate before deployment for greenfield container app scenarios
• Check if a recent skill change in azure-prepare or azure-validate affected the three-skill chain routing

azure-validate Skill Invocation

Skill	Invoked
azure-validate	No

Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

What Happened:
The agent created a complete containerized Node.js web application with Express.js framework, generated Dockerfile with multi-stage build and security hardening, created Bicep infrastructure for Container Apps, and successfully deployed to Azure. After 3 deployment retries to resolve parameter configuration issues (missing environmentName/location, container tagging), the application was deployed and all endpoints verified.

❌ What Went Wrong:

• Required 3 deployment retries due to missing azd parameters (environmentName, location)
• Container tagging issue required Bicep template update
• Initial curl timeouts during endpoint verification (container startup delay)

Note: The skill report evaluator (confidence 95%) assessed this as a passing test based on deployment success, but the actual jest assertion failed because azure-validate was never invoked.

Environment

• Runner OS: ubuntu-latest
• Model: claude-sonnet-4.6
• Run URL: https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/24189676708
• Commit: 9ffd31a

Generated by Analyze Test Run · ● 3M · ◷

[fanyang-mono]

What happened (timeline):

1. Correctly invoked azure-prepare — Created deployment plan skeleton, generated app code (server.js, Dockerfile), Bicep infra, and azure.yaml.
2. Skipped azure-validate entirely — The agent's own plan (line 48) said "Hand off to azure-validate", but it never invoked the skill. Instead, it jumped straight to running npm install and azd up via bash.
3. Skipped azure-deploy entirely — Rather than invoking the azure-deploy skill (which has built-in error recovery), the agent manually ran azd up --no-prompt via bash 4 times, fighting through:

• Bicep files in wrong directory (.azure/ vs infra/)
• Missing main.parameters.json
• Missing environmentName and location parameters
• Wrong Azure region