Add CI test to validate Azure MCP tool names in skill markdown files by Copilot · Pull Request #1709 · microsoft/GitHub-Copilot-for-Azure

Copilot · 2026-04-03T21:33:34Z

Skill markdown files reference Azure MCP tools by name, but there was no automated check to catch stale or mistyped tool names that don't match what the server actually exposes.

Changes

New test tests/skills-validation/mcp-tool-names.test.ts — unit test that runs as part of the existing test:unit suite (picked up by the test-all-skills.yml CI workflow). It:
- Reads the valid tool list from a static snapshot file tests/skills-validation/azure-mcp-tools.json — fast, offline, and deterministic (no live server, no npm registry access, no 120s timeout)
- Recursively scans all .md files under plugin/skills/ for two naming conventions:
  - mcp_azure_mcp_<tool> (GitHub Copilot MCP tool reference format)
  - azure__<tool> (alternative format)
- Fails with file path + line number diagnostics for every invalid tool name found
New snapshot tests/skills-validation/azure-mcp-tools.json — static list of the 61 tool names currently exposed by @azure/mcp server start. Update this file when new tools are added to the server (tracked in issue Add scheduled CI workflow to refresh MCP tool name snapshot #1933).
Bug fixes in 9 MD files — discovered by the new test: mcp_azure_mcp_get_bestpractices is not a real tool; the correct name is mcp_azure_mcp_get_azure_bestpractices. Fixed across azure-upgrade, azure-cloud-migrate, and azure-prepare skill reference files.

Failure output example

When a file references a non-existent tool, the test produces:

● all Azure MCP tool names referenced in skill markdown files are valid

  expect(received).toHaveLength(0)

  Received:
    ["azure-upgrade/SKILL.md:28 — "get_bestpractices" is not a valid Azure MCP tool name",
     ...]

…eferences Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/362a2400-266c-4070-9a8b-625601205ea3 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

kvenkatrajan · 2026-04-03T22:04:21Z

@copilot - bump the version on 'azure-cloud-migrate' , 'azure-prepare', 'azure-upgrade' skills

…de skills Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/5a67a30f-b6d1-43a9-8c38-a2ad05209e1b Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot · 2026-04-03T22:08:36Z

@copilot - bump the version on 'azure-cloud-migrate' , 'azure-prepare', 'azure-upgrade' skills

Done in commit 243a564. Bumped versions for the three modified skills:

azure-cloud-migrate: 1.0.1 → 1.0.2
azure-prepare: 1.1.6 → 1.1.7
azure-upgrade: 1.0.0 → 1.0.1

Copilot

Pull request overview

Adds automated validation to ensure Azure MCP tool names referenced in skill markdown stay in sync with the tools actually exposed by @azure/mcp, and fixes existing stale tool references found by that validation.

Changes:

Add a new unit test that boots @azure/mcp and validates referenced Azure MCP tool names across plugin/skills/**/*.md.
Fix stale tool references (get_bestpractices → get_azure_bestpractices) across affected skill docs.
Update azure-upgrade and azure-cloud-migrate SKILL tool lists/usages to the corrected tool name.

Show a summary per file

File	Description
tests/skills-validation/mcp-tool-names.test.ts	New unit test that queries Azure MCP `tools/list` and validates tool references in skill markdown.
plugin/skills/azure-upgrade/SKILL.md	Fixes Azure MCP best-practices tool name in rules/tool table.
plugin/skills/azure-upgrade/references/services/functions/assessment.md	Fixes best-practices tool reference in tip.
plugin/skills/azure-upgrade/references/global-rules.md	Fixes best-practices tool reference in global rules.
plugin/skills/azure-prepare/references/recipes/terraform/README.md	Fixes best-practices tool reference in Terraform recipe table.
plugin/skills/azure-prepare/references/recipes/azcli/README.md	Fixes best-practices tool reference in Azure CLI recipe table.
plugin/skills/azure-cloud-migrate/SKILL.md	Fixes Azure MCP best-practices tool name in rules.
plugin/skills/azure-cloud-migrate/references/services/functions/global-rules.md	Fixes best-practices tool reference in rules.
plugin/skills/azure-cloud-migrate/references/services/functions/code-migration.md	Fixes best-practices tool reference in prerequisites/steps.
plugin/skills/azure-cloud-migrate/references/services/functions/assessment.md	Fixes best-practices tool reference in tip.

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 3

kvenkatrajan · 2026-04-03T22:26:07Z

Fixes #778

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

jongio

The approach works well - validating tool names against the live server catches drift that's hard to spot manually, and the 9 fixes prove the test's value.

Three suggestions on the test implementation, plus a note that the PR currently has merge conflicts to resolve.

jongio · 2026-04-08T18:06:04Z

+// Tests
+// ---------------------------------------------------------------------------
+
+describe("Azure MCP Tool Name Validation", () => {


This test spawns a live MCP server, needs npm registry access, and has a 120s timeout - the firewall warnings in the PR description show it can't run in all CI environments. Consider either moving it to an integration test path or extracting a static tool-name snapshot for unit tests (with a separate integration test to keep the snapshot fresh).

Ive created - #1933 to update MCP tool snapshot

@copilot - implement Jon's suggestion on unit test Reads azure-mcp-tools.json (fast, offline, deterministic)

Done in commit c38ddb5. Added tests/skills-validation/azure-mcp-tools.json (snapshot of the 61 tools currently exposed by @azure/mcp) and replaced the live-server logic in the test with a synchronous fs.readFileSync call. The child_process import and the 120s beforeAll timeout are gone — the test now completes in ~2s.

…mes test Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/3db4fef5-f378-4506-83b4-7a2f8ee1c5d4 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

…apshot in unit test - Add tests/skills-validation/azure-mcp-tools.json with current tool list from @azure/mcp - Refactor mcp-tool-names.test.ts to read from snapshot (fast, offline, deterministic) - Remove spawn/child_process dependency, 120s beforeAll timeout, and live server logic - Test now completes in ~2s instead of up to 2 minutes Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/123eec45-bcc4-4e61-815f-89ffe090b2da Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot · 2026-04-17T16:26:34Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

centralus-2.in.applicationinsights.azure.com
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start (dns block)
westus-0.in.applicationinsights.azure.com
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

jongio

Previous feedback addressed cleanly - the snapshot approach eliminates the live-server dependency, the 120s timeout, and the Windows spawn issue in one shot.

One remaining observation on the regex (inline). PR is also currently CONFLICTING with main - needs a rebase before it can merge.

jongio · 2026-04-17T16:42:47Z

+const MCP_AZURE_MCP_RE = /mcp_azure_mcp_([a-z][a-z0-9_]*)/g;
+// Matches: azure__<tool>
+const AZURE_DOUBLE_UNDERSCORE_RE = /azure__([a-z][a-z0-9_]*)/g;
+


Greedy match is fine today because every azure__<tool> in current skills is bounded by backticks or whitespace - I grepped. But if a skill ever writes the MCP function-call form azure__storage_blob_list as a single token, this will fail: greedy capture grabs storage_blob_list, which isn't in the snapshot (only storage is).

Two low-cost options if you want to harden it:

After extracting the suffix, try progressively shorter prefixes down to the first underscore before declaring invalid.

Leave it strict and document the convention in the snapshot header so authors know compound forms aren't supported.

Not a blocker - current skills are all safe. Flagging it as latent so it doesn't surprise someone in 6 months.

Initial plan

9106121

Copilot AI assigned Copilot and kvenkatrajan Apr 3, 2026

Copilot started work on behalf of kvenkatrajan April 3, 2026 21:33 View session

Copilot AI linked an issue Apr 3, 2026 that may be closed by this pull request

PR CI tests for ensuring that the tool names in md file reflect actual Azure MCP server tools #778

Open

Add CI test for Azure MCP tool name validation and fix invalid tool r…

80120cb

…eferences Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/362a2400-266c-4070-9a8b-625601205ea3 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add CI tests for validating Azure MCP tool names in markdown files~~ Add CI test to validate Azure MCP tool names in skill markdown files Apr 3, 2026

Copilot finished work on behalf of kvenkatrajan April 3, 2026 21:55

Copilot AI requested a review from kvenkatrajan April 3, 2026 21:55

kvenkatrajan marked this pull request as ready for review April 3, 2026 21:59

kvenkatrajan requested review from saikoumudi, tmeschter and wbreza as code owners April 3, 2026 21:59

Copilot AI review requested due to automatic review settings April 3, 2026 21:59

Copilot started reviewing on behalf of kvenkatrajan April 3, 2026 22:03 View session

Copilot started work on behalf of kvenkatrajan April 3, 2026 22:04 View session

Bump versions for azure-cloud-migrate, azure-prepare, and azure-upgra…

243a564

…de skills Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/5a67a30f-b6d1-43a9-8c38-a2ad05209e1b Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot finished work on behalf of kvenkatrajan April 3, 2026 22:09

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Comment thread tests/skills-validation/mcp-tool-names.test.ts Outdated

Comment thread plugin/skills/azure-upgrade/SKILL.md

Comment thread plugin/skills/azure-cloud-migrate/SKILL.md

Copilot started work on behalf of kvenkatrajan April 3, 2026 22:19 View session

Reject early on MCP server process exit with stderr diagnostics

26d67d8

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot finished work on behalf of kvenkatrajan April 3, 2026 22:31

jongio reviewed Apr 8, 2026

View reviewed changes

Copilot started work on behalf of kvenkatrajan April 17, 2026 15:57 View session

Add shell:true to spawn and defensive pagination check in mcp-tool-na…

7cf081e

…mes test Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/3db4fef5-f378-4506-83b4-7a2f8ee1c5d4 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 17, 2026 16:00

Copilot AI requested a review from MadhuraBharadwaj-MSFT as a code owner April 17, 2026 16:00

Copilot AI review requested due to automatic review settings April 17, 2026 16:00

Copilot finished work on behalf of kvenkatrajan April 17, 2026 16:01

Copilot started work on behalf of kvenkatrajan April 17, 2026 16:01 View session

Copilot finished work on behalf of kvenkatrajan April 17, 2026 16:03

kvenkatrajan mentioned this pull request Apr 17, 2026

Add scheduled CI workflow to refresh MCP tool name snapshot #1933

Open

Copilot started work on behalf of kvenkatrajan April 17, 2026 16:20 View session

Copilot AI mentioned this pull request Apr 17, 2026

Add scheduled Azure MCP tool snapshot refresh and snapshot-based skill reference validation #1934

Draft

Copilot AI requested review from Copilot and removed request for Copilot April 17, 2026 16:25

Copilot finished work on behalf of kvenkatrajan April 17, 2026 16:26

jongio reviewed Apr 17, 2026

View reviewed changes

Conversation

Copilot AI commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Failure output example

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

Copilot AI commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

jongio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jongio Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

kvenkatrajan Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 17, 2026

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

jongio left a comment

Choose a reason for hiding this comment

Uh oh!

jongio Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Apr 3, 2026 •

edited

Loading