Freeze conformance requirements per spec revision, run each at its wire

[felixweinberger]

Adds requirements/<revision>.yaml, a frozen list of the scenarios a spec revision
requires, and a --requirements flag that runs exactly that set at that
revision's wire version. Ships two sets, 2025-11-25 and 2026-07-28; tier-check
takes both at once and Tier 1 means every listed revision passes.

Motivation and Context

Today there is no way to answer "which scenarios must my implementation pass to
conform to the spec released on 2026-07-28". The suite accumulates scenarios
continuously, so the set an implementation is measured against keeps growing
after the revision has shipped.

typescript-sdk is the live case. json-schema-2020-12-preservation merged on
2026-07-31 (#335), three days after the 2026-07-28 spec shipped and four days
after 0.2.0-alpha.10 was published. typescript-sdk pins alpha.10, so it has
never been able to run that scenario. On the same commit it passes 100% measured
against alpha.10 and less than 100% measured against main.

That matters because SEP-1730 puts Tier 1 at exactly 100% and relegates Tier 1 to
Tier 2 when any conformance test fails for four weeks. A scenario added after a
release starts a demotion clock against an implementation that did nothing wrong
and changed nothing.

--spec-version does not solve this. It filters on which revision a scenario
targets, not on when the scenario became a requirement, so a post-release
addition tagged [2025-11-25, 2026-07-28] is indistinguishable from one that
existed at ship.

What this adds

# what does conforming to 2026-07-28 require?
npx @modelcontextprotocol/conformance list --requirements 2026-07-28

# run exactly that
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --requirements 2026-07-28

• --requirements <revision> on the client, server, list and tier-check
  commands. It replaces --suite and --spec-version, and passing it with
  either is an error rather than a silent precedence rule.

• A requirement set covers the two roles the specification defines: an MCP server
  acting as an OAuth resource server, and an MCP client acting as an OAuth
  client. It covers no authorization-server scenarios, because the spec puts
  authorization server implementation
  beyond its own scope
  ("The implementation details of the authorization server are beyond the scope
  of this specification"), so those scenarios serve people deploying an
  authorization server rather than implementations of MCP itself. A set naming an
  authorization section is rejected with that reason, since the scenarios carry
  a dated tag for every revision and so read as core spec at a glance.

• Passing nothing keeps today's behaviour and runs everything, so the suite stays
  the place completeness lives.

• Scenarios a revision runs without scoring go under not_scored, each carrying
  the leg that runs it and why it does not count:

  Reason	Meaning
  extension	Optional by definition. SEP-1730: "Experimental features and protocol extensions (such as Tasks and MCP Apps) are not required for any tier."
  added-after-release	The scenario did not exist when the revision shipped, so no implementation could have been passing it.

  Both still run and both appear in the report with their reason, so a failing
  extension stays visible without moving a pass rate. Promoting an entry into the
  required lists stays a deliberate, reviewable change, which is how the suite
  grows without retroactively failing anyone.

• tier-check reports requirements_revision and scores both legs against the
  set, replacing the spec-version matrix with a plain required/passed summary.

The 2026-07-28 set is generated from 0.2.0-alpha.10, published the day before
that revision shipped, minus the scenarios pending in that release (SEP-1730
scores "applicable required tests" only). Older revisions are deliberately not
manifested: reconstructing them means archaeology on version-bump commits for
much less value, and with no flag nothing changes.

A requirement set is this project's contract, which is why it lives here and why
--requirements takes a revision rather than a path. It is the opposite of an
expected-failures baseline, which lives in an
implementation's own repository and records what that implementation knows it
fails. A baselined failure is still a failure against a requirement set, so the
two are independent inputs and both can be passed in the same run.

How Has This Been Tested?

Driven end to end against the typescript-sdk everything server at
main (cc4b41617):

• tier-check --requirements 2026-07-28 reports Server 37/37 and Client 32/32,
  both 100%, alongside Not scored (17 run, 14 failing), and conformance drops
  off the Tier 1 blocker list, leaving only triage and p0_resolution. Without
  the flag the same commit scores Client: Core 4/5, failing on the post-release
  scenario.
• Against a deliberately broken fixture server the same command scores 1/37 and
  server_conformance does appear in the blockers, with the 10 extension
  failures still held apart. The split suppresses the right failures and only
  those.
• server --requirements 2026-07-28 runs 37 scenarios, 120 checks, 0 failed.
• list --requirements 2026-07-28 prints the 69-scenario set plus what is run
  without being scored.
• Conflicting and malformed input: combined with --spec-version or --suite,
  an unknown revision, a path-shaped revision, and an empty value each exit 1
  with a message naming the problem.
• Unit tests assert every name in every requirement set resolves to a scenario
  this build provides, so a rename or removal cannot silently shrink what
  conformance means.

Breaking Changes

None. The flag is opt-in and the default path is unchanged.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• I have read the MCP Documentation
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have added or updated documentation as needed

Additional context

Two things deliberately left out, happy to add either:

• No CI check that a shipped revision's lists never change. The freeze is
  currently a comment in the file plus review. A test pinning each shipped file's
  hash would enforce it, at the cost of a deliberate two-step when a set does
  need amending.
• No change to the 2026-07-28 draft classification. DRAFT_PROTOCOL_VERSION
  is still 2026-07-28 even though the spec repo has published
  schema/2026-07-28. Promoting it shifts the active suite for every SDK, so it
  belongs in its own change. A requirement set does not depend on it either way.

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@443

commit: 9d390e9