Validate full plugin v1.1.0 demo with Scanner v3.4.1

[taarikashenafi]

Summary

• expand the existing example/site-with-errors/alt-text-errors.html fixture into one controlled demo for every presentation-facing v1.1.0 feature
• add a focused dry-run Scanner workflow pinned to v3.4.1 that installs @github/accessibility-scanner-alt-text-plugin@1.1.0 from npm alongside Axe
• separate real deterministic/hosted findings from fixed mocked model and Azure evidence
• add credential-free machine-readable verification plus an optional credential-gated live GitHub Models/Azure path
• document exact commands, expected evidence, limitations, and supporting reliability coverage

The broad four-site .github/workflows/scan-static-sites.yml workflow has no diff. This PR consolidates the reusable verifier/live-demo concepts from draft PR #63 onto the existing controlled fixture; it does not add #63's separate advanced page or PNG. PR #63 is closed as superseded.

Feature coverage

Feature	Evidence	Expected result
Scanner v3.4.1	Real hosted workflow	Immutable commit 745198705c4aecb2af83732eaafe1f6e3f7787c2 runs successfully.
npm plugin v1.1.0	Real hosted workflow	Scanner logs installation, discovery, and execution of alt-text-scan.
Missing, placeholder, filename, vague, and repeated alt	Real hosted artifact + local test	Exactly one plugin finding per deterministic rule.
Rule configuration	Real local test	Disabling missing-alt-text suppresses only that rule.
Context extraction	Real extraction + fake judge	Page title, nearest heading, and figcaption reach the quality judge.
Keyword stuffing	Fixed fake judge	Production mapping emits the tailored SEO-abuse finding.
Inaccurate alt + remediation	Fixed fake judge	Production mapping emits inaccurate plus the suggested replacement.
Decorative + accurate controls	Fixed fake judge	Decorative recommends alt=""; accurate emits no finding.
Azure caption/OCR/tags	Fake Azure client	Production enrichment adds high-confidence context signals.
Azure failure fallback	Fake Azure client	Production enrichment falls back to Copilot-only context.
Axe	Real hosted workflow	Axe and plugin findings share the scanner artifact.
Live GitHub Models/Azure	Optional manual command	Runs only with credentials; output is intentionally not asserted in CI.

Supporting cache, URL-redaction, image-retry, and accessibility-tree behavior remains covered by existing unit/extraction suites rather than presentation output.

Evidence boundaries

Real hosted evidence: Scanner execution; npm installation/import of plugin v1.1.0; all five credential-free plugin rules; Axe; dry-run result serialization.

Deterministic mocked evidence: only external model/Azure responses are replaced. Production extraction, quality-rule mapping, remediation, Finding emission, Azure enrichment, and fallback code run unchanged.

Optional live evidence: npm run demo:live -- --github-models-token ... calls GitHub Models and automatically adds Azure enrichment when Azure credentials are supplied. It is manual because secrets are unavailable to PR workflows and model output is nondeterministic.

The Scanner action cannot inject the repository fake judge into the published npm package, so the hosted action proves only npm loading and default deterministic behavior. Model-backed behavior is validated at the plugin layer by demo:verify and targeted tests rather than falsely claimed as a live hosted model call.

Validation

• targeted tests: 6/6 passed across tests/example-site.test.ts and tests/unit/azure-augmented-judge.test.ts
• npm run typecheck passed
• npm run lint passed
• focused Prettier checks passed
• npm run --silent demo:verify produced plugin version 1.1.0, 5 real deterministic findings, 3 mocked quality findings, and mocked Azure caption/OCR/tag signals
• demo:live credential guard correctly refuses to run without a model token
• final hosted run: https://github.com/github/accessibility-scanner-alt-text-plugin/actions/runs/30582459668
  • all steps passed in 68 seconds at commit d5cfe57
  • logs show npm installation (added 1 package), discovery, and execution of @github/accessibility-scanner-alt-text-plugin@1.1.0
  • scanner-results.json: 5 alt-text-scan findings plus 2 Axe findings (image-alt, color-contrast)
  • demo-evidence.json: plugin v1.1.0, 5 deterministic findings, 3 mocked quality findings, and 3 mocked Azure signals
  • artifact controlled-demo-results-30582459668, ID 8775169960, contains root-level scanner-results.json and demo-evidence.json
• final-head checks: lint/format, Node 22/24/26 tests, CodeQL, and controlled scan all passed

Safety

• Scanner uses dry_run: true; no issues, labels, caches, or Copilot assignments are written
• PR workflows require no model or Azure secrets
• live quality is opt-in on manual dispatch only
• review is required; this PR must not merge automatically

[Copilot]

Pull request overview

Pins the hosted static-site scan to scanner v3.4.1 while retaining Axe and alt-text plugin v1.1.0.

Changes:

• Updates the scanner action to the immutable v3.4.1 commit SHA.

Show a summary per file

File	Description
.github/workflows/scan-static-sites.yml	Pins the hosted scan to scanner v3.4.1.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (1)

.github/workflows/scan-demo-site.yml:53

• The workflow can pass even when the npm plugin fails to load—the v3.4.1 scanner's npm loader only warns and skips an unloadable plugin, so Axe still runs and scanner-results.json is produced. As written, a recurrence of the module-resolution bug this workflow is meant to validate would therefore produce a green run. Assert that the controlled fixture emitted all five deterministic alt-text rule IDs; keeping artifact upload under always() will still preserve failed evidence.

          token: ${{ github.token }}

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (2)

.github/workflows/scan-demo-site.yml:59

• The workflow treats any completed scan as a successful validation but never checks the controlled fixture's documented five plugin rule IDs. The linked green run already omitted repeated-alt-text, so this check can pass while the scanner/plugin integration misses an expected finding. Parse the results and fail unless every expected plugin rule is present.

      - name: Upload scanner results

.github/workflows/scan-demo-site.yml:69

• This cleanup runs even when Ruby setup or the build fails before the PID file is created, so cat and then kill add a second misleading step failure. Guard the PID file and tolerate an already-exited server so cleanup remains best-effort.

        run: kill "$(cat "$RUNNER_TEMP/demo-site.pid")"

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (3)

.github/workflows/scan-demo-site.yml:88

• The Scanner step does not validate the published plugin's output; the action succeeds and writes a results file even if alt-text-scan emits zero findings. As a result, this check can stay green without proving the five deterministic v1.1.0 rules claimed in the PR. Assert the exact plugin rule IDs before uploading the artifact.

          dry_run: true
          scans: |
            ["axe", {"name": "alt-text-scan", "package": "@github/accessibility-scanner-alt-text-plugin", "version": "1.1.0"}]

.github/workflows/scan-demo-site.yml:14

• This workflow executes demo:verify, but its pull-request filter excludes both that script and demo-support.ts. A future change to either runtime path can therefore merge without running its only end-to-end check. Add the demo scripts (and package scripts that invoke them) to this path filter.

  pull_request:
    paths:
      - .github/workflows/scan-demo-site.yml
      - example/site-with-errors/**

example/site-with-errors/alt-text-errors.html:57

• All four live model cases use this SVG fixture. loadImageAsDataUrl forwards it as an image/svg+xml data URL, but the default openai/gpt-4o vision request accepts raster image formats rather than SVG. The rule catches those request failures, so demo:live can exit successfully with no findings. Use a PNG/JPEG fixture for the model-backed cases and serve that asset from the demo harness.

<figure>
  <img
    data-model-case="keyword-stuffing"
    src="/assets/img/test-image.svg"

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (2)

.github/workflows/scan-demo-site.yml:14

• The trigger does not cover several inputs executed by this workflow. A PR that changes scripts/verify-demo.ts, either supporting demo script, index.ts/src/**, or the package files can break demo:verify without running this controlled-demo job; the general test workflow only runs npm run test, not the verifier. Include these dependencies in the path filter so this evidence cannot silently go stale.

    paths:
      - .github/workflows/scan-demo-site.yml
      - example/site-with-errors/**

.github/workflows/scan-demo-site.yml:65

• The hosted run cited in the PR does not validate this step or the current workflow revision. Run 30581692971 executed head a13f4aa, and its job contains no Node setup, dependency install, demo:verify, live-demo, or evidence-staging steps; the PR head had since advanced to d5cfe57. Please rerun the workflow at the current head before treating the linked run as authoritative validation of the newly added evidence path.

      - name: Produce deterministic demo evidence
        run: npm run --silent demo:verify > "$RUNNER_TEMP/demo-evidence.json"

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (5)

scripts/verify-demo.ts:101

• The verifier records these outputs but never checks the promised five deterministic rule IDs or three mapped model findings; only the Azure signal block below can make the command fail. Consequently demo:verify and the workflow remain green if scanning or verdict mapping regresses to empty/incorrect output. Assert the expected counts, rule IDs, and key remediation mappings before emitting the JSON.

  const results = await altTextQuality.evaluate({url: fixtureUrl, images: modelCases})
  const mockedModelFindings: Finding[] = []
  await emitFindings(altTextQuality, results, fixtureUrl, async finding => {
    mockedModelFindings.push(finding)
  })

.github/workflows/scan-demo-site.yml:14

• This workflow executes the local entry point, src implementation, demo scripts, and npm dependency lock, but PRs changing those files do not trigger it. After this PR (which triggers because the workflow itself changed), the controlled evidence can regress without this check running. Include index.ts, src/**, scripts/**, package.json, and package-lock.json in the path filter.

    paths:
      - .github/workflows/scan-demo-site.yml
      - example/site-with-errors/**

example/site-with-errors/README.md:32

• The workflow's Python HTTP server serves the entire _site directory; only the Scanner target is restricted to this route. Reword this to avoid overstating the serving boundary.

builds this Jekyll site, serves only `/alt-text-errors/`, and runs Scanner

example/site-with-errors/README.md:117

• The plural subject “tests” requires parallel plural verbs: “run,” “assert,” “check,” and “exercise.”

The targeted tests load
[`alt-text-errors.html`](alt-text-errors.html), runs the real `alt-text-scan`
plugin against it, asserts exactly one finding for each deterministic rule,
checks the four model cases through fixed fake-judge verdicts, and exercise the
production Azure enrichment and fallback layers with a fake Azure client. No

example/site-with-errors/alt-text-errors.html:57

• All four live model cases reference this SVG. loadImageAsDataUrl preserves it as image/svg+xml, but SVG is not a supported vision input for GitHub Models or Azure Image Analysis. altTextQuality.evaluate catches each resulting request failure and returns no results, so demo:live can exit successfully with an empty findings array. Use a raster PNG/JPEG fixture for the model-backed cases and serve that asset from the demo harness.

    src="/assets/img/test-image.svg"

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Medium