visual-report-status-from-vitest

[yashgoyal0110]

fixes: #9113

Problem

visual-report.js inferred status from disk (hasExpected && hasActual && !hasDiff), so a test with no actual screenshot was indistinguishable from one that failed.

Confirmed against the tree: 322 metadata.json directories, 71 under WebGPU/ (CI runs --project=unit-tests, which excludes cases/webgpu.js), plus one { skip: true } in cases/noise.js:34. 71 + 1 + 1 real failure = the 73 reported.

Fix

The report now reads vitest's json reporter output as the source of truth, matching each result to its screenshot directory via ancestorTitles and the same / → %2F escaping visualTest.js uses. Four states: passed, failed,
skipped, not run. Percentages are taken against tests run. Falls back to the old disk heuristic with a visible banner when the JSON is absent, so node visual-report.js still works locally.

Testing

Ran the full suite locally, then generated the report with both the old and new
generator against the same test-results.json and screenshots on disk:

npm test -- --project=unit-tests --reporter=default --reporter=json
--outputFile.json=./test/unit/visual/test-results.json
node visual-report.js

	before	after
Total tests	322	322
Passed	249 (77%)	249 (99.6% of tests run)
Failed	73 (23%)	1 (0.4% of tests run)
Skipped	—	1
Not run	—	71

The 71 "not run" are the WebGPU/ tests, excluded because CI runs
--project=unit-tests. The 1 skip is { skip: true } in cases/noise.js:34.
The 1 failure is a pre-existing local rendering diff in
Typography > textWeight > can control non-variable fonts, unrelated to this change.

Also exercised the fallback paths: missing JSON (disk heuristic + warning banner),
malformed JSON (same), and empty testResults (all tests reported as not run).

[yashgoyal0110]

@Vaivaswat2244 kindly review

[Vaivaswat2244]

Hi @yashgoyal0110 ! Thanks a lot for your pr. It looks great. I've left a few comments. Have a look and let me know if you have any questions or need clarification.

[Vaivaswat2244]

When vitest says passed, the disk is never consulted, so a stale -diff.png renders a Diff panel next to a PASS badge, and a missing actual renders the missing-image notice under one. Both reproducible locally.

Could skip rendering the diff and the missing-image notice when the status is PASSED, since neither is meaningful then. If you'd rather not hide a real disagreement, a third state that says so is fine too, but showing both without comment is the one option I'd avoid.

[Vaivaswat2244]

notCaptured shouldn't be in this denominator, it's labelled "% of screenshots compared" below and those are the ones that were never compared.

Dropping it from executedScreenshots is enough, it's already reported on its own line.

[Vaivaswat2244]

If the reporter moves into config, this should just say to re-run npm test, since there'd be no flags for a contributor to pass.

[Vaivaswat2244]

I think this should go in the Vitest config, since a regular contributor won't add this many flags just to generate a report, especially given that most of our contributors are students.
That's on me, the command in the issue was a CLI example and it reads like it belongs in the workflow. Config is the better home for it.

Can you check once again if that works?