Skip to content

feat: run Playwright accessibility scan on pull requests - #1580

Open
coseeian wants to merge 3 commits into
processing:mainfrom
coseeian:a11y-ci-scan
Open

feat: run Playwright accessibility scan on pull requests#1580
coseeian wants to merge 3 commits into
processing:mainfrom
coseeian:a11y-ci-scan

Conversation

@coseeian

@coseeian coseeian commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Goal

This PR wires the existing Playwright + axe-core a11y suite into CI so regressions get caught on PRs.

Notes for reviewers

  • The scan currently covers the EN homepage (both desktop/mobile). We'll expand coverage to other pages once we've monitored the CI stability on PRs from other contributors for a short period.
  • By default, axe-core runs automatic accessibility check rules tagged for WCAG 2.0 and WCAG 2.1 at Level A and Level AA, along with selected best practices.
  • A failed scan generates a GitHub Action summary (sample) containing the scan results and downloadable artifacts. The default message format should be understandable, but I'm happy to refine the report output for other contributors down the line if we see a need.
  • The check is currently blocking - any new axe violations will fail the build. Once we confirm the CI is stable, I’d suggest we consider making this newly added action a required check in the branch protection rules, so PRs can’t be merged with violations.

@coseeian coseeian added the Accessibility: Best Practice No obvious current impact, but which may lead to problems in the future label Aug 20, 2026
@ksen0

ksen0 commented Aug 25, 2026

Copy link
Copy Markdown
Member

cc @Vaivaswat2244 if you have any thoughts on this one too, please share!

@Vaivaswat2244

Copy link
Copy Markdown

Hi folks! Thank you @coseeian for putting this up. Its really helpful. The CI vs local split in the config reads clean.

I looked at the report in your last run's artifact. The two tests took about 3s each, but the whole Playwright run was 425s. So roughly 422s of it is npm run build && npm run preview starting up before any test runs.

test.yml already has a build job running npm run build with the same env vars, so I think we end up building the site twice on every PR. I think one option to avoid this is to upload dist/ from the existing build job and have this job download it and just run npm run preview. That would need the webServer command to be overridable. Other than that the report looks great.

@Vaivaswat2244

Copy link
Copy Markdown

One more thing I had in mind, since the build is almost all of the runtime, scanning a few more pages only costs a few seconds each.A reference page, a tutorial page and a listing page would cover markup the homepage won't. It's just extra page.goto calls, the server will already serve the whole site. Might be better to add them now, before this becomes a required check, since any existing violations there are easier to deal with early.

On that, I'd suggest continue-on-error: true for the first couple of weeks. We still get the signal, without a red check on people who haven't seen axe output before, and turning it back on is one line. Thoughts?

@coseeian
coseeian force-pushed the a11y-ci-scan branch 2 times, most recently from 8b1ede3 to 03b4767 Compare August 31, 2026 08:12
@coseeian

coseeian commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks so much @Vaivaswat2244 for the thoughtful feedback! These are super helpful suggestions.

On reusing the build:
I tried simply concatenating the scan to the end of the existing build job so it picks up the dist folder directly. That felt like the simplest path forward. However, I'm more than happy to split it back out using an artifact upload/download step if keeping build and a11y as separate jobs is more preferred.

On scanning more pages & continue-on-error:
Great point! I was initially worried about introducing a huge wall of violations all at once, but pairing the extra pages with continue-on-error: true makes sense. I went ahead and expanded the scope to 19 pages. As expected, it picked up a few existing violations. My plan is to tackle those violations in follow-up PRs, and once everything is clean, we can drop continue-on-error and set up the branch protection check.

Let me know how this sounds to you. Thanks again!

@Vaivaswat2244

Copy link
Copy Markdown

Sounds great @coseeian!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Accessibility: Best Practice No obvious current impact, but which may lead to problems in the future

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants