Skip to content

Commit

Permalink
[WNMGDS-2722] New browser testing wrapper script (#3006)
Browse files Browse the repository at this point in the history
* Start to a wrapper script for browser tests

* This is readable, but using exec crashes because the buffer fills up

Really I need to use spawn, but that means all my args need to be in array form

* This works!

* Build prereqs first unless `--no-build` is passed

* Update all the browser test commands

Updating any of these is as simple as passing `-u` now. We can even use one of these aliases and pass `--no-docker` to it, like `yarn test:browser:examples --no-docker`, and it will work

* Update the unit test commands to match

* Rewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testingRewrite the README content on browser testing

* Add an example debug command
  • Loading branch information
pwolfert committed Apr 4, 2024
1 parent 1267118 commit e1c4cb1
Show file tree
Hide file tree
Showing 5 changed files with 131 additions and 35 deletions.
48 changes: 27 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,17 @@ These scripts can all be run from the root level of the repo:
- Alias of `yarn test:unit`
- `yarn test:unit`
- Runs JS unit tests for all packages
- `yarn test:unit:update` updates [Jest snapshots](http://facebook.github.io/jest/docs/en/snapshot-testing.html)
- `yarn test:unit -u` updates [Jest snapshots](http://facebook.github.io/jest/docs/en/snapshot-testing.html)
- `yarn test:browser`
- Runs accessibility and visual-regression tests using [Playwright](https://playwright.dev/). See [Visual regression testing](#visual-regression-testing) section below for details.
- Note that you need to build Storybook statically (`yarn build:storybook`) before you can run the tests
- `yarn test:browser:update` updates reference screenshots used for visual regression testing. Update these only when we expect the visual changes
- `yarn test:browser --project <name>` runs only one of the named projects found in [playwright.config.ts](/tests/browser/playwright.config.ts)
- Runs accessibility and visual-regression tests using [Playwright](https://playwright.dev/). See [Visual regression testing](#visual-regression-testing) section below for details. This is the base command that defaults to the basic tests that cover Storybook stories, but we have several different kinds of browser tests under different sub-commands.
- Note that this command will [accept any arguments that Playwright accepts](https://playwright.dev/docs/test-cli).
- `yarn test:browser -u` updates reference screenshots used for visual regression testing. Update these only when we expect the visual changes. You can use this argument on any of the browser-test sub-commands to update snapshots for specific kinds of tests.
- `yarn test:browser --no-build` will skip building the tests' pre-requisites. This is useful if you've already done it and haven't made any changes to the source.
- `yarn test:browser --grep "Alert"` will only run tests with "Alert" in the name.
- `yarn test:browser:interaction` runs VRT interaction tests to validate visual state of components after interaction.
- `yarn test:browser:examples` runs VRT tests for our example projects.
- `yarn test:browser:all` runs all VRT, static and interactive.
- `yarn test:browser:storybook-docs` checks for regressions in prop tables in storybook docs
- `yarn test:browser:storybook-docs` checks for regressions in prop tables in storybook docs.
- `yarn test:browser:all` runs all of our visual regression tests.
- `yarn lint`
- Runs just the linting portion of the tests, eslint and stylelint
- `yarn deploy-demo`
Expand All @@ -81,24 +82,29 @@ These scripts can all be run from the root level of the repo:

### Visual regression testing

We use [Playwright](https://playwright.dev/) to test our components for visual regressions. It uses our existing Storybook stories, taking screenshots of them within a docker container and comparing those screenshots with ones previously taken and committed to version control. The tests assume that Storybook has been built to `./storybook-static` using `yarn build:storybook`.
We use [Playwright](https://playwright.dev/) to test our components for visual regressions. We have several suites of visual regression tests, but our main suite uses Storybook stories as references. These tests will load a story or other reference material, take a screenshot within a docker container (for consistency), and compare those screenshots with ones previously taken and committed to version control.

Running the browser tests locally requires that you be signed into Docker or have playwright installed locally.
Tests can be run in a docker container or out, but we only check in VRT reference images taken inside the docker container, because taking them outside of a container will produce inconsistent results from machine to machine. The reason you might run them outside of docker is if you're working on the tests themselves and want to run in headful mode so you can see what's happening and troubleshoot Note that updating the visual regression test reference images locally requires that you be signed into Docker.

#### If using Docker:
There are a lot of tests, so it can be helpful to constrain the tests you run locally by [using Playwright's `--grep` argument](https://playwright.dev/docs/test-cli).

1. Open the Docker app, and make sure you're signed in (Docker Desktop requires a license now)
2. Run `yarn test:browser` to begin comparing component images
1. If differences are detected and unexpected, evaluate your changes - we only want to update and commit references when we expect the visual changes detected
2. If differences are detected and expected, run `yarn test:browser:update`

#### If running Locally:
#### Updating the visual regression tests with Docker:

**Note that running without using the docker image will occasionally throw false positives, depending on your system configuration. It's recommended that you utilize Docker when running DS visual regression tests.**

1. If you have run `npx playwright install` and installed the playwright dependencies locally you can run the tests using their yarn commands directly.
2. For example, to run the CMSDS VRT Tests for inteaction states: `yarn playwright test --config tests/browser/interaction.config.ts`
3. The `-u` flag can be added to the `yarn playwright test` command to update snapshots.
1. Open the Docker app, and make sure you're signed in (Docker Desktop requires a license now).
2. Run `yarn test:browser` to begin comparing component images
1. If differences are detected and unexpected, evaluate your changes - we only want to update and commit references when we expect the visual changes detected.
2. If differences are detected and expected, run `yarn test:browser -u`, verify the changes, and then commit them.

#### Development outside of Docker:

- Run any of the browser-test sub-commands with the `--no-docker` flag, like `yarn test:browser:examples --no-docker`.
- If you've never installed Playwright, running the `yarn test:browser` command will prompt you to install it, which you will want to do. The reason we don't need it installed when running it in Docker is because the Docker image contains all of its own dependencies.
- Because snapshots will probably fail outside of the Docker container, pass `--ignore-snapshots` if you want to just see that the tests execute properly.
- You can turn off headless mode by using Playwright's `--headed` flag.
- Playwright's `--debug` flag is another helpful argument because it will pause and allow you to step through the tests.
- Don't forget that you can run a subset of tests by using Playwright's `--grep` argument.
- Remember that passing `--no-build` will skip re-building the source material like Storybook stories if you haven't made any changes to them and are only changing the tests themselves.
- Here's an example of a command you might run to debug the dropdown tests: `yarn test:browser:interaction --no-docker --no-build --headed --debug --ignore-snapshots --grep "Dropdown"`

## Design Assets

Expand Down
16 changes: 6 additions & 10 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,14 @@
"posttest": "yarn lint",
"test:unit:coverage": "yarn test:unit --collectCoverage",
"test:unit": "yarn jest --config=tests/unit/jest.config.js",
"test:unit:update": "yarn test:unit --updateSnapshot",
"test:unit:preact": "PREACT=true yarn jest --config=tests/unit/jest.config.js",
"test:unit:wc": "WC=true yarn jest --config=tests/unit/jest.config.js",
"test:browser:all": "yarn test:browser && yarn test:browser:interaction && yarn test:browser:storybook-docs",
"test:browser": "docker run --rm --network host -v $(pwd):/work/ -w /work/ mcr.microsoft.com/playwright:v1.31.0-focal yarn playwright test --config tests/browser/playwright.config.ts",
"test:browser:smoke": "docker run --rm --network host -v $(pwd):/work/ -w /work/ --env SMOKE=true mcr.microsoft.com/playwright:v1.31.0-focal yarn playwright test --config tests/browser/playwright.config.ts",
"test:browser:interaction": "docker run --rm --network host -v $(pwd):/work/ -w /work/ mcr.microsoft.com/playwright:v1.31.0-focal yarn playwright test --config tests/browser/interaction.config.ts",
"test:browser:update": "yarn build:storybook && yarn test:browser -u && yarn test:browser:interaction -u",
"test:browser:storybook-docs": "docker run --rm --network host -v $(pwd):/work/ -w /work/ mcr.microsoft.com/playwright:v1.31.0-focal yarn playwright test --config tests/browser/storybook-docs.config.ts",
"test:browser:storybook-docs:update": "yarn build:storybook && yarn test:browser:storybook-docs -u",
"test:browser:examples": "docker run --rm --network host -v $(pwd):/work/ -w /work/ mcr.microsoft.com/playwright:v1.31.0-focal yarn playwright test --config tests/browser/examples.config.ts",
"test:browser:examples:update": "yarn build:examples && yarn test:browser:examples -u",
"test:browser": "ts-node ./scripts/browser-tests.ts",
"test:browser:smoke": "yarn test:browser --smoke",
"test:browser:interaction": "yarn test:browser --config tests/browser/interaction.config.ts",
"test:browser:storybook-docs": "yarn test:browser --config tests/browser/storybook-docs.config.ts",
"test:browser:examples": "yarn test:browser --config tests/browser/examples.config.ts",
"test:browser:all": "yarn test:browser && yarn test:browser:interaction --no-build && yarn test:browser:storybook-docs --no-build && yarn test:browser:examples",
"type-check": "yarn tsc --noEmit"
},
"husky": {
Expand Down
90 changes: 90 additions & 0 deletions scripts/browser-tests.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
import yargs from 'yargs';
import { hideBin } from 'yargs/helpers';
import { sh, shI } from './utils';

const DOCKER_IMAGE = 'mcr.microsoft.com/playwright:v1.31.0-focal';

function verifyPlaywrightInstalled() {
try {
sh('yarn playwright --version');
} catch (error) {
console.log('Playwright command is unavailable. Install it with `npx playwright install`.');
process.exit(1);
}
}

(async () => {
// Get command line args
const argv = await yargs(hideBin(process.argv))
.scriptName('yarn test:browser')
.parserConfiguration({ 'unknown-options-as-args': true })
.options({
// Note that you can negate it with --no-build
build: {
boolean: true,
description: 'Builds storybook first',
default: true,
},
config: {
string: true,
description: 'Path to Playwright config file',
default: 'tests/browser/playwright.config.ts',
},
docker: {
boolean: true,
description: 'Runs tests inside Playwright Docker container',
default: true,
},
smoke: {
boolean: true,
description: 'Runs only the smoke tests',
default: false,
},
})
.help().argv;

// Build whatever is necessary to run these tests
if (argv.build) {
if (argv.config.includes('examples.config.ts')) {
shI('yarn', ['build:examples']);
} else {
shI('yarn', ['build:storybook']);
}
}

// Create the array of args for the playwright command
const configArgs = ['--config', argv.config];
const extraArgs = argv._.map((v) => '' + v);
const playwrightArgs = ['test', ...configArgs, ...extraArgs];

if (argv.docker) {
// Create the array of args for the docker command
const dockerArgs = [
...['run', '--rm', '--network', 'host', '-v', `${process.cwd()}:/work/`, '-w', '/work/'],
// Environment vars need to be passed to the docker container
...(argv.smoke ? ['--env', 'SMOKE=true'] : []),
DOCKER_IMAGE,
...['yarn', 'playwright', ...playwrightArgs],
];

// And run docker
shI('docker', dockerArgs);
} else {
// To run outside of docker, we need to have Playwright installed separately
verifyPlaywrightInstalled();

// Environment vars need to be passed to the spawned process through the config object
let config;
if (argv.smoke) {
config = {
env: {
...process.env,
SMOKE: 'true',
},
};
}

// Run Playwright directly through yarn
shI('yarn', ['playwright', ...playwrightArgs], config);
}
})();
2 changes: 1 addition & 1 deletion scripts/release.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ function printNextSteps() {
(async () => {
// Get command line args
const argv = await yargs(hideBin(process.argv))
.scriptName('npx release')
.scriptName('yarn release')
.options({
undo: {
alias: 'u',
Expand Down
10 changes: 7 additions & 3 deletions scripts/utils.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import c from 'chalk';
import { execSync, spawnSync } from 'node:child_process';
import { execSync, spawnSync, SpawnSyncOptionsWithBufferEncoding } from 'node:child_process';
import { select } from '@inquirer/prompts';

/**
Expand All @@ -16,8 +16,12 @@ export function sh(command: string, hideOutput?: boolean): string {
* you must pass all args as an array instead of including them in the command
* string.
*/
export function shI(command: string, args: string[]) {
spawnSync(command, args, { stdio: 'inherit' });
export function shI(
command: string,
args: string[],
config: SpawnSyncOptionsWithBufferEncoding = {}
) {
spawnSync(command, args, { stdio: 'inherit', ...config });
}

export function verifyGhInstalled() {
Expand Down

0 comments on commit e1c4cb1

Please sign in to comment.