diff --git a/test/e2e/docs/config_values.md b/test/e2e/docs/config_values.md index e96bbb45b00872..528ea8fa539dbb 100644 --- a/test/e2e/docs/config_values.md +++ b/test/e2e/docs/config_values.md @@ -1,11 +1,14 @@ # Config Values -A full list of config values are: +This list may be out of date as configuration value are added/removed. + +When in doubt, the values contained within the `config/*.json` files should be taken as authoritative. | Name | Description | Example | Required | Store in file? | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------- | | accounts | A JSON list of wordpress.com account information (as JSON objects). | [ { "username": "username", "password": "password", "site": "site.wordpress.com", "features": [] }, { "email": "passwordless@gmail.com", "features": [ "passwordless" ] } ] | No | **NO** | | afterHookTimeoutMS | This is the maximum total time in milliseconds that an after test hook can take including capturing the screenshots | 20000 | Yes | Yes | +| artifacts | The directory structure for test artifacts (video, screenshots, logs). | { 'screenshots': 'screenshots'} | Yes | Yes | | browser | The browser to use: either `firefox` or `chrome` | `chrome` | Yes | Yes | | calypsoBaseURL | The home page for calypso | | Yes | Yes | | calypsoDocker | A boolean indicating whether the tests will be run against a local Calypso Docker instance (required to ensure login works) | true | No | Yes | @@ -16,6 +19,7 @@ A full list of config values are: | explicitWaitMS | The explicit wait time in milliseconds to wait for an element to appear - for example a widget loading data via an API | 10000 | Yes | Yes | | highlightElements | Boolean to indicate whether to visually highlight elements being interacted with on the page | true | No | Yes | | inviteInboxId | This is an inbox id from mailosaur used for **invite** email testing | sad34id44ss | Yes | **NO** | +| locale | Default locale to be used unless overridden. | en | Yes | Yes | | mailosaurAPIKey | This is the API key from mailosaur used for testing email | hsjdhsjdh7878sd | Yes | **NO** | | mochaTimeoutMS | This is the maximum total time in milliseconds a single mocha end to end test can take to finish - otherwise it will time out. | 120000 | Yes | Yes | | passwordForNewTestSignUps | This is the password that will be set for new sign ups created for testing purposes | alongcomplexpassword%### | Yes | **NO** | @@ -41,10 +45,15 @@ A full list of config values are: | testUserName | This is an existing test WordPress.com account for testing purposes - this account should have a **single** site | testuser123 | Yes | **NO** | | testUserNameJetpack | This is an existing test WordPress.com account for testing purposes **that has a single jetpack site** | testuser123 | Yes | **NO** | | testUserNameMultiSite | This is an existing test WordPress.com account for testing purposes **that has multiple sites** | testuser123 | Yes | **NO** | +| viewportName | Default viewport name to be used unless overridden. | desktop | Yes | Yes | +| viewportSize | Viewport size specified as width and height in pixels for each viewportName that is supported. | desktop | No | Yes | diff --git a/test/e2e/docs/environment_variables.md b/test/e2e/docs/environment_variables.md index b0a67e742b0317..9102c6436032ab 100644 --- a/test/e2e/docs/environment_variables.md +++ b/test/e2e/docs/environment_variables.md @@ -2,20 +2,46 @@ Environment Variables are values that are defined at the system level to serve as configuration for programs. -## Standalone Environment Variables +## Required -| Name | Description | Example | Required | Store in file? | -| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------- | -------------- | -| MAGELLANDEBUG | If this is set, the full mocha output is printed while running Magellan | 1 | No | **NO** | -| SAUCEDEBUG | If this is set, on test failure a breakpoint will be set in SauceLabs, enabling you to continue interacting with the browser for troubleshooting | 1 | No | **NO** | +| Name | Description | Example | Required | Store in file? | +| --------------- | --------------------------------------------------- | --------- | -------- | -------------- | +| CONFIG_KEY | Secrets decryption key obtained from a8c key store. | hunter2 | No | **NO** | +| NODE_CONFIG_ENV | Name of decrypted secrets file. | decrypted | No | **NO** | -## CircleCI Environment Variables +## Magellan -These environment variables are intended for use inside CircleCI, to control which tests are being run. +| Name | Description | Example | Required | Store in file? | +| ------------- | ------------------------------------------------------------------------ | ------- | -------- | -------------- | +| MAGELLANDEBUG | If this is set, the full mocha output is printed while running Magellan. | 1 | No | **NO** | -| Name | Description | Default | Required | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -------- | ------- | -------------- | --- | -| DISABLE_EMAIL | Setting this to `true` will cause the Invite and Signup tests to be skipped | false | No | -| SKIP_TEST_REGEX | The value of this variable will be used in the `-i -g *****` parameter, to skip any tests that match the given RegEx. List multiple keywords separated by a ` | `(i.e.`Invite | Domain | Theme`) | `Empty String` | No | -| SKIP_DOMAIN_TESTS | If this value is set to `true`, the tests that attempt domain registration with be skipped. | false | No | -| SKIP_JETPACK | If this value is set to `true`, the Jetpack tests with be skipped. Useful to disable scheduled tests without modifying circleCI config | false | No | +## Test Framework (Selenium) + +| Name | Description | Example | Required | Store in file? | +| --------------- | -------------------------------------------------------- | ---------------- | -------- | -------------- | +| BROWSERLOCALE | Specify the target browser locale. | en | No | **NO** | +| BROWSERSIZE | Specify the target browser window size. | desktop | No | **NO** | +| HEADLESS | Configure browser headless/headful mode. | 1 | No | **NO** | +| SCREENSHOTDIR | Override the default path for screenshots to be written. | /tmp/screenshots | No | **NO** | +| TEMP_ASSET_PATH | Override the base artifacts path. | /tmp/artifacts | No | **NO** | +| TEST_VIDEO | Configure if video recording is to be used. | true | No | **NO** | + +## Test Framework (Playwright) + +| Name | Description | Example | Required | Store in file? | +| --------------- | ------------------------------------------------------------- | ---------------- | -------- | -------------- | +| DISPLAYSIZE | Specify the target browser window size. | desktop | No | **NO** | +| HEADLESS | Configure browser headless/headful mode. | 1 | No | **NO** | +| LOCALE | Specify the target browser locale. | en | No | **NO** | +| SCREENSHOTDIR | Override the default path for screenshots to be written. | /tmp/screenshots | No | **NO** | +| TEMP_ASSET_PATH | Override the base artifacts path. | /tmp/artifacts | No | **NO** | +| VIDEODIR | Override the default path for video recordings to be written. | /tmp/recordings | No | **NO** | + + diff --git a/test/e2e/docs/running-tests.md b/test/e2e/docs/running-tests.md index 39b1ac6de43062..a5abb096b9ba01 100644 --- a/test/e2e/docs/running-tests.md +++ b/test/e2e/docs/running-tests.md @@ -6,25 +6,66 @@ - [Run tests](#run-tests) - [Table of Contents](#table-of-contents) - - [Staging](#staging) - - [All tests default](#all-tests-default) - - [Spec files](#spec-files) + - [Overview](#overview) + - [Target environment](#target-environment) + - [Staging](#staging) + - [Localhost](#localhost) + - [Running tests Selenium](#running-tests-selenium) + - [All tests](#all-tests) + - [Individual spec files](#individual-spec-files) - [Individual suite](#individual-suite) + - [Running tests Playwright](#running-tests-playwright) + - [All tests](#all-tests) + - [Individual spec files](#individual-spec-files) + - [Individual suite](#individual-suite) + - [Options](#options) - [Headless](#headless) - - [Other options](#other-options) - - [Locally Development Environment](#locally-development-environment) - [TeamCity](#teamcity) - - [Sauce Labs](#sauce-labs) -## Staging +## Overview + +Calypso e2e tests use a combination of [`mocha`](https://mochajs.org/) and [`magellan`](https://github.com/TestArmada/magellan) to run its tests. + +`mocha` is the test framework and runner. +`magellan` is a test runner runner that parallelizes `mocha` instances and adds automatic retries and suite support. + +Calypso e2e when run in the CI environment uses `magellan` to spin up multiple `mocha` processes, each tasked with running one scenario (tagged with `@parallel`) from beginning to end. If any failures are encountered, the `mocha` process bails early (due to the `--bail` flag) and `magellan` schedules an automatic retry of the suite until the maximum number of attempts are reached. + +Calypso e2e when run locally can use either `magellan` for identical behavior as in CI, or alternatively invoke `mocha` directly. + +## Target environment + +### Staging + +By default, end-to-end tests run from the developer's hardware will hit the WPCOM Staging environment. This is specified inside the [`default.json`](config/default.json) file, under `calypsoBaseUrl`. + +### Localhost + +Local development environment refers to a locally served instance of the `wp-calypso` frontend. + +1. ensure required [dependencies](setup.md#software-environment#steps) are set up. + +2. change the `calypsoBaseURL` value in `test/e2e/config/default.json` to `http://calypso.localhost:3000`. + + Alternatively: create a new local-.json under `test/e2e/config` and set the `calypsoBaseURL` value to `http://calypso.localhost:3000`. + +3. start the webapp: + +```shell +yarn start +``` + +4. once webapp is started, open `http://calypso.localhost:3000` in your browser. -By default, end-to-end tests run from the developer's hardware will hit the WPCOM Staging environment. +5. ensure requests to `http://calypso.localhost:3000` are registering in your local instance. -Prior to executing any end-to-end tests, ensure the steps at [setup](setup.md) have been followed. +The local environment is now ready for testing. When a test is run, it will hit againt the local development server instead of the WordPress.com staging environment. -### All tests (default) +## Running tests (Selenium) + +### All tests ``` ./run.sh -g @@ -32,13 +73,9 @@ Prior to executing any end-to-end tests, ensure the steps at [setup](setup.md) h Configuration values for this command is read from `magellan.json`. -This is the same process used in the CI environment. - -Note that this command will search for tests tagged with `@parallel`. If you add new e2e tests, ensure it is tagged with the `@parallel` keyword. For more information, see this page. +This command will run all tests in the `test/e2e/spec` directory using `magellan` with retries enabled. -### Spec file(s) - -Calypso e2e tests use `mocha` as the test runner. +### Individual spec file(s) Specify spec file(s) directly to mocha: @@ -46,22 +83,37 @@ Specify spec file(s) directly to mocha: ./node_modules/.bin/mocha ``` -eg. +
+Example (mocha) ``` ./node_modules/.bin/mocha specs/wp-calypso-gutenberg-coblocks-spec.js ``` -Using `.only` syntax when writing tests: +
-eg. +
+Example (magellan) + +``` +yarn magellan --test=specs/wp-log-in-out-spec.js +``` + +
+ +Alternatively, append the `.only` postfix to a `describe` block: + +
+Example ``` describe.only( 'Logging In and Out:', function() { ``` -**!NOTE**: ensure this syntax should be removed once the test is to be committed to the repository. -There is an eslint rule that will prevent committing tests with the `.only` syntax, but please also exercise due diligence. +
+ +:warning: ensure this syntax should be removed once the test is to be committed to the repository. +There is an ESLint rule that checks for `.only` syntax, but please also exercise due diligence. ### Individual suite @@ -75,62 +127,100 @@ eg. ./node_modules/.bin/mocha specs/wp-calypso-gutenberg-coblocks-spec.js -g 'Insert a Pricing Table block' ``` -### Headless +## Running tests (Playwright) + +Playwright is the new framework and the intention is to migrate most (if not all) of our Selenium-based tests to this new framework. + +The steps to run calypso e2e tests developed for Playwright are very similar to Selenium-based suites. + +### All tests -By default the tests start their own Selenium server in the background, which in turn launches a Chrome browser on your desktop where you can watch the tests execute. This can be a bit of a headache if you're trying to do other work while the tests are running, as the browser may occasionally steal focus back (although that's mostly been resolved). +Specify a directory to `mocha`: -The easiest way to run "headlessly" without a visible window is to add the `-x` flag when running `run.sh`: +
+Example (mocha) ``` -./run.sh -g -x +mocha --config .mocharc_playwright.yml specs/specs-playwright ``` -or using the `HEADLESS=1` environment variable which will run Chrome with the `--headless` flag: +
+ +
+Example (magellan) ``` -export HEADLESS=1 -./node_modules/.bin/mocha +yarn magellan --config=magellan-playwright.json ``` -### Other options +
-The `run.sh` script takes a number of parameters that can be mixed-and-matched. +### Individual spec file(s) -For the list of current supported flags, use `run.sh -h`. +Specify the file(s) directly to `mocha`: -## Locally (Development Environment) +``` +mocha --config .mocharc_playwright.yml specs/specs-playwright/ +``` -Local development environment refers to a locally served instance of the `wp-calypso` frontend. +
+Example (mocha) -1. ensure required [dependencies](setup.md#software-environment#steps) are set up. +``` +mocha --config .mocharc_playwright.yml specs/specs-playwright/wp-log-in-out-spec.js +``` -2. change the `calypsoBaseURL` value in `test/e2e/config/default.json` to `http://calypso.localhost:3000`. +
-3. start the webapp as follows: +
+Example (magellan) -```shell -yarn start -./run.sh -g -u http://calypso.localhost:3000 +``` +yarn magellan --config=magellan-playwright.json --test=specs-playwright/wp-log-in-out-spec.js ``` -4. ensure requests to `http://calypso.localhost:3000` are registering in your local instance. +
-When calypso e2e test are now run, they will hit againt the local development server instead of WordPress.com staging environment. +### Individual suite -## TeamCity +Specify the name to `mocha`: -Calypso end-to-end tests have migrated to TeamCity as of 2021-01. + -Both sets of E2E Tests (desktop, mobile) are run against all branches, PRs and trunk. This process is automatic. +
+Example (mocha) + +``` +mocha --config .mocharc_playwright.yml specs/specs-playwright -g 'Subsuite 2-1 @parallel' +``` + +
-Note that interactions in TeamCity are available only to Automatticians. For OSS Citizens, please request that e2e tests be run in the PR. +## Options + +### Headless -## Sauce Labs +By default the tests start their own Selenium server in the background, which in turn launches a Chrome browser on your desktop where you can watch the tests execute. This can be a bit of a headache if you're trying to do other work while the tests are running, as the browser may occasionally steal focus back. -To run tests on Sauce Labs, add the following environment variable to CircleCI: +If using `run.sh`, add the `-x` flag: +```shell +./run.sh -g -x ``` -export SAUCE_ARG="-l osx-chrome" + +If using `mocha` or `magellan`, export an environment variable: + +```shell +export HEADLESS=1 +./node_modules/.bin/mocha ``` -Note that tests tend to run slower on Sauce Labs and this option should be used sparingly. +## TeamCity + +Calypso end-to-end tests have migrated to TeamCity as of 2021-01. + +Both sets of E2E Tests (desktop, mobile) are run against all branches, PRs and trunk. This process is automatic. + +Note that access to TeamCity is available only to Automatticians. + +OSS Citizens (including Trialmatticians), please request an Automattician to execute the required e2e tests in the PR. diff --git a/test/e2e/docs/setup.md b/test/e2e/docs/setup.md index 06ae4d37e4dec0..79f286bf94503b 100644 --- a/test/e2e/docs/setup.md +++ b/test/e2e/docs/setup.md @@ -10,22 +10,20 @@ This document will cover the environment setup process to run the `wp-calypso` e - [Table of contents](#table-of-contents) - [Software Environment](#software-environment) - [Required software](#required-software) - - [Steps](#steps) - - [Intel-based macOS](#intel-based-macos) - - [Apple Silicon-based macOS](#apple-silicon-based-macos) + - [Setup steps macOS 10.15 and 11.0](#setup-steps-macos-1015-and-110) + - [Intel architecture](#intel-architecture) + - [Apple Silicon architecture](#apple-silicon-architecture) + - [Environment Variables](#environment-variables) - [Configuration File](#configuration-file) - [Overview](#overview) - - [In-repo configuration](#in-repo-configuration) - - [Custom configurations optional](#custom-configurations-optional) - - [Environment Variables](#environment-variables) - - [Naming Branches](#naming-branches) + - [Quick start](#quick-start) + - [Custom configuration](#custom-configuration) + - [Credentials File](#credentials-file) ## Software Environment -The following instructions are geared towards macOS users. - ### Required software - [Node.js](https://nodejs.org/en/download/package-manager/#macos) @@ -37,9 +35,11 @@ Node.js can also be installed using [brew](https://nodejs.dev/learn/how-to-insta It is strongly recommended to use `nvm` to manage multiple Node.js versions. -### Steps +## Setup steps (macOS 10.15 and 11.0) -#### Intel-based macOS +### Intel architecture + +Install the list of software listed under [Required software](#required-software). Up-to-date instructions on installing individual pieces of required software can be found on their respective sites. @@ -57,11 +57,12 @@ cd /tests/e2e yarn install ``` -#### Apple Silicon-based macOS +### Apple Silicon architecture -It appears that key dependencies do not support ARM64 yet, notably `mocha`. This is true as of 2021-01 and may be subject to change in the coming months. +It appears that as of 2021-01, key NodeJS modules do not support ARM64 yet, notably `mocha`. +As with any cutting-edge software, this is subject to change in the coming weeks and months. -Currently the restrictions mean we must install and run x86_64 binaries. +To work around compatibility issues, we will install Intel architecture binaries. 1. install i386 Homebrew: @@ -75,15 +76,15 @@ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebr arch -x86_64 /usr/local/bin/brew install nvm ``` -**This is critical as if nvm is installed using ARM64 Homebrew, installed Node versions will also be the ARM64 variant!** +**This is critical! If nvm is installed using ARM64 Homebrew, installed Node versions will also be the ARM64 variant and nothing will work!** -3. using `nvm`, install the current version of node used in `wp-calypso`: +3. using `nvm`, install the current version of NodeJS used in `wp-calypso`: ``` nvm install ``` -4. use the version of node installed: +4. instruct `nvm` to use the version of NodeJS installed in previous step: ``` nvm use @@ -115,58 +116,55 @@ arch -x86_64 yarn install --frozen-lockfile At any point, run `arch` to verify whether shell is running with Rosetta 2 emulation. +## Environment Variables + +Environment Variables are values that are defined at the system level to serve as configuration for programs. + +For e2e tests, the following are required environment variables without which e2e tests will not run: + +``` +export NODE_CONFIG_ENV= +export CONFIG_KEY= +``` + +Each of these variables serve a specific purpose and will be covered in the next sections. + +Additionally, see the list of other environment variables [here](environment_variables.md). + ## Configuration File ### Overview -The tests use the node [config](https://www.npmjs.com/package/config) library to specify config values for the tests. +The tests use the node [config](https://www.npmjs.com/package/config) library to automatically load the appropriate configuration file depending on the execution environment. Under the `tests/e2e/config` directory are JSON files for predefined environments: -- `default.json` is for all environments -- `development.json` is for local -- `test.json` for CI +- `default.json` contains common values that should persist across environments. +- `development.json` is for local. +- `test.json` for CI. -It is also possible to use custom configuration files that are not part of the repo. See next section. +### Quick start -### In-repo configuration +The default configuration will suffice for most purposes. To use the default configuration, nothing needs to be changed. -There is a 'standard' configuration already in the GitHub repo under `test/e2e/config/`. +### Custom configuration -This configuration must be decrypted prior to running any e2e tests. To decrypt, please follow the steps outlined in the Field Guide. - -**Trialmatticians**: please contact a team member in your Slack chat for the decryption key. - -### Custom configurations (optional) +> :warning: **ensure username/passwords and other configuration values are not committed by accident!** Custom config files should be added under `test/e2e/config/` and should follow the naming scheme: ``` -local-.json +local-.json ``` -**Note** that `.gitignore` ensures that custom configurations prefixed with `local-` will not be commited to the repository. With tha said however, **please ensure username/passwords and other configuration values are not committed by accident!** +The `local-` prefix ensures that custom configurations will not be commited to the repository. Values found in the local configuration file will override ones found in `default.json`. -This is useful to test various configurations in the local environment. -e.g. testing on local Calypso instance, instead of production by setting the `calypsoBaseURL` property to `http://calypso.localhost:3000`. +For the full list of possible configuration values, refer to the following page: [config values](config_values.md). -For the full list of possible configuration values, please see the following page: [config values](config_values.md). +## Credentials File -## Environment Variables +In addition to the configuration files, there is an encrypted file that holds test account credentials. This must be decrypted prior to use. For instructions on how to decrypt this file, please refer to the Field Guide page. -Environment Variables are values that are defined at the system level to serve as configuration for programs. - -For e2e tests, the following are required environment variables without which e2e tests will not run: - -``` -export NODE_CONFIG_ENV= -export CONFIG_KEY= -``` - -Additionally, see the list of other environment variables [here](environment_variables.md). - -## Naming Branches - -Please refer to the Automattic [branch naming scheme](https://github.com/Automattic/wp-calypso/blob/HEAD/docs/git-workflow.md#branch-naming-scheme). +**Trialmatticians**: please contact a team member in your Slack chat for the decryption key. diff --git a/test/e2e/docs/style-guide.md b/test/e2e/docs/style-guide.md index 11c2317caa40c7..bd7e4b38b9a79d 100644 --- a/test/e2e/docs/style-guide.md +++ b/test/e2e/docs/style-guide.md @@ -3,8 +3,13 @@ - [Style Guide](#style-guide) + - [Naming Branches](#naming-branches) - [Async / Await](#async--await) - [Tags](#tags) + - [At most 1 top-level describe block](#at-most-1-top-level-describe-block) + - [Viewport size](#viewport-size) + - [Specify one viewport](#specify-one-viewport) + - [Specify multiple viewports](#specify-multiple-viewports) - [Page Objects](#page-objects) - [Use of this, const and lets](#use-of-this-const-and-lets) - [Arrow functions](#arrow-functions) @@ -15,6 +20,10 @@ +## Naming Branches + +We follow the Automattic [branch naming scheme](https://github.com/Automattic/wp-calypso/blob/HEAD/docs/git-workflow.md#branch-naming-scheme). + ## Async / Await We use async functions and `await` to wait for commands to finish. This lets asynchronous methods execute like synchronous methods. @@ -47,7 +56,7 @@ async function openModal() { ## Tags -Tags are labels used by `mocha` (our test runner) to determine what tests should be run, depending on the environment. Consider it a form of metadata that conveys various test parameters to the runner. +Tags are labels used by `mocha` and `magellan` to determine what tests should be run and how it can be parallelized. Consider it a form of metadata that conveys various test parameters to the runner. Typical example: @@ -73,28 +82,46 @@ Some examples of tags: - jetpack - signup -## Modes +Notably, test suites not tagged with the `@parallel` tag will not be recognized by `magellan` as a valid test suite and thus will not be run in CI. + +Furthermore, if any test steps fail within the suite tagged with `@parallel` tag, rest of the test steps will be skipped an execution will begin on a new test suite. + +## At most 1 top-level describe block + +Each test file should only contain at most 1 top-level `describe` block. + +There is no restriction on the number `describe` blocks that are not top-level, nor a restriction on the depth of `describe` blocks. + +## Viewport size All tests should be written to work in three modes: desktop (1440 wide), tablet (1024 wide) and mobile (375 wide). Tests can be run in different modes by setting an environment variable `BROWSERSIZE` to either `desktop`, `tablet` or `mobile`. -### Specify one mode +### Specify one viewport + +eg. using environment variables ``` -env BROWSERSIZE= ./node_modules/.bin/mocha +BROWSERSIZE= ./node_modules/.bin/mocha ``` -Alternatively, use the `-s` flag when calling `run.sh`: +eg. using run.sh ``` -./run.sh -g -s +./run.sh -g -s ``` -### Specify multiple modes +### Specify multiple viewports ``` -./run.sh -g -s , +./run.sh -g -s , +``` + +eg. using run.sh + +``` +./run.sh -g -s mobile,desktop ``` ## Page Objects @@ -178,7 +205,7 @@ step( 'We can set the sandbox cookie for payments', async function() { Use destructuring for default values as this makes calling the function explicit and avoids boolean traps. -Avoid +Avoid: ``` constructor( driver, visit = true, culture = 'en', flow = '', domainFirst = false, domainFirstDomain = '' ) { @@ -204,7 +231,7 @@ new StartPage( driver, true, 'en', '', true, '' ).displayed(); ## Nesting step blocks -Since we have a bail suite option, it is not necessary to nest `step` blocks. +Do not nest test steps. This is a general structure of an e2e test scenario: @@ -212,17 +239,15 @@ This is a general structure of an e2e test scenario: describe( 'An e2e test scenario @parallel', function() { - - before( async function() { return await driverManager.ensureNotLoggedIn( driver ); } ); - step( 'First step', async function() { + it( 'First step', async function() { // Do something with a page } ); - step( 'Second step', async function() { + it( 'Second step', async function() { // Do something next - this will only execute if the first step doesn't fail } ); @@ -233,8 +258,6 @@ describe( ); ``` -**Note:** The `describe` blocks shouldn't be `async` - ## Catching errors in a step block Sometimes we don't want a `step` block to fail on error - say if we're cleaning up after doing an action and it doesn't matter what happens. As we use async methods using a standard try/catch won't work as the promise itself will still fail. Instead, return an async method that catches the error result: @@ -256,7 +279,7 @@ step( 'Can delete our newly created account', async function() { When waiting for elements we should always use a quantity of the config value defined as `explicitWaitMS` instead of hardcoding values. This allows us to change it readily, and also adjust this for different environments, for example the live branch environment is not as fast as production so it waits twice as long. -Instead of: +Avoid: ``` export default class TransferDomainPrecheckPage extends AsyncBaseContainer { @@ -266,7 +289,7 @@ export default class TransferDomainPrecheckPage extends AsyncBaseContainer { } ``` -do: +instead: ``` export default class TransferDomainPrecheckPage extends AsyncBaseContainer { @@ -280,5 +303,3 @@ export default class TransferDomainPrecheckPage extends AsyncBaseContainer { } } ``` - -this achieves the same thing as the default explicit wait is presently 20000, and it allows us to adjust for environmental performance. diff --git a/test/e2e/docs/writing_tests.md b/test/e2e/docs/writing_tests.md index 34026bfc1cf37c..a778d4f29b39ca 100644 --- a/test/e2e/docs/writing_tests.md +++ b/test/e2e/docs/writing_tests.md @@ -1,6 +1,6 @@ # Writing Tests -This document will outline tips to write successful tests. +This document will outline tips to write successful tests for both Selenium and Playwright suites. Refer to the [style guide](docs/style-guide.md) for coding style information. @@ -11,9 +11,8 @@ Refer to the [style guide](docs/style-guide.md) for coding style information. - [Writing Tests](#writing-tests) - [Table of contents](#table-of-contents) - [Selector](#selector) - - [How](#how) - [Component](#component) - - [Page Object Model](#page-object-model) + - [Page](#page) - [Structure](#structure) - [Guidelines](#guidelines) - [Flow](#flow) @@ -35,30 +34,30 @@ Ideally, a selector satisfies all of the following: - **reliable**: the same element is selected with each iteration. - **brief**: selector is short and easy to read. -### How - -Use the browser's built-in Web Inspector tool to find selectors. - ## Component -Components are smaller in scope than page objects, but fundamentally the concept is to capture attributes and actions associated with page elements that can appear across multiple pages. +Components cover elements that persist across multiple pages. -On `wp-calypso`, some possible components may be: +Encapsulating behavior of a component in an object permits code reuse, promotes object oriented thinking and separation of duties. + +On `wp-calypso`, some components are: - left sidebar - master bar -## Page Object Model +## Page + +Page Object Model (or _POM_ for short) is a common technique used for automated end-to-end testing. -Page Object Model (or _POM_ for short) is a common technique used with Selenium WebDriver-based automated tests. +Similar to a `Class` in software development, the POM groups together attributes, functions and other code on a page. -Similar to a `Class` in software development, the POM groups together various attributes, functions and other code related to a certain page. Automated e2e tests interact with the page through an instance of the POM. +Automated end-to-end tests create instances of page objects to invoke actions on the page. -THe use of POM encourages the following: +Similar to comonents, page objects encourage: -- **Don't Repeat Yourself (DRY)**: actions can be called from POM instead of being re-implemented each time. -- **maintainability**: if a page changes, update the corresponding POM. -- **readability**: named variables and functions are much easier to decipher than series of actions. +- **Don't Repeat Yourself (DRY)**: common actions can be called from the page object. +- **maintainability**: if a page changes, update the page object at one spot. +- **readability**: named variables and functions are much easier to decipher than series of strings. Developers should add a new page object under `test/e2e/lib/pages` upon completion of a feature that adds a new page not covered by existing page objects. @@ -111,7 +110,7 @@ For the purpose of this document however, a flow typically refers to a set of ac ### Structure -Similar to the POM, flows are at heart JavaScript classes that perform actions on elements. +Flows are larger in scope than page objects, typically executing actions across multiple (related) pages. ``` external dependencies @@ -151,7 +150,7 @@ export default class LoginFlow { ### Guidelines - aggressively refactor such that basic actions can be extended by other functions. -- tightly control scope so that flows do not become the e2e tests. +- tightly control scope so that flows do not become the e2e tests themselves. ## Gutenberg Blocks