Transitioning End-to-end Tests to More Useful Test Types

[pwolfert]

RFC: Transitioning End-to-end Tests to More Useful Test Types

Date: October 04, 2021
Author: @scheul93
Original PR: #1169
Status: Approved

Background

End-to-End Test Definition

"End to end testing (E2E testing) refers to a software testing method that involves testing an application’s workflow from beginning to end. This method basically aims to replicate real user scenarios so that the system can be validated for integration and data integrity." -BrowserStack

State of e2e Testing in Core DS

Currently, the core Design System has end-to-end tests that are not really end-to-end tests because they do not test a component's full workflow or integration with other technologies (like APIs).

Many components have a .e2e.js test file which contains:

• Accessibility test using axe-core
• A browser test to confirm that a component is rendered at a specific doc site URL (using Selenium)
• Some contain more detailed interaction tests that appear equivalent to unit tests

Problem

End-to-end tests in the core DS are not helpful tests for the Design System because they are redundant with unit tests, they are unstable and, as a result, slow down the development and release processes.

Proposal

The core DS should stop using the term "end-to-end" tests and instead focus on other forms of testing that can achieve the same results as the current end-to-end tests.

This includes focusing on:

1. Unit tests
2. Accessibility Tests
3. Visual Regression Tests

Unit Tests

"Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended" -Wikipedia

Currently, every component has unit tests which test the functionality of a "unit" of code (in this case, a "unit" is a component). For any component who's end-to-end tests are testing functionality, ensure that those test cases are covered by unit tests.

Accessibility Tests

Accessibility tests in the core Design System codebase are a suite of automated tests that use axe-core by Deque to check for accessibility violations.

Right now, each .e2e.js file includes an accessibility check for a specific component. It is recommended that these tests are kept, but pull them into their own file (or rename the current file) to signify that they are accessibility tests, not end-to-end tests.

Accessibility Test Considerations

If the Design System team chooses to move forward with implementing Storybook, the current accessibility tests may be able to be improved or replaced with Storybook-integrated tests. See Storybook RFC for details.

Visual Regression Tests

"A visual regression test checks what the user will see after any code changes have been executed by comparing screenshots taken before and after code changes. This is why visual regression tests are also sometimes called visual snapshot tests." - BrowserStack

The core Design System has visual regression tests run with backstop.js. Visual regression tests should be able to provide the same information that some of the current end-to-end tests provide. For example, most of the e2e tests will attempt to render a component to the browser and check that it rendered. A visual regression test should be able to do the same thing. It will raise a warning / error of a component's snapshot looks different than the approved snapshot, so if a component fails to render, a warning should be raised.

Visual Regression Test Considerations

It would be ideal if the visual regression testing tool could be configured to support browsers outside of Chrome. Then, there would be tests for some of the "trickier" browsers like IE or Safari.

Implementation Timeline

Transitioning away from end-to-end tests and to more useful tests could be done in the following steps:

1. Ensure that unit tests cover any functionality currently in e2e tests
2. Remove obsolete e2e tests (browser tests) that are now fully covered by unit tests
3. Rename ".e2e.test.js" files to be ".a11y.test.js" since only tests included should be accessibility tests
4. Have visual regression tests run automatically as part of merge or deployment pipeline

Future Testing Considerations

Not directly related to end-to-end tests, but future testing improvements to consider:

• Is Backstop.js the correct visual regression testing technology? Are there alternatives?
  • Visual regression tests are explored more in Storybook RFC
• Simplify unit test technologies. Commit to either Enzyme or Testing Library.

Benefits

Moving forward with these changes will:

• Remove unstable tests from codebase to allow for faster development and releases
• Allow the DS team to focus on more meaningful testing efforts
• Reduce number of testing technologies the DS team will have to support
• Child DS could implement similar test setups for improved testing throughout the ecosystem
• Allow the opportunity to expand test coverage to other browsers / environments

Risks

Moving forward with these changes:

• Will take development effort
• Will necessitate updates with Child Design Systems by either updating existing child DS tests, or creating some in the same pattern
• Will look scary since e2e test files will be removed. But per the above plan, test quality should be the same or improved