Skip to content

Testing Strategy

Jump to bottom
Katka Pilátová edited this page Apr 2, 2024 · 66 revisions

In this document you can find information about the CLI test suite and guidelines for writing tests for the CLI.

Test suites

On this project, we distinguish three types of functional tests by scope - unit tests, integration tests and E2E tests. All tests are written using Vitest.

Unit tests

  • Their scope refers to an isolated unit of code.
  • Aim to only test the internal logic and use mocks where needed in order to narrow down the testing scope (file system and other).
  • These tests are located next to their production code counterpart and have the .unit.test.ts suffix.
  • One can run unit tests for a given package via npx nx run <package>:unit-test
  • One can run all unit tests via npx nx run-many -t unit-test

Integration tests

  • Their scope refers to a bigger unit of code in isolation or an integration with a 3rd-party tool.
  • Aim to test the output of the unit as a whole. When communicating with a 3rd-party tool, only tests its usage in CodePushup.
  • These tests are also located next to their production code counterpart and have the .integration.test.ts suffix.
  • Run integration tests for a given package via npx nx run <package>:integration-test
  • Run all integration tests via npx nx run-many -t integration-test

End-to-end tests

  • Test the whole CLI application and use actual deployed environment.
  • These tests are located in a separate project cli-e2e and their test files have the .e2e.test.ts suffix.
  • Run all E2E tests via npx nx run cli-e2e:e2e.

Automation

The test suites are automatically run within the CI workflow in the following scenarios:

  • All affected tests shall be run inside a pull request for all major OS (Linux, Windows, MacOS) (see CI pipeline).
  • Each affected test suite shall have its linter run inside a pull request.
  • All tests are run on one platform after a pull request is merged to update the code coverage (see code coverage pipeline).

Note

In order to maintain a smooth DX, integration tests with a longer run-time (above 20 seconds) are excluded by default when executing tests locally. To include all tests, define INCLUDE_SLOW_TESTS=true in your .env file.

Testing utilities

In order to keep the maintenance cost lower, there is a folder dedicated to testing utilities called testing. Any fixtures, helper functions, constants or mocks usable by multiple tests should be added to the test-utils subfolder.

The folder structure should be fixtures for fixed data or examples and utils for utility functions.

The exception to this may be a utility which is applicable only to the current package. In this case, each package shall use the mocks folder to mirror the hierarchy of the testing-utils - fixtures go into fixtures subfolder and functions into utils subfolder. TypeScript is set up to include this folder only for its test configuration.

Mocking

Each test suite has a predefined set of setup files which mock appropriate parts of the logic. These mock setups may be found in the setup folder of the testing-utils library.

One may apply them inside vitest.(unit|integration|e2e).config.ts in the following way:

export default defineConfig({
  // ...
  test: {
    // ...
    setupFiles: [
      // any setup files go here, e.g. for mocking the console
      '../../testing-utils/src/lib/setup/console.mock.ts',
    ],
  },
});

Tip

There is an additional file reset.mocks.ts which may be included in the setupFiles to automatically reset your mocks.

The main setup is currently divided between unit and integration tests in the following way:

  • Unit tests always have the console, file system and current working directory (points to the in-memory file system) mocked.
  • Integration tests always have the console mocked.

Test outputs

In order to keep the repository clean, test outputs should target ignored folders, such as node_modules or tmp.

The tmp folder flow should be as follows:

  • Any test that needs outputs should create its own subfolder under tmp, e.g. in beforeAll of a given test suite.
  • After the test suite is run, this subfolder should be removed in afterAll.

Important

A test suite should not expect the tmp folder to be created. Therefore, the subfolder should be created via mkdir with { recursive: true }. Likewise, a test suite should not remove the tmp folder itself, only the subfolder needed for the given test suite.

Tip

setupTestFolder and teardownTestFolder utility functions from test-folder-setup may be used.

Test quality

Good practices

Motivation

I found it best to imagine I am writing tests for a colleague who hasn't implemented that feature. They are most likely reviewing my code, debugging a failing test or checking the test coverage. So they want to either find out what behaviour is being tested (and how) or where the tested values come from. And the less time they spend on trying to understand the tests, the better.

Principles

The one message that overlaps all the good practices is the optimisation for DAMP (Descriptive and Meaningful Phrases).

Important

In test code, most time is spent by reading it and maintaining it.

Your test code is most likely to be looked at by other people than yourself. Therefore, the most important quality of your test code is readability. The following principles should be followed when writing tests:

  • The test description should clearly define the use case for which the test was written (no should return correct data). It should not merely describe the implementation (no when X is set to true, Y is returned).
  • The test implementation should cover the defined use case (and only that use case) in an expected way. This means no need for investigation of what the test is doing - it should be obvious.
  • A test should only focus on and verify relevant parts of a given use case.
  • The test flow should be linear (no loops or conditionals) and one should be able to read it from top to bottom without having to scroll around or look into other files too much.
  • Use explicit values. A test knows which exact values it uses. Using too many helper variables and functions can obscure the logic and coverage and increases the time necessary to understand the test.
  • Realistic values should be used (no mock-data or similar). The test should tell a story which involves an actual user interacting with the application.
  • If it is necessary to use variables or helper functions, their name should be self-explanatory (no mock()). The nature of the entity should also be clear (refrain from using names similar to production code).

Now, that doesn't mean that DRY (Don't Repeat Yourself) doesn't have its place in the test code. It is a balance (see relevant discussion). Here are some of our good practices based on the DRY principle:

  • There should only be one test that verifies a specific behaviour. That test should be implemented for the smallest unit possible where the use case is testable.
  • If the same function or flow is tested with different sets of inputs (most of the implementation is the same), dynamic testing (it.each) should be used.
  • Reusable testing utilities should live in testing-utils, see Testing utilities section.
  • Shared setup should be configured in a vitest configuration file, see Mocking section.

ESLint rules

CI

Vitest is a powerful tool. Its methods should be fully utilised to serve the tests. In order to ensure the quality of our test code and promote Vitest's good practices, we have a set of ESLint rules in place for our test files. We use our own set of ESLint rules which can be found and reused from @code-pushup/eslint-config. The test rules can be found in @code-pushup/eslint-config/vitest.

Run ESLint on a specific package via npx nx run <package>:lint.

Code coverage

codecov

Code coverage is a metric that measures the percentage of lines of code run in automated tests.

Note

Code coverage should not to be confused with test coverage which aims to measure how much of the application logic based on business criteria is covered by tests.

We use codecov tool to measure the ratio of Code PushUp logic executed within our test suites. The code coverage is measured for unit and integration tests only (E2E tests only test critical happy paths).

Our code coverage trends may be found here.

Clone this wiki locally