Skip to content

TESTING.md

Latest commit

 

History

History
231 lines (168 loc) · 9.57 KB

TESTING.md

File metadata and controls

231 lines (168 loc) · 9.57 KB

Testing Cascade

Table of Content

Unit tests

Unit tests can be run as usual for Rust projects using cargo test:

  1. cargo test
  2. cargo test --no-default-features
  3. cargo test --all-features

Integration/System testing with act

The GitHub Action workflow in integration-tests/system-tests.yml is currently only for use with https://github.com/nektos/act via the act-wrapper script at the root of this repository, which creates a custom container with a freshly built cascade.

TL;DR

Run all tests with: ./act-wrapper

Run a single test with: ./act-wrapper --job your-test

Create a new test with:

  • ./integration-tests/scripts/add-test.sh your-test "Your test name"

Creating a test

The workflow file integration-tests/system-tests.yml only contains "stub" runners for the tests, with the tests themselves being written in actions in integration-tests/tests/.

You can easily generate the scaffolding for a test with the script ./integration-tests/scripts/add-test.sh <job-name> "<test name/description>" [<PR-number>].

The test environment provides a few nameservers and zones for use in tests (see section Provided Nameservers and Zones).

Running single jobs/tests

You can run single jobs with act using the --job option. However, if the job has the needs option set to depend on other jobs, those jobs will always be run before.

Overriding the build profile to use

By default tests are run using debug profile builds of the Cascade binaries. Except for tests that explicitly choose the debug or release build profile, the build profile used can be overridden by invoking ./act-wrapper with argument --input build-profile=XXX where XXX must be one of debug or release.

Overriding the log level to use

By default Cascade is configured to log at debug level. You can change this by invoking ./act-wrapper with argument --input log-level=XXX where XXX must be one of error, warning, info, debug or trace.

Network requirement (why --network default)

In the test environment, Unbound needs to bind to localhost:53, which is not possible with act's default network. This is because localhost:53 is already in use by your system's stub resolver (probably systemd-resolved). Instead of act's default network selection (which instructs Docker/Podman to use the host's network), you need to specify a different container network to use. Docker and Podman each provide default networks (not to be confused with act's default network selection, which is Docker/Podman's host network). Docker's default network is called default, while Podman's default network is called podman. Therefore, the act-wrapper automatically sets the --network option for act. If you want to use a different network than the default one, you can simply run act-wrapper --network <your-network> and it will override the default network set by the act-wrapper.

Limitations

No init or systemd

Act runs the workflow in a container without init or systemd. Therefore, when running other daemons, you either need to make use of their appropriate daemonization features, or handle background jobs yourself.

Maybe running act with --container-options --init would work to add a dumb init process, but isn't verified, yet.

All nameservers on the same address

Currently, it is not possible to add additional listener addresses on the loopback (or any) network device in the act container. Therefore, all nameservers are listening on 127.0.0.1 on different ports:

  • Unbound: 127.0.0.1:53
  • Primary NSD: 127.0.0.1:1055
  • Secondary NSD: 127.0.0.1:1054
  • Bind (authoritative for .test): 127.0.0.1:1053

It might be possible to change this in future versions of this setup with the --container-options --cap-add=NET_ADMIN option for act, but this needs to be tried out.

Managing act's verbosity

act prints a lot of information on the terminal. To reduce or manage the amount of text printed you can:

  • Use act's --concurrent-jobs 1 option to limit the number of jobs run by act at once, which will avoid interlacing output of different jobs.
  • Use the --quiet option to disable logging of output from steps, which reduces the amount of output generated by act. However, by using this option you might miss valuable output when a test fails and have to re-run the test.
  • Write the output to a file with act ... 2>&1 | tee /tmp/act.log (optionally using unbuffer from the expect package; left as an excercise for the user)

Running act with Podman

act uses Docker. If you want to use Podman instead, you will need to enable the Podman daemon and set the DOCKER_HOST variable accordingly. If you are using rootless Podman, you will likely need to run systemctl --user enable --now podman.socket and set DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock in your shell's rc file (e.g. .bashrc). With that set, all docker programs will use Podman as their backend instead of the Docker daemon. Additionally, you will need to make sure that docker is in your path. As you are using Podman, you can create a symlink, which instructs Podman to act in a Docker compatibility mode: sudo ln -s "$(which podman)" /usr/local/bin/docker

Miscellaneous notes

  • By default, tests are run using a debug build for both Cascade and dnst.
    • This can be changed per test using the set-build-profile action: 
      - uses: ./.github/actions/set-build-profile
  with:
    build-profile: release
  • cascade, cascaded, and dnst are added to the $PATH.
  • By default, cascade is configured to use the directory: ${GITHUB_WORKSPACE}/cascade-dir
  • The default paths for configuration files can be fetched using the integration-tests/scripts/get-default-path.sh script (this script is intended to be used from the default working directory of the test job; aka do not cd somewhere, or the reported paths will be wrong).
  • The workflow action .github/actions/setup-and-start-cascade also generates a default policy with cascade template policy.
  • If you encounter the error bash: /root/cargo-debug/bin/cascaded: cannot execute: required file not found, you can run ./act-wrapper +build-inside ... to build cascade inside of the container build step. This lacks the benefit of cargo's build caching, but works around the issue until it is properly fixed.

Docker dependencies

When using Docker to run the integration tests, you need to make sure that the docker-buildx plugin is installed, otherwise Docker will complain about unknown flags.

Provided nameservers and zones

The test environment provides a number of nameservers (a primary NSD, a secondary NSD, Bind, and the resolver Unbound) and the zone example.test. and it's parent test..

Unbound is used as the system's stub resolver forwarding most queries to Quad9 or Cloudflare. Queries to test. are redirected to Bind on port 1053 and queries to example.test. are redirected to the secondary NSD instance on port 1054.

Bind is configured as an authoritative for the zone test. and is used to enable updating the zone test. during a test, e.g. with dnst update, to update the DS RR for example.test., without having to fiddle with modifying the zonefile (but you still can). (Currently, the zone test. is not signed, which is ok for the current implementation of dnst keyset, but this may need to change in the future.)

Both NSD instances are configured as authoritative for example.test.. The primary NSD loads the zone from a zonefile and provides AXFR and IXFR to anyone with IP 127.0.0.1. The secondary NSD is configured to transfer the zone from Cascade (currently always using AXFR). Both NSD instances allow notifies from 127.0.0.1.

Monitoring resource usage

Using ./act-wrapper +stats-report </path/to/write/report.csv> or ./act-wrapper +stats-graph </path/to/write/report.png> will generate a report on approximate resource usage covering the period of the execution of the tests, as CSV raw data and/or as a PNG formatted graph.

Measured resources are CPU and memory, as reported by the docker stats command.

Note that this is only intended to be used when a single test is run at once, i.e. when using --job XXX or --concurrent-jobs 1. If there are containers running prior to execution of the tests attempting to use +stats-report or +stats-graph or if multiple lines of statistics are output by docker stats (i.e. for more than one active container) an error will be output as the numbers are assumed to be for a single active container at a time.

Graph generation requires that you have the gnuplot tool installed.