|
| 1 | +# CN Testing |
| 2 | + |
| 3 | +CN has testing capabilities available via the `cn test` subcommand. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +Currently, CN supports only per-function tests, but additional types of testing may become available in the future. |
| 8 | + |
| 9 | +Running `cn test <filename.c>` generates C files with the testing infrastructure, the instrumented program under test, and a build script named `run_tests.sh`. |
| 10 | +This script compiles the C files and runs the tests. |
| 11 | + |
| 12 | +By default, running `cn test` will automatically run `run_tests.sh`, which produces a test executable `tests.out`. |
| 13 | +This can be disabled by using the `--no-run` flag. |
| 14 | + |
| 15 | +The output directory for these files can be set by using `--output-dir=<DIR>`. |
| 16 | +If the directory does not already exist, it is created. |
| 17 | + |
| 18 | +### Per-function tests |
| 19 | + |
| 20 | +When testing, there are currently two types of tests, constant tests and generator-based tests. |
| 21 | +For *each function with a body*, CN will create either a constant test or generator-based test. |
| 22 | + |
| 23 | +If a function takes no arguments, does not use accesses on global variables, and is correct, it should always return the same value and free any memory it allocates. |
| 24 | +In this case, a constant test is generated, which runs the function once and uses [Fulminate](FULMINATE_README.md) to check for post-condition violations. |
| 25 | + |
| 26 | +In all other cases, it creates generator-based tests, which are in the style of property-based testing. |
| 27 | +A "generator" is created, which randomly generates function arguments, values for globals accessed and heap states, all of which adhere to the given function's pre-condition. |
| 28 | +It calls the function with this input and uses [Fulminate](FULMINATE_README.md) similar to the constant tests. |
| 29 | + |
| 30 | +#### Understanding errors |
| 31 | + |
| 32 | +Currently, the tool does not print out counterexamples, but this is [planned](https://github.com/rems-project/cerberus/issues/841). |
| 33 | +Until then, `tests.out` can be run with the `--trap` flag in a debugger. |
| 34 | +Since seeds are printed each time the tool runs, `--seed <seed>` can be used to reproduce the test case. |
| 35 | +The debugger should automatically pause right before rerunning the failing test case. |
| 36 | + |
| 37 | +#### Writing custom tests |
| 38 | + |
| 39 | +There is currently no way to write custom property-based tests. |
| 40 | +However, once lemmas can be tested, a lemma describing the desired property could be written to test it. |
| 41 | + |
| 42 | +In terms of unit tests, one can simply define a function that performs the desired operations. |
| 43 | +This function will get detected by `cn test` and turned into a constant test. |
| 44 | +Any assertions that one would make about the result would have to be captured by the post-condition. |
| 45 | +In the future, existing infrastructure like `cn_assert` might be adapted for general use. |
0 commit comments