Skip to content

Commit d7d4e67

Browse files
committed
[CN-Test-Gen] Add TESTING.md
1 parent df0d280 commit d7d4e67

File tree

1 file changed

+45
-0
lines changed

1 file changed

+45
-0
lines changed

backend/cn/TESTING.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# CN Testing
2+
3+
CN has testing capabilities available via the `cn test` subcommand.
4+
5+
## Overview
6+
7+
Currently, CN supports only per-function tests, but additional types of testing may become available in the future.
8+
9+
Running `cn test <filename.c>` generates C files with the testing infrastructure, the instrumented program under test, and a build script named `run_tests.sh`.
10+
This script compiles the C files and runs the tests.
11+
12+
By default, running `cn test` will automatically run `run_tests.sh`, which produces a test executable `tests.out`.
13+
This can be disabled by using the `--no-run` flag.
14+
15+
The output directory for these files can be set by using `--output-dir=<DIR>`.
16+
If the directory does not already exist, it is created.
17+
18+
### Per-function tests
19+
20+
When testing, there are currently two types of tests, constant tests and generator-based tests.
21+
For *each function with a body*, CN will create either a constant test or generator-based test.
22+
23+
If a function takes no arguments, does not use accesses on global variables, and is correct, it should always return the same value and free any memory it allocates.
24+
In this case, a constant test is generated, which runs the function once and uses [Fulminate](FULMINATE_README.md) to check for post-condition violations.
25+
26+
In all other cases, it creates generator-based tests, which are in the style of property-based testing.
27+
A "generator" is created, which randomly generates function arguments, values for globals accessed and heap states, all of which adhere to the given function's pre-condition.
28+
It calls the function with this input and uses [Fulminate](FULMINATE_README.md) similar to the constant tests.
29+
30+
#### Understanding errors
31+
32+
Currently, the tool does not print out counterexamples, but this is [planned](https://github.com/rems-project/cerberus/issues/841).
33+
Until then, `tests.out` can be run with the `--trap` flag in a debugger.
34+
Since seeds are printed each time the tool runs, `--seed <seed>` can be used to reproduce the test case.
35+
The debugger should automatically pause right before rerunning the failing test case.
36+
37+
#### Writing custom tests
38+
39+
There is currently no way to write custom property-based tests.
40+
However, once lemmas can be tested, a lemma describing the desired property could be written to test it.
41+
42+
In terms of unit tests, one can simply define a function that performs the desired operations.
43+
This function will get detected by `cn test` and turned into a constant test.
44+
Any assertions that one would make about the result would have to be captured by the post-condition.
45+
In the future, existing infrastructure like `cn_assert` might be adapted for general use.

0 commit comments

Comments
 (0)