Skip to content

Commit 1d5e40b

Browse files
committed
docs(check): update the test suite README
1 parent 3161237 commit 1d5e40b

File tree

1 file changed

+212
-46
lines changed

1 file changed

+212
-46
lines changed

check/README.md

Lines changed: 212 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,87 +1,89 @@
1-
FORM Test Suite
2-
===============
1+
# FORM Test Suite
32

43
This directory contains a collection of test cases that can be used for
54
verifying the behaviour of FORM. It also has a script to run the test cases and
65
check the results.
76

8-
Prerequisites
9-
-------------
7+
## Prerequisites
108

119
The test runner script is written in [Ruby](https://www.ruby-lang.org/)
12-
and requires Ruby 1.9 or later. The script uses the so-called `test/unit`
10+
and requires Ruby 2.0 or later. The script uses the so-called `test/unit`
1311
library. In some Linux distributions the library is installed together with
1412
Ruby, while some distributions may have the library as an optional package,
1513
or one may need to manually install
1614
[test-unit](http://test-unit.github.io/test-unit/en/) via the `gem` command.
17-
Currently, the script runs only on Unix-like systems.
1815

19-
Usage
20-
-----
16+
## Usage
2117

2218
### From the build system
2319

2420
To use the test suite from the automatic build system
2521
(see also the [INSTALL](../INSTALL) file),
26-
run
22+
run the following command:
2723

28-
```
24+
```bash
2925
# in the root build directory
3026
make check
3127
```
3228

3329
which tests the executables (release versions) compiled by the build system.
3430

35-
### Testing in the standalone mode
31+
### Testing in standalone mode
3632

3733
Alternatively, one can run the test runner script directly:
3834

39-
```
35+
```bash
4036
# in the "check" directory
4137
./check.rb
4238
```
4339

44-
By default, it tests `form` found in $PATH.
45-
To check another executable, give the path as a command line option:
40+
By default, this tests `form` found in `$PATH`.
41+
To test another executable, specify its path as a command-line argument:
4642

47-
```
43+
```bash
4844
./check.rb /path/to/form
4945
```
5046

51-
One can specify a TFORM (or ParFORM) executable in this way.
47+
One can also specify a TFORM (or ParFORM) executable in this way.
5248
TFORM and ParFORM will be run with 4 CPUs (can be changed by the `--cpu N`
5349
option).
5450

5551
By default, all test cases in all FORM files (`*.frm`) found in the `check`
5652
directory (not in subdirectories) are used. To select test cases or FORM files
57-
to be run, give their names as command line options, for example,
53+
to be run, specify their names as command-line arguments. For example:
5854

59-
```
60-
./check.rb examples.frm
55+
```bash
6156
./check.rb Issue8
57+
./check.rb 'divmod_*'
58+
./check.rb examples.frm
6259
```
6360

64-
For more advanced options, see the help message shown by the `--help` option.
61+
For more advanced options, refer to the help message using the `--help` option.
6562

66-
Writing tests
67-
-------------
63+
## Writing tests
6864

6965
### Where to add test cases?
7066

71-
Currently, the standard test set (run by default) consists of 3 files:
67+
Currently, the standard test set (run by default) consists of 4 files:
7268

73-
- `examples.frm`: Examples found in the manual.
69+
- `examples.frm`: Examples provided in the manual.
7470
- `features.frm`: Test cases for newly added features.
7571
- `fixes.frm`: Test cases for bug fixes.
72+
- `user.frm`: Test cases contributed by users.
7673

7774
Each test case in these files should finish in a short time: the timeout is set
7875
to 10 seconds. Bigger tests that take more time are put in subdirectories
79-
(e.g., forcer) and should be specified by command-line options when the test
80-
suite is invoked.
76+
(e.g., `forcer`) and should be specified by command-line options when the test
77+
suite is invoked:
78+
79+
```bash
80+
./check.rb -C forcer # The Forcer library must be available in FORMPATH.
81+
```
8182

8283
### Structure of a test case
8384

84-
A test case is given as a fold in a FORM file. A simple example is:
85+
A test case is given as a fold in a FORM file.
86+
The following is a simple example:
8587

8688
```
8789
*--#[ Test1 :
@@ -95,16 +97,22 @@ assert result("F") =~ expr("1 + 2*x + x^2")
9597
```
9698

9799
The fold name `Test1` gives the name of the test case, which should be unique.
98-
The part before `.end` is a normal FORM program. After `.end`, one can write
99-
a Ruby program to check the results. In this example, `assert` method (which is
100-
provided by some unit test class) is used for checking whether its argument is
101-
`true`. The first assertion checks `succeeded?`, which gives `true` if the FORM
102-
successfully finishes. The second assertion checks the printed result of the
103-
expression `F` by a regular expression matching (`=~`). In the left-hand side,
104-
`result("F")` returns the (lastly) printed output for the expression `F` as
105-
a string. In the right-hand side, `expr("...")` makes a regular expression with
106-
removing white spaces in its argument. Since `expr()` removes all white spaces,
107-
one can also put new lines, for example,
100+
The part before `.end` is a normal FORM program.
101+
After `.end`, one can write a Ruby program to check the results.
102+
103+
The `assert` method checks whether its argument evaluates to `true`.
104+
In this example:
105+
- The first assertion verifies `succeeded?`, which returns `true` if the FORM finishes successfully.
106+
- The second assertion checks the printed result of the
107+
expression `F` by a regular expression matching (`=~`).
108+
- On the left-hand side, `result("F")` returns the (lastly) printed output
109+
for the expression `F` as a string.
110+
- On the right-hand side, `expr("...")` creates a regular expression
111+
by removing white spaces in its argument.
112+
113+
Since `expr()` removes all white spaces,
114+
one can include new lines in the argument.
115+
For example:
108116

109117
```
110118
*--#[ Test2 :
@@ -123,13 +131,171 @@ assert result("F") =~ expr("
123131

124132
which is convenient to copy and paste a long output from a terminal.
125133

126-
### Tips
134+
Two or more FORM programs, separated by `.end`, can be put in a test case.
135+
The part after the last `.end` is considered as a Ruby program.
136+
For example:
137+
```
138+
*--#[ Test3 :
139+
S x;
140+
G F = (1+x)^2;
141+
P;
142+
.store
143+
Save out.sav;
144+
.end
145+
Load out.sav;
146+
L G = F;
147+
P;
148+
.end
149+
assert succeeded?
150+
assert result("F") =~ expr("1 + 2*x + x^2")
151+
assert result("G") =~ expr("1 + 2*x + x^2")
152+
*--#] Test3 :
153+
```
154+
155+
Some test cases need to run only under specific conditions.
156+
In such cases, one can use special instructions starting with `#`.
157+
For example:
158+
```
159+
*--#[ Test4 :
160+
S x;
161+
L F =
162+
#pipe echo "(1+x)^2"
163+
;
164+
P;
165+
.end
166+
#require unix?
167+
assert succeeded?
168+
assert result("F") =~ expr("1 + 2*x + x^2")
169+
*--#] Test4 :
170+
```
171+
In this example, `#require unix?` ensures that the test runs
172+
only on Unix, where `#pipe` is expected to work.
173+
174+
### Available methods
175+
176+
#### Execution configuration
177+
178+
- `timeout → integer or float`
179+
Timeout duration in seconds.
180+
- `ncpu → integer`
181+
Number of assigned CPUs.
182+
- `total_memory → integer`
183+
Total physical memory available in bytes.
184+
- `serial? → bool`
185+
`true` if FORM is the serial version, otherwise `false`.
186+
- `threaded? → bool`
187+
`true` if FORM is the multithreaded version (TFORM), otherwise `false`.
188+
- `mpi? → bool`
189+
`true` if FORM is the MPI version (ParFORM), otherwise `false`.
190+
- `valgrind? → bool`
191+
`true` if FORM is running under Valgrind, otherwise `false`.
192+
- `wordsize → integer`
193+
Word size in bytes used by FORM (`4` on 64-bit systems).
194+
- `cygwin? → bool`
195+
`true` if running on Cygwin, otherwise `false`.
196+
- `mac? → bool`
197+
`true` if running on macOS, otherwise `false`.
198+
- `linux? → bool`
199+
`true` if running on Linux, otherwise `false`.
200+
- `unix? → bool`
201+
`true` if running on Unix, otherwise `false`.
202+
- `windows? → bool`
203+
`true` if running on Windows, otherwise `false`.
204+
- `travis? → bool`
205+
`true` if running on Travis CI, otherwise `false`.
206+
- `github? → bool`
207+
`true` if running on GitHub Actions, otherwise `false`.
208+
209+
#### Job status
210+
211+
- `return_value → integer`
212+
Exit status of the FORM job.
213+
- `finished? → bool`
214+
`true` if the FORM job finished within the timeout, otherwise `false`.
215+
- `succeeded? → bool`
216+
`true` if the FORM job finished without any problems, otherwise `false`.
217+
- `warning? → bool`
218+
`true` if the FORM job issued a warning, otherwise `false`.
219+
- `warning?(expected_message : string) → bool`
220+
`true` if the FORM job issued the expected warning, otherwise `false`.
221+
222+
The following methods are similar to `warning?`,
223+
but they check for preprocessor errors, compile-time errors,
224+
and run-time errors, respectively:
225+
226+
- `preprocess_error? → bool`
227+
`preprocess_error?(expected_message : string) → bool`
228+
- `compile_error? → bool`
229+
`compile_error?(expected_message : string) → bool`
230+
- `runtime_error? → bool`
231+
`runtime_error?(expected_message : string) → bool`
232+
233+
#### Standard streams
234+
235+
- `stdout → string`
236+
Standard output of the FORM job.
237+
- `stderr → string`
238+
Standard error of the FORM job.
239+
240+
#### Expressions
241+
242+
The following methods assume the default format for printing expressions:
243+
244+
- `result(expr_name : string) → string`
245+
The last printed output of the specified expression.
246+
- `result(expr_name : string, index : integer) → string`
247+
The printed output of the specified expression at the given index (zero-based).
248+
- `exact_result(expr_name : string) → string`
249+
`exact_result(expr_name : string, index : integer) → string`
250+
Similar to `result`, but returns the exact output, preserving line breaks and whitespaces.
251+
252+
The following methods assume the default format for statistics:
253+
254+
- `nterms(expr_name : string) → integer`
255+
`nterms(expr_name : string, index : integer) → integer`
256+
The number of terms as reported in the statistics for the specified expression.
257+
- `bytesize(expr_name : string) → integer`
258+
`bytesize(expr_name : string, index : integer) → integer`
259+
The size in bytes as reported in the statistics for the specified expression.
260+
261+
#### Helper methods
262+
263+
- `exact_pattern(str : string) → regexp`
264+
Regular expression constructed from the given text with escaping any special characters.
265+
- `pattern(str : string) → regexp`
266+
Similar to `exact_pattern`, but ignores whitespaces.
267+
- `expr(str : string) → regexp`
268+
Similar to `pattern`, but matches only with the whole expression.
269+
- `file(filename : string) → string`
270+
`read(filename : string) → string`
271+
Text in the specified file.
272+
- `write(filename : string, text : string) → nil`
273+
Writes a text into the specified file.
274+
275+
### Available instructions
127276

128-
- To verify that FORM finishes with a certain error, one can use
129-
`assert compile_error?` or `assert runtime_error?`.
130-
- Two or more FORM programs, separated by `.end`, can be put in a test case.
131-
Then the part after the last `.end` is for Ruby.
132-
- To skip a test case for some condition, one can specify it by `#pend_if`.
133-
(See the result of grepping `pend_if` in the existing files.)
134-
- When a test case requires other text files, one can use `#prepare write`.
135-
(See the result of grepping `prepare` in the existing files.)
277+
- `#require <condition>`
278+
Ensures that the test is executed only if the specified `<condition>` is met.
279+
- `#pend_if <condition>`
280+
Marks the test as pending if the specified `<condition>` is met.
281+
- `#prepare <statement>`
282+
Executes the given `<statement>` before running the test.
283+
For example:
284+
```
285+
#prepare write "foo.prc", "#procedure foo\n#message foo\n#endprocedure"
286+
```
287+
- `#ulimit <limits>`
288+
Sets the resource limits. This is done via the `ulimit` command.
289+
For example:
290+
```
291+
#require linux?
292+
#ulimit -v 8_000_000
293+
```
294+
This sets the maximum amount of virtual memory available to
295+
8,000,000 KiB (~ 8GB).
296+
- `#time_dilation <dilation>`
297+
Multiplies the timeout by the specified `<dilation>` factor.
298+
For example:
299+
```
300+
#time_dilation 2.0
301+
```

0 commit comments

Comments
 (0)