1
- FORM Test Suite
2
- ===============
1
+ # FORM Test Suite
3
2
4
3
This directory contains a collection of test cases that can be used for
5
4
verifying the behaviour of FORM. It also has a script to run the test cases and
6
5
check the results.
7
6
8
- Prerequisites
9
- -------------
7
+ ## Prerequisites
10
8
11
9
The test runner script is written in [ Ruby] ( https://www.ruby-lang.org/ )
12
- and requires Ruby 1.9 or later. The script uses the so-called ` test/unit `
10
+ and requires Ruby 2.0 or later. The script uses the so-called ` test/unit `
13
11
library. In some Linux distributions the library is installed together with
14
12
Ruby, while some distributions may have the library as an optional package,
15
13
or one may need to manually install
16
14
[ test-unit] ( http://test-unit.github.io/test-unit/en/ ) via the ` gem ` command.
17
- Currently, the script runs only on Unix-like systems.
18
15
19
- Usage
20
- -----
16
+ ## Usage
21
17
22
18
### From the build system
23
19
24
20
To use the test suite from the automatic build system
25
21
(see also the [ INSTALL] ( ../INSTALL ) file),
26
- run
22
+ run the following command:
27
23
28
- ```
24
+ ``` bash
29
25
# in the root build directory
30
26
make check
31
27
```
32
28
33
29
which tests the executables (release versions) compiled by the build system.
34
30
35
- ### Testing in the standalone mode
31
+ ### Testing in standalone mode
36
32
37
33
Alternatively, one can run the test runner script directly:
38
34
39
- ```
35
+ ``` bash
40
36
# in the "check" directory
41
37
./check.rb
42
38
```
43
39
44
- By default, it tests ` form ` found in $PATH.
45
- To check another executable, give the path as a command line option :
40
+ By default, this tests ` form ` found in ` $PATH ` .
41
+ To test another executable, specify its path as a command- line argument :
46
42
47
- ```
43
+ ``` bash
48
44
./check.rb /path/to/form
49
45
```
50
46
51
- One can specify a TFORM (or ParFORM) executable in this way.
47
+ One can also specify a TFORM (or ParFORM) executable in this way.
52
48
TFORM and ParFORM will be run with 4 CPUs (can be changed by the ` --cpu N `
53
49
option).
54
50
55
51
By default, all test cases in all FORM files (` *.frm ` ) found in the ` check `
56
52
directory (not in subdirectories) are used. To select test cases or FORM files
57
- to be run, give their names as command line options, for example,
53
+ to be run, specify their names as command- line arguments. For example:
58
54
59
- ```
60
- ./check.rb examples.frm
55
+ ``` bash
61
56
./check.rb Issue8
57
+ ./check.rb ' divmod_*'
58
+ ./check.rb examples.frm
62
59
```
63
60
64
- For more advanced options, see the help message shown by the ` --help ` option.
61
+ For more advanced options, refer to the help message using the ` --help ` option.
65
62
66
- Writing tests
67
- -------------
63
+ ## Writing tests
68
64
69
65
### Where to add test cases?
70
66
71
- Currently, the standard test set (run by default) consists of 3 files:
67
+ Currently, the standard test set (run by default) consists of 4 files:
72
68
73
- - ` examples.frm ` : Examples found in the manual.
69
+ - ` examples.frm ` : Examples provided in the manual.
74
70
- ` features.frm ` : Test cases for newly added features.
75
71
- ` fixes.frm ` : Test cases for bug fixes.
72
+ - ` user.frm ` : Test cases contributed by users.
76
73
77
74
Each test case in these files should finish in a short time: the timeout is set
78
75
to 10 seconds. Bigger tests that take more time are put in subdirectories
79
- (e.g., forcer) and should be specified by command-line options when the test
80
- suite is invoked.
76
+ (e.g., ` forcer ` ) and should be specified by command-line options when the test
77
+ suite is invoked:
78
+
79
+ ``` bash
80
+ ./check.rb -C forcer # The Forcer library must be available in FORMPATH.
81
+ ```
81
82
82
83
### Structure of a test case
83
84
84
- A test case is given as a fold in a FORM file. A simple example is:
85
+ A test case is given as a fold in a FORM file.
86
+ The following is a simple example:
85
87
86
88
```
87
89
*--#[ Test1 :
@@ -95,16 +97,22 @@ assert result("F") =~ expr("1 + 2*x + x^2")
95
97
```
96
98
97
99
The fold name ` Test1 ` gives the name of the test case, which should be unique.
98
- The part before ` .end ` is a normal FORM program. After ` .end ` , one can write
99
- a Ruby program to check the results. In this example, ` assert ` method (which is
100
- provided by some unit test class) is used for checking whether its argument is
101
- ` true ` . The first assertion checks ` succeeded? ` , which gives ` true ` if the FORM
102
- successfully finishes. The second assertion checks the printed result of the
103
- expression ` F ` by a regular expression matching (` =~ ` ). In the left-hand side,
104
- ` result("F") ` returns the (lastly) printed output for the expression ` F ` as
105
- a string. In the right-hand side, ` expr("...") ` makes a regular expression with
106
- removing white spaces in its argument. Since ` expr() ` removes all white spaces,
107
- one can also put new lines, for example,
100
+ The part before ` .end ` is a normal FORM program.
101
+ After ` .end ` , one can write a Ruby program to check the results.
102
+
103
+ The ` assert ` method checks whether its argument evaluates to ` true ` .
104
+ In this example:
105
+ - The first assertion verifies ` succeeded? ` , which returns ` true ` if the FORM finishes successfully.
106
+ - The second assertion checks the printed result of the
107
+ expression ` F ` by a regular expression matching (` =~ ` ).
108
+ - On the left-hand side, ` result("F") ` returns the (lastly) printed output
109
+ for the expression ` F ` as a string.
110
+ - On the right-hand side, ` expr("...") ` creates a regular expression
111
+ by removing white spaces in its argument.
112
+
113
+ Since ` expr() ` removes all white spaces,
114
+ one can include new lines in the argument.
115
+ For example:
108
116
109
117
```
110
118
*--#[ Test2 :
@@ -123,13 +131,171 @@ assert result("F") =~ expr("
123
131
124
132
which is convenient to copy and paste a long output from a terminal.
125
133
126
- ### Tips
134
+ Two or more FORM programs, separated by ` .end ` , can be put in a test case.
135
+ The part after the last ` .end ` is considered as a Ruby program.
136
+ For example:
137
+ ```
138
+ *--#[ Test3 :
139
+ S x;
140
+ G F = (1+x)^2;
141
+ P;
142
+ .store
143
+ Save out.sav;
144
+ .end
145
+ Load out.sav;
146
+ L G = F;
147
+ P;
148
+ .end
149
+ assert succeeded?
150
+ assert result("F") =~ expr("1 + 2*x + x^2")
151
+ assert result("G") =~ expr("1 + 2*x + x^2")
152
+ *--#] Test3 :
153
+ ```
154
+
155
+ Some test cases need to run only under specific conditions.
156
+ In such cases, one can use special instructions starting with ` # ` .
157
+ For example:
158
+ ```
159
+ *--#[ Test4 :
160
+ S x;
161
+ L F =
162
+ #pipe echo "(1+x)^2"
163
+ ;
164
+ P;
165
+ .end
166
+ #require unix?
167
+ assert succeeded?
168
+ assert result("F") =~ expr("1 + 2*x + x^2")
169
+ *--#] Test4 :
170
+ ```
171
+ In this example, ` #require unix? ` ensures that the test runs
172
+ only on Unix, where ` #pipe ` is expected to work.
173
+
174
+ ### Available methods
175
+
176
+ #### Execution configuration
177
+
178
+ - ` timeout → integer or float `
179
+ Timeout duration in seconds.
180
+ - ` ncpu → integer `
181
+ Number of assigned CPUs.
182
+ - ` total_memory → integer `
183
+ Total physical memory available in bytes.
184
+ - ` serial? → bool `
185
+ ` true ` if FORM is the serial version, otherwise ` false ` .
186
+ - ` threaded? → bool `
187
+ ` true ` if FORM is the multithreaded version (TFORM), otherwise ` false ` .
188
+ - ` mpi? → bool `
189
+ ` true ` if FORM is the MPI version (ParFORM), otherwise ` false ` .
190
+ - ` valgrind? → bool `
191
+ ` true ` if FORM is running under Valgrind, otherwise ` false ` .
192
+ - ` wordsize → integer `
193
+ Word size in bytes used by FORM (` 4 ` on 64-bit systems).
194
+ - ` cygwin? → bool `
195
+ ` true ` if running on Cygwin, otherwise ` false ` .
196
+ - ` mac? → bool `
197
+ ` true ` if running on macOS, otherwise ` false ` .
198
+ - ` linux? → bool `
199
+ ` true ` if running on Linux, otherwise ` false ` .
200
+ - ` unix? → bool `
201
+ ` true ` if running on Unix, otherwise ` false ` .
202
+ - ` windows? → bool `
203
+ ` true ` if running on Windows, otherwise ` false ` .
204
+ - ` travis? → bool `
205
+ ` true ` if running on Travis CI, otherwise ` false ` .
206
+ - ` github? → bool `
207
+ ` true ` if running on GitHub Actions, otherwise ` false ` .
208
+
209
+ #### Job status
210
+
211
+ - ` return_value → integer `
212
+ Exit status of the FORM job.
213
+ - ` finished? → bool `
214
+ ` true ` if the FORM job finished within the timeout, otherwise ` false ` .
215
+ - ` succeeded? → bool `
216
+ ` true ` if the FORM job finished without any problems, otherwise ` false ` .
217
+ - ` warning? → bool `
218
+ ` true ` if the FORM job issued a warning, otherwise ` false ` .
219
+ - ` warning?(expected_message : string) → bool `
220
+ ` true ` if the FORM job issued the expected warning, otherwise ` false ` .
221
+
222
+ The following methods are similar to ` warning? ` ,
223
+ but they check for preprocessor errors, compile-time errors,
224
+ and run-time errors, respectively:
225
+
226
+ - ` preprocess_error? → bool `
227
+ ` preprocess_error?(expected_message : string) → bool `
228
+ - ` compile_error? → bool `
229
+ ` compile_error?(expected_message : string) → bool `
230
+ - ` runtime_error? → bool `
231
+ ` runtime_error?(expected_message : string) → bool `
232
+
233
+ #### Standard streams
234
+
235
+ - ` stdout → string `
236
+ Standard output of the FORM job.
237
+ - ` stderr → string `
238
+ Standard error of the FORM job.
239
+
240
+ #### Expressions
241
+
242
+ The following methods assume the default format for printing expressions:
243
+
244
+ - ` result(expr_name : string) → string `
245
+ The last printed output of the specified expression.
246
+ - ` result(expr_name : string, index : integer) → string `
247
+ The printed output of the specified expression at the given index (zero-based).
248
+ - ` exact_result(expr_name : string) → string `
249
+ ` exact_result(expr_name : string, index : integer) → string `
250
+ Similar to ` result ` , but returns the exact output, preserving line breaks and whitespaces.
251
+
252
+ The following methods assume the default format for statistics:
253
+
254
+ - ` nterms(expr_name : string) → integer `
255
+ ` nterms(expr_name : string, index : integer) → integer `
256
+ The number of terms as reported in the statistics for the specified expression.
257
+ - ` bytesize(expr_name : string) → integer `
258
+ ` bytesize(expr_name : string, index : integer) → integer `
259
+ The size in bytes as reported in the statistics for the specified expression.
260
+
261
+ #### Helper methods
262
+
263
+ - ` exact_pattern(str : string) → regexp `
264
+ Regular expression constructed from the given text with escaping any special characters.
265
+ - ` pattern(str : string) → regexp `
266
+ Similar to ` exact_pattern ` , but ignores whitespaces.
267
+ - ` expr(str : string) → regexp `
268
+ Similar to ` pattern ` , but matches only with the whole expression.
269
+ - ` file(filename : string) → string `
270
+ ` read(filename : string) → string `
271
+ Text in the specified file.
272
+ - ` write(filename : string, text : string) → nil `
273
+ Writes a text into the specified file.
274
+
275
+ ### Available instructions
127
276
128
- - To verify that FORM finishes with a certain error, one can use
129
- ` assert compile_error? ` or ` assert runtime_error? ` .
130
- - Two or more FORM programs, separated by ` .end ` , can be put in a test case.
131
- Then the part after the last ` .end ` is for Ruby.
132
- - To skip a test case for some condition, one can specify it by ` #pend_if ` .
133
- (See the result of grepping ` pend_if ` in the existing files.)
134
- - When a test case requires other text files, one can use ` #prepare write ` .
135
- (See the result of grepping ` prepare ` in the existing files.)
277
+ - ` #require <condition> `
278
+ Ensures that the test is executed only if the specified ` <condition> ` is met.
279
+ - ` #pend_if <condition> `
280
+ Marks the test as pending if the specified ` <condition> ` is met.
281
+ - ` #prepare <statement> `
282
+ Executes the given ` <statement> ` before running the test.
283
+ For example:
284
+ ```
285
+ #prepare write "foo.prc", "#procedure foo\n#message foo\n#endprocedure"
286
+ ```
287
+ - ` #ulimit <limits> `
288
+ Sets the resource limits. This is done via the ` ulimit ` command.
289
+ For example:
290
+ ```
291
+ #require linux?
292
+ #ulimit -v 8_000_000
293
+ ```
294
+ This sets the maximum amount of virtual memory available to
295
+ 8,000,000 KiB (~ 8GB).
296
+ - ` #time_dilation <dilation> `
297
+ Multiplies the timeout by the specified ` <dilation> ` factor.
298
+ For example:
299
+ ```
300
+ #time_dilation 2.0
301
+ ```
0 commit comments