-
Notifications
You must be signed in to change notification settings - Fork 10
Extending qspec
qspec is not just a way to run unit tests on the command line. While the test runner is a simple starting point for test automation, users may have more advanced needs and may want alternative ways of running their tests and interacting with the results.
The principle way that you will want to extend qspec is by using the test objects that are created by the @.tst.desc@ functions and are run with the .tst.runSpec specification runner function. We’ll go through these objects (principally dictionaries) and a few of the functions (like .tst.runSpec) that can be used with them.
Specification objects contain 5 fields: result, title, context, tstPath, and expectations
result is a field that contains symbols. The possible values for the result field are `didNotRun, `pass, and `fail
title contains the description of the specification. This will be a character list.
context is the namespace that the test was defined in. For example if \d . was placed before a specification was defined, the context would be `., but if \d .foo was used instead, the context would be `.foo
tstPath is the path to the file that the specification was loaded from. This is important for determining where to look for fixtues, for example
expectations is the list of expectation objects within the specification object
Expectation objects have a variable number of fields depending on their type. They do share several fields though:
type This indicates the type of the expectation, `test and `fuzz are currently the only valid values, representing the standard expectation and a fuzz expectation, currently.
result This will contain a symbol indicating whether the expectation passed, failed, errored or wasn’t run. It’s possible values are `pass, `didNotRun, `beforeError, `afterError, `testFail, `fuzzFail, `testError, and `fuzzError. Of course, a fuzz expectation couldn’t have a `testError or `testFail and a standard expectation couldn’t have a `fuzzError or `fuzzFail for this value.
errorText If this field has been set, it will contain the text of the error caught by the test runner while trying to run the expectation as a character list. If there was no error, it will be an empty list.
desc This contains a textual description of the expectation
code contains the function body to be run
before contains the before function to run before running the function in code
after contains the after function to run after running the function in code
failures contains a list of failure messages produced by the assertions. Each failure message will be a character list. If there’s only one failure message, that means that it will be an enlisted character list, if there are two, then it will be a list of two character lists, and so on.
assertsRun contains an integer recording how many assertions were run in the test. This is perhaps not so useful for fuzz tests.
Fuzz tests have a few additional properties, most of these are the possible values that can be provided to the second argument to @holds@, please refer to the documentation for @holds@ for these properties.
Additionally, there is the failRate property.
failRate contains the observed ratio of failures to successes seen when running the fuzz expectation. This value is compared against maxFailRate to determine whether or not result should be set to `pass or `fuzzFail
qspec provides the all important .tst.runSpec function for running tests. There are also some output functions that were created for the test runner that could conceivably be repurposed and callbacks (please see the Callbacks page for their descriptions).
.tst.runSpec takes a specification object and runs it, filling in results and other fields. It also manages Directory Fixture state as that is handled separately than the state of other variables set using the @mock@ keyword.
.tst.loadOutputModule is a special function intended to allow various modes of test output to be distributed with the qspec framework and allow for a somewhat standard way of making output code work. It takes one argument, a character list indicating the “output module” to load. The only available output module currently is “text” which is used with the test runner.
The idea behind the “output modules” is that all functions that produce output based on specification or expectation object input should go into a dictionary .tst.output so that anyone who would like to define their own “output module” could safely package it within the .tst namespace without clobbering any internals. By using .tst.loadOutputModule it would be clear which type of output would be used since only one dictionary of output functions would occupy .tst.output at a time.
If you would like to develop your own module, place it in “lib/output” and name it how you would like .tst.loadOutputModule to load it.
This is the function that the test runner uses to print information about the test results to the console. Given a specification object that has passed, it will not print anything. If the specification object failed it will print information about all expectations that failed or errored. It will also check a variable .tst.output.mode to see if it is set to `run or `describe. If it is set to `describe, it will only print the descriptions from the specifications and expectations (a specification that has not run will have a result of `didNotRun, not `pass).