Polytest is configured via a TOML configuration file named polytest.toml by default. All Polytest configuration, including test suite definitions, is done via this file. This file is the one source of truth for Polytest, ensuring there is no confusion about the current state of the test plan.
The test plan is named via the name field. This is generally only used for display purposes, but can also be used during document generation.
name = "Shapes Test Plan"The package_name defines the name of the package being tested. This is primarily used in some of the templates for importing. For example, in Swift the template starts with a import {{ package_name | convert_case('Pascal')
Test suites are a collection of tests. For most implementations this roughly maps to the tests in a single file. Test cases are defined under the suite table.
The desc field within a suite table is used to describe the suite.
The groups field within a suite table is used to define the test groups that belong to the suite.
[suite.circle]
desc = "A circle is a shape defined by all points on a plane that are equidistant from a given point"
groups = ["shape", "circle"]
[suite.rectangle]
desc = "A rectangle is a polygon with four right angles"
groups = ["shape", "polygon", "rectangle"]
[suite.triangle]
desc = "A triangle is a polygon with three edges"
groups = ["shape", "polygon"]Groups are a collection of test cases that typically share something in common. You could also think of a group as a label on a test case, but there is a strict "one group" -> "many testcases" relationship.
The desc field within a group table is used to describe the group.
[group.polygon]
desc = "Tests that only apply to polygons"Test cases are defined under the test table within a group. These map to one test case that will have a pass or fail status when ran.
The desc field describes what is being tested.
[group.polygon.test.vertex_count]
desc = "A polygon should accurately count the number of verticies it contains"
[group.polygon.test.edge_count]
desc = "A polygon should accurately count the number of edges it contains"Test targets are defined under the target table. Test targets are the testting frameworks for the implementations (i.e. pytest for python).
These are the targets supported by Polytest out of the box. Custom targets can also be defined.
pytestbun
The out_dir field is used to define the output directory for the generated test scaffolding and is used as the working directory when the test command is ran.
[target.pytest]
out_dir = "./implementations/python/tests"
[target.bun]
out_dir = "./implementations/bun/__tests__"Custom test targets can be defined under the custom_target table. Custom targets give you full control of scaffolding templates, test execution, and parsing.
The out_dir field is used to define the output directory for the generated test scaffolding and is used as the working directory when the test command is ran. The path is relative to the location of the configuration file.
The suite_file_name_template field is used to define the template for the suite file name. This string is a minijinja template.
The variables available for use in the template. See templates documentation for more information on how these variables can be used.
suite- Suite struct
The test_regex_template field is used to define regex that Polytest can use to find test implementations in the suite file(s). This string is a minijinja template.
The variables available for use in the template. See templates documentation for more information on how these variables can be used.
name- The name of the test case (i.e. fortest.some_test,some_test)
The template_dir field is used to define the directory that contains the templates for the test target. This directory is used to find the templates for the test target. The path is relative to the location of the configuration file. This directory must contain one file that matches each of the following globs:
suite*group*test*
See templates documentation for more information on the expected contents of these files and available variables.
runner is a table that defines how to run the test suites and parse the results. There can be multiple runners defined for one target (for example, testing multiple platforms).
Each runner will inherit the fields of the previously-defined runner if they are not defined.
The command field is used to define the command to run the tests.
The work_dir field is used to define the working directory for the runner. If not defined, the out_dir of the target will be used.
The fail_regex_template field is used to define regex that Polytest can use on the output to determine if a test has failed. This string is a minijinja template.
The variables available for use in the template. See templates documentation for more information on how these variables can be used.
file_name- The name of the file that contains the test (this is the renderedsuite_file_name_template)suite_name- The name of the suite that contains the test (i.e. forsuite.some_suite,some_suite)group_name- The name of the group that contains the test (i.e. forgroup.some_group,some_group)test_name- The name of the test (i.e. fortest.some_test,some_test)
The pass_fail_regex_template field is used to define regex that Polytest can use on the output to determine if a test has passed. This string is a minijinja template.
The variables available for use in the template. See templates documentation for more information on how these variables can be used.
file_name- The name of the file that contains the test (this is the renderedsuite_file_name_template)suite_name- The name of the suite that contains the test (i.e. forsuite.some_suite,some_suite)group_name- The name of the group that contains the test (i.e. forgroup.some_group,some_group)test_name- The name of the test (i.e. fortest.some_test,some_test)
[custom_target.minitest_unit.runner."rake test"]
command = "bundle exec rake test A='--verbose'"
fail_regex_template = "Test{{ suite_name | convert_case('Pascal') }}#test_{{ test_name }} = \\d+\\.\\d+ s = (F|E)"
pass_regex_template = "Test{{ suite_name | convert_case('Pascal') }}#test_{{ test_name }} = \\d+\\.\\d+ s = \\."A document is a generated file that is regenered each time polytest generate is ran.
The path to the file that will be generated. The path is relative to the location of the configuration file.
If not given, use the default template for the given name. If given, it is the path to the template file. The path is relative to the location of the configuration file.
name- The name of the Polytest plansuites- A list of allSuitestructs in the test plangroups- A list of allGroupstructs in the test plantests- A list of allTeststructs in the test plan
[document.markdown]
out_file = "./documents/plan.md"
[document.test_cases_csv]
out_file = "./documents/test_cases.csv"
template = "./templates/test_cases.csv.jinja"# Polytest Test Plan
## Test Suites
{% for suite in suites %}
### {{ suite.name | convert_case('Title') }}
| Name | Description |
| --- | --- |
{% for group in suite.groups %}
| [{{ group.name }}](#{{ group.name | convert_case('Kebab') }}) | {{ group.desc }} |
{% endfor %}
{% endfor %}
## Test Groups
{% for group in groups %}
### {{ group.name | convert_case('Title') }}
| Name | Description |
| --- | --- |
{% for test in group.tests %}
| [{{ test.name }}](#{{ test.name | convert_case('Kebab') }}) | {{ test.desc }} |
{% endfor %}
{% endfor %}
## Test Cases
{% for test in tests %}
### {{ test.name }}
{{ test.desc }}
{% endfor %}