You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: conformance/README.md
+58-28Lines changed: 58 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,34 +2,35 @@
2
2
3
3
## Motivation
4
4
5
-
[PEP 729](https://peps.python.org/pep-0729/) provides a structured and documented way to specify and evolve the Python type system. In support of this effort, an official [Python typing spec](https://github.com/python/typing/tree/main/docs/spec) has been drafted. This spec consolidates details from various historical typing-related PEPs. The spec will be modified over time to clarify unspecified and under-specified parts of the type system. It will also be extended to cover new features of the type system.
5
+
[PEP 729](https://peps.python.org/pep-0729/) provides a structured and documented way to specify and evolve the Python type system. In support of this effort, an official [Python typing spec](https://typing.python.org/en/latest/spec/) has been drafted. This spec consolidates details from various historical typing-related PEPs. The spec will be modified over time to clarify unspecified and under-specified parts of the type system. It will also be extended to cover new features of the type system.
6
6
7
7
Accompanying the typing specification is this conformance test suite which validates the behavior of static type checkers against the specification.
8
8
9
9
## Structure & Name
10
10
11
11
This project contains test cases for behaviors defined in the Python typing spec. Tests are structured and grouped in accordance with the specification's chapter headings.
A test file is a ".py" file. The file name should start with one of the above names followed by a description of the test (with words separated by underscores). For example, `generics_paramspec_basic_usage.py` would contain the basic usage tests for `ParamSpec`. Each test file can contain multiple individual unit tests, but these tests should be related to each other. If the number of unit tests in a single test file exceeds ten, it may be desirable to split it into separate test files. This will help maintain a consistent level of granularity across tests.
35
36
@@ -46,27 +47,41 @@ Test cases use the following conventions:
46
47
* Lines that are expected to produce a type checker error should have a comment starting with # E",
47
48
either by itself or followed by an explanation after a colon (e.g., "# E: int is not a subtype
48
49
of str"). Such explanatory comments are purely for human understanding, but type checkers are not
49
-
expected to use their exact wording.
50
+
expected to use their exact wording. There are several syntactic variations; see "Test Case Syntax"
51
+
below.
50
52
* Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
51
53
marked with "# E?" instead of "# E".
52
54
* If a test case tests conformance with a specific passage in the spec, that passage should be
53
55
quoted in a comment prefixed with "# > ".
54
56
57
+
## Test Case Syntax
58
+
59
+
Test cases support the following special comments for declaring where errors should be raised:
60
+
61
+
*`# E`: an error must be raised on this line
62
+
*`# E?`: an error may be raised on this line
63
+
*`# E[tag]`, where `tag` is an arbitrary string: must appear multiple times in a file with the same tag.
64
+
Exactly one line with this tag must raise an error.
65
+
*`# E[tag+]`: like `# E[tag]`, but errors may be raised on multiple lines.
66
+
67
+
Each comment may be followed by a colon plus an explanation of the error; the explanation is ignored
68
+
by the scoring system.
69
+
55
70
## Running the Conformance Test Tool
56
71
57
72
To run the conformance test suite:
58
73
* Clone the https://github.com/python/typing repo.
59
-
*Create and activate a Python 3.12 virtual environment.
60
-
* Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
61
-
*Switch to the `src` subdirectory and run `python main.py`.
74
+
*Install [uv](https://docs.astral.sh/uv/)and ensure Python 3.12 is available.
75
+
* Switch to the `conformance` subdirectory and install locked dependencies (`uv sync --python 3.12 --frozen`).
76
+
*Run the conformance tool (`uv run --python 3.12 --frozen python src/main.py`).
62
77
63
-
Note that some type checkers may not run on some platforms. For example, pytype cannot be installed on Windows. If a type checker fails to install, tests will be skipped for that type checker.
78
+
Note that some type checkers may not run on some platforms. If a type checker fails to install, tests will be skipped for that type checker.
64
79
65
80
## Reporting Conformance Results
66
81
67
82
Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
68
83
69
-
Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyre, pyright, and pytype. It is the goal and desire to add additional type checkers over time.
84
+
[Conformance results](https://htmlpreview.github.io/?https://github.com/python/typing/blob/main/conformance/results/results.html) are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, zuban, ty, and pycroscope. It is the goal and desire to add additional type checkers over time.
70
85
71
86
## Adding a New Test Case
72
87
@@ -78,7 +93,22 @@ If a test is updated (augmented or fixed), the process is similar to when adding
78
93
79
94
## Updating a Type Checker
80
95
81
-
If a new version of a type checker is released, re-run the test tool with the new version. If the type checker output has changed for any test cases, the tool will supply the old and new outputs. Examine these to determine whether the conformance status has changed. Once the conformance status has been updated, re-run the test tool again to regenerate the summary report.
96
+
Type checker versions are locked in `uv.lock`.
97
+
98
+
To bump all supported checkers to their latest released versions:
99
+
100
+
```bash
101
+
python scripts/bump_type_checkers.py
102
+
```
103
+
104
+
After bumping, install the new lockfile and rerun conformance:
105
+
106
+
```bash
107
+
uv sync --python 3.12 --frozen
108
+
uv run --python 3.12 --frozen python src/main.py
109
+
```
110
+
111
+
If checker output changes for any test cases, examine those deltas to determine whether the conformance status has changed. Once the conformance status has been updated, rerun the tool to regenerate the summary report.
0 commit comments