[syntax-errors] Remaining syntax errors raised by the compiler

[ntBre]

Summary

A number of semantic or compile-time syntax errors have already been implemented
in #11934, but there are still many remaining ones. The purpose of this issue is
to track progress on this remainder. The initial source for most of these is
this comment, which reports fuzzer results comparing Python's compiler and
ruff, but others may be included over time.

The next section lists the syntax errors that still need to be implemented, and
the sections after that include some more information on getting started with an
implementation.

Syntax Errors

Errors mapping to current rules

These might be the easiest to start with because the existing rules have working
implementations and corresponding tests. However, they could possibly require
new context methods (see below), which could complicate things.

• return with value in async generator (B901)
  • [syntax-errors]: semantic syntax error for async generator with return #21200
  • partial overlap with B901, which also applies to sync generators. This is
    only a syntax error in the async case
• no binding for nonlocal (PLE0117)
  • [syntax-error]: no binding for nonlocal PLE0117 as a semantic syntax error #21032
• break outside loop (F701)
  • [syntax-errors] Reimplement break-outside-loop (F701) as a semantic syntax error #20556
  • [ty] continue and break statements outside loops are syntax errors #20944
• continue not properly in loop (F702)
  • [syntax-errors]: implement F702 as semantic syntax error #20869
  • [ty] continue and break statements outside loops are syntax errors #20944
• yield from in async function (PLE1700)
  • [syntax-errors] Detect yield from inside async function #20051
• future feature is not defined (F407)
  • [syntax-errors]: future-feature-not-defined (F407) #20554
• import * only allowed at module level (F406)
  • [syntax-errors]: import from * only allowed at module scope (F406) #20166
• multiple starred expressions in assignment (F622)
  • [syntax-errors]: multiple-starred-expressions (F622) #20243

Errors mapping to current parse errors

These are also straightforward and will be similar to #17131.

• duplicate keyword argument

  def foo(x): ...
  
  foo(x=1, x=2)  # SyntaxError: keyword argument repeated: x

New errors

• name is parameter and global

  def f(a): global a

• name is parameter and nonlocal

  def f(a): nonlocal a

• annotated name can't be global

  x: int = 1
  
  def f():
      global x
      x: str = "foo"  # SyntaxError: annotated name 'x' can't be global

• alternative patterns bind different names ([syntax-errors] Alternative match patterns bind different names #20682)

  match 42:
      case [x] | [y]: ...

  This can probably be handled in the MatchPatternVisitor along with the other match-related errors.

• can't use starred expression here ([syntax-errors] detect single starred expression assignment x = *y #17624)

  x = *value

• nonlocal declaration not allowed at module level ([syntax-errors] nonlocal declaration at module level #17559)

  nonlocal x

Extensions to existing syntax errors

• Assigning to and deleting attributes named __debug__ is also a syntax error

  __debug__ = False  # Currently caught by ruff
  x.__debug__ = False  # Not currently caught, but also a syntax error

Encoding issues

These syntax errors are all related to #6791 and should only be implemented if
we decide to respect these alternate encodings in general.

These can also be a bit tricky to put into a file, so most of these have shell
code snippets for generating the input rather than Python code directly.

• ascii codec can't decode byte ... in position ...: ordinal not in range(128)

  printf '# -*- coding: ascii -*-\n\xc2' > example.py

• charmap codec can't decode byte ...

  printf '# -*- coding: cp1252 -*-\n\x8d' > example.py

• utf7 codec can't decode byte 0xc3

  printf '# -*- coding: utf-7 -*-\n\xc3' > example.py

  This one reports an E902 error, but only if it's enabled. Otherwise we only print a warning about invalid UTF-8.

• encoding problem: ... with BOM

  printf '\ufeff# -*- coding: ascii -*-' > example.py

  I think any encoding other than UTF-8 with a BOM (\uFEFF) will cause this issue.

• invalid character

• invalid non-printable character

  • both this and invalid character mostly seem to happen with the
    iso-8859-1 encoding and a unicode character

• unknown encoding

  # coding: not-a-real-encoding

• source code string cannot contain null bytes

  printf '# \x00' > example.py

  This mostly seems like an issue in comments. This also isn't exactly an encoding error, but it's more closely related to these. It's also raised by the CPython parser, not the compiler.

Implementation

Issue #11934 contains links to each PR implementing a new semantic syntax error.
These can be used as examples of what a new implementation looks like, but this
section has a few more general notes.

All of the semantic syntax errors currently live in semantic_errors.rs and
correspond to a variant of the SemanticSyntaxErrorKind enum. The
SemanticSyntaxChecker is responsible for emitting these errors, but unlike
other AST visitors, it does not generally traverse the AST on its owns¹. Instead,
it relies on a parent visitor to call its visit_stmt and visit_expr methods
in the parent visitor's visit_stmt or visit_expr methods.

The SemanticSyntaxChecker also tracks very little of its own state. Instead,
it again defers to the parent visitor via the SemanticSyntaxContext trait.
Examples of methods on this trait include:

• python_version -- returns the target Python version for linting
• in_async_context -- returns whether or not async code like await or async for loops are allowed in the current scope

Typically the parent visitor, like the Checker from the ruff_linter crate,
will implement SemanticSyntaxContext and pass itself to any
SemanticSyntaxChecker method requiring a context.

Another important context method is report_semantic_error, which may require
special handling of new SemanticSyntaxErrorKind variants. For example, some
syntax errors emitted by CPython overlap with existing ruff rules (e.g.
await-outside-async
(PLE1142)). These need
to be transformed into normal ruff Diagnostics instead of being reported
directly as syntax errors. This happens within the
Checker::report_semantic_error method.

Testing

There are two main ways of testing new semantic errors. The first, and easier,
method is to write inline parser tests using test_ok and test_err comments,
for example:

ruff/crates/ruff_python_parser/src/semantic_errors.rs

Lines 80 to 95 in 014bb52

	Stmt::Assign(ast::StmtAssign { targets, .. }) => {
	if let [Expr::Starred(ast::ExprStarred { range, .. })] = targets.as_slice() {
	// test_ok single_starred_assignment_target
	// (*a,) = (1,)
	// *a, = (1,)
	// [*a] = (1,)
	// test_err single_starred_assignment_target
	// *a = (1,)
	Self::add_error(
	ctx,
	SemanticSyntaxErrorKind::SingleStarredAssignment,
	*range,
	);
	}
	}

These are automatically extracted from the source code by the parser's
generate_inline_tests function and then run like normal snapshot tests.
These work very well for simple errors that don't require detailed information
from the SemanticSyntaxContext.

For errors that do require such contextual information tracked by the parent
visitor, a better alternative is to use integration tests, such as those found
in linter.rs in the ruff_linter crate:

ruff/crates/ruff_linter/src/linter.rs

Lines 1015 to 1025 in 014bb52

	#[test_case(
	"deferred_function_body",
	"
	async def f(): [x for x in foo()] and [x async for x in foo()]
	async def f():
	def g(): ...
	[x async for x in foo()]
	",
	PythonVersion::PY310
	)]
	fn test_async_comprehension_in_sync_comprehension(

ruff/crates/ruff_linter/src/linter.rs

Lines 1064 to 1066 in 014bb52

	#[test_case(Rule::YieldOutsideFunction, Path::new("yield_scope.py"))]
	#[test_case(Rule::ReturnOutsideFunction, Path::new("return_outside_function.py"))]
	fn test_syntax_errors(rule: Rule, path: &Path) -> Result<()> {

The second of these can be reused for other semantic errors that correspond to
ruff rules, while something like the first example can be used for new
syntax-error-specific tests. In both cases, these integration tests will take
advantage of the existing context tracking done by the real Checker instead
of trying to duplicate that tracking in the parser's test
SemanticSyntaxCheckerVisitor.

Comparing to the fuzzer results

Again, the initial errors in this issue were extracted from this comment in
#7633 by following this procedure:

1. Download and unzip the fuzzer results from here
2. Run the Python script below to get a partially-deduplicated list of errors
3. Double check these errors against ruff and CPython's parser and compiler

Python script for processing fuzzer results

import argparse
import compileall
import contextlib
import dataclasses
import io
import os
import re
import subprocess
from collections import Counter

FILE = re.compile(r'File "([^"]+)", line (\d+)')
MESSAGE = re.compile(r"SyntaxError: (.*)")


@dataclasses.dataclass
class Error:
    file: str
    line_number: int
    message: str


# Ruff rules already re-implemented as semantic syntax errors, not just rules
# that match CPython syntax errors.
RULES = [
    "F404",
    "F704",
    "F706",
    "PLE0118",
    "PLE1142",
]


def run_ruff(ruff_bin, path):
    return subprocess.run(
        [
            ruff_bin,
            "check",
            "--no-cache",
            "--silent",
            "--preview",
            "--config",
            f"lint.select={RULES}",
            path,
        ]
    ).returncode


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--ruff-bin", default="ruff")

    args = parser.parse_args()

    with (
        contextlib.redirect_stdout(io.StringIO()) as out,
        contextlib.redirect_stderr(os.devnull),
    ):
        compileall.compile_dir(".", force=True, quiet=1)

    seen_messages = Counter()
    errors = list()

    file, line_number = None, None
    for line in out.getvalue().splitlines():
        if m := FILE.search(line):
            file, line_number = m.groups()
        if m := MESSAGE.search(line):
            message = m[1]
            # combine repetitive messages into single entries
            for prefix in [
                "unknown encoding",
                "invalid escape sequence",
                "'ascii' codec can't decode",
                "future feature",
                "invalid character",
                "invalid non-printable character",
                "no binding for nonlocal",
            ]:
                if message.startswith(prefix):
                    message = prefix
                    break
            if (
                message not in seen_messages
                and run_ruff(args.ruff_bin, file) == 0
            ):
                errors.append(Error(file, line_number, message))

            seen_messages[message] += 1

    for error in sorted(errors, key=lambda e: e.message):
        count = seen_messages[error.message]
        print(f"{error.file}:{error.line_number}: {error.message} ({count})")


if __name__ == "__main__":
    main()

Footnotes

1. There are some local violations of this principle. For example, the
   ReboundComprehensionVisitor recursively visits the expressions in a
   comprehension looking for any expressions rebinding one of the
   comprehension's iteration variables. However, this is a highly-localized AST
   traversal and won't have the same kind of performance implications as an
   additional full AST traversal. ↩

[abhijeetbodas2001]

generator expression must be parenthesized

Regardless of whether there is a trailing comma or not, the exact same AST is generated by ruff for the code (only the ranges are different).
Given this, how do we detect the trailing comma during the semantic syntax error phase?
Should this rather be detected during parsing?
For example, the ast module does not give an AST at all for this:

>>> import ast; ast.parse("max(i for i in range(3),)")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<unknown>", line 1
    max(i for i in range(3),)
        ^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized