grammars

The grammars of the PHPDoc language

The files in this directory describe the language phpstan/phpdoc-parser reads, in the PP3 format the phplrt compiler reads. They are the specification of the language, and they are what tests/PHPStan/Parser/FuzzyTest.php writes its corpus from.

File	What it holds
lexemes.pp3	Every token the language is read into
common.pp3	What the grammars share: names, brackets, line breaks
types.pp3	The type language of TypeParser
const-expr.pp3	The constant expressions of ConstExprParser
phpdoc-block.pp3	The PHPDoc itself: its tags, its text, its Doctrine annotations
type.pp3	The entry point starting at Type
constant-expr.pp3	The entry point starting at ConstantExpr
phpdoc.pp3	The entry point starting at PhpDoc

What they are for

A grammar says what a PHPDoc may be written as, so it can be walked the other way round and asked for PHPDocs instead of being asked about one:

make grammars-install                  # the toolchain, which asks for PHP 8.4
php vendor/bin/phpunit --filter FuzzyTest

FuzzyTest runs tools/phplrt/fuzz.php itself, once per grammar, every time it runs: the tool compiles a grammar, walks its rules at random and writes down what comes out, and the test then asks the parser to read every one of them in full, and to read a type back as the very same type once it has been printed. There is no corpus to keep, and none to refresh when a rule changes — what is generated changes the moment the grammar is read again. The inputs of the last run are left in temp/fuzzy to be looked at.

That is a great deal more of the language than a hand-written corpus covers, and it is what replaced the abnfgen-driven fuzzer this project used before.

What keeps them in step

The fuzzer walks the grammars one way; tests/PHPStan/Parser/GrammarSyncTest.php walks them the other. It reads the inputs of every other test in this project — by reflection over the providers they are written with, so the corpus grows on its own as they do — and asks each grammar about them through tools/phplrt/recognize.php:

• everything the parser reads in full and without complaint, the grammar has to recognize;
• everything the parser stops at or turns into an Invalid... node, the grammar has to turn down.

So a feature added to the parser and written down as a test is one the grammars have to describe by the next run, and a rule written too widely is one the fuzzer walks into. Only reading is asked of the grammar there: nothing is built out of what it reads, so there is no second implementation of the AST to keep in step — only the language itself.

The few inputs the two are known to disagree about are listed in that test with the reason why, and the list has to be exactly right: an input that starts agreeing has to be taken off it.

Nothing in src/ reads these files, and the library needs neither the toolchain nor PHP 8.4: where either is missing, FuzzyTest skips itself. Only the Grammars job of .github/workflows/build.yml runs it for real.

How they are written

The rules are named after the methods of PhpDocParser, TypeParser and ConstExprParser they stand for and are written in the order those methods try things in, so that a grammar and the parser it describes can be read side by side.

What the grammars describe

A PHPDoc that is written correctly. The parser reads a broken one as well, by turning whatever it cannot read into an InvalidTagValueNode carrying the very error it has raised, and a grammar has no way of writing that error down. So what a broken PHPDoc means is left to the parser, and everything the grammars describe is something the parser has to read in full.

Two things follow from wanting that to hold for every input rather than for most of them:

• A place the parser raises an error at is written as something the grammar cannot recognize. Most of them are written as a ! predicate forbidding whatever the error would have been raised on. For instance a name followed by a < has to go on into a generic type or into a callable, because Foo< is an error rather than the type Foo followed by something else:

  IdentifierAtomic
    : ...
    | !ShapeBrace() Identifier() !<T_DOUBLE_COLON> ( IdentifierSuffix() | !<T_OPEN_ANGLE_BRACKET> )
    ;
  

  The same predicate is what keeps a rule from giving back what it has read. @template T of is an error rather than a template named T with the description of, so the bound is written as "either a bound or no of at all":

  TemplateUpperBound
    : <T_KEYWORD_OF> Type()
    | <T_KEYWORD_AS> Type()
    | !<T_KEYWORD_OF> !<T_KEYWORD_AS>
    ;
  

• A rule reads exactly the tokens its method reads, down to the line breaks around it.

The tokens are not read by phplrt

A grammar of this directory is not read by the lexer it declares: it is read by the very tokens PHPStan\PhpDocParser\Lexer\Lexer produces, handed over by tools/phplrt/Fuzzer/TokenStream.php.

The %token declarations therefore name the tokens and document the language without being what reads it. Some of them describe something the lexer never reads as a token of its own, and TokenStream is what tells those apart in the stream:

• a word the parser compares by value (is, array, covariant, static, …) — every one of them is still an ordinary name as well, which is why they are all listed among the alternatives of Identifier;
• a tag whose value a rule of its own reads (@param, @return, …), told apart from the tags nothing reads the value of;
• a bracket or an asterisk whose neighbouring whitespace decides what it means, which is what tells array{a: int} from the type array followed by a brace, and Foo[0] from Foo [0];
• a tag a space is written before, which is what tells the @since of @author Foo @since 1.0 from the @baz of @author Foo <foo@baz.com>;
• a < opening what the parser recognizes as an HTML tag, so that @return Foo<br>see below</br> keeps meaning the type Foo followed by a description.

Telling them apart there is what lets the grammars be written without semantic predicates, which the PP3 format has none of.

What is left out

Three corners of the language are left out on purpose, because a grammar cannot say what the parser does there. Each of them is written up where the rule that skirts it is written:

• a description that ends at a tag written in the middle of a line, which the parser decides by reading the tag and looking at what it turns out to be;
• the same, on a line after the first, where the parser reads that line twice: once as part of the description and again as whatever comes next;
• a tag whose value a rule reads, written with a parenthesis after it, where whether the description ends there depends on whether that value can be read at all.