A bit confused" Does the order in which lexer tokens are defined in the g4 file, matter?

[Korporal]

I've made excellent progress using Antlr4 for designing my new language grammar, but I'm still not fully familiar with the many facets of it.

Early on I had parsing failing and "moved some tokens around" in the g4 file and the parsing worked again, this was weeks ago and I can't recall the details, I might have made other changes as well.

So does the physical ordering of Lexer definitions matter in any way or is it totally irrelevant?

[ericvergnaud]

It matters: it defines token precedence.

[kaby76]

Yes, order matters, but only so much.

The rule is:

• The lexer token that matches the "longest string wins" above all else, regardless of the order of the lexer rules.
• If two or more tokens match the same string, then the "first rule winds".

[ericvergnaud]

that's why 'generic' tokens (such as ID) always need to be placed after 'specific' ones.
Consider the following:

IF: 'if';
ID: [a-z]+;

the input 'if' will match IF token

now reverse the tokens:

ID: [a-z]+;
IF: 'if';

the same input will match ID token.

So I'd say it matters a lot, since ID is such a common token...

[KvanTTT]

For good, in the second case a warning should be reported because IF token is unreachable. We have issues on that, for instance #1072

[Korporal]

Thanks folks. I get this but these work the same way irrespective of order, yes?

STAR_1:   '*';
STAR_2:   '**';

The lexer will match ** correctly, no matter how the rules are ordered, yes?

[kaby76]

Yes. Input ** is matched as one token **, not two (* and *) even though STAR_1 is first. Antlr matches the longest token first regardless of lexer rule order.