Simple parser but does not discern numbers

[mwpowellhtx]

Goal is support a simple Location DSL, nothing too fancy, but is a smidge more complex than I want to fuss with getting regex correct, or cruder forms of tokenization, split strings, etc. However, it does not seem to discern numbers quite right, if I understand the listener rule enter/exit edges correctly. Couple of example use cases, the interpretation around which sometimes is positional, sometimes not, depending whether an identifier assignment is present.

// in its most elaborate form:
X=123 Y=234 Z=345
// in its most bare form:
123 234 345
// can be in between, i.e.
X=123 234 345

The grammar, tentatively:

grammar ScumParser;

// Lexer rules

DIGIT: [0-9];
X: [xX];
Y: [yY];
Z: [zZ];

OP_EQ: '=';

DOT: '.';

// Carriage Return, Line Feed.
fragment CR: '\r';
fragment LF: '\n';
// Space.
fragment SP: ' ';
// Tab, or Horizontal Tab.
fragment HT: '\t';
// Form Feed, or Vertical Tab.
fragment VT: '\f';

// Whitespace
WS: ( SP | HT | VT )+ -> skip;

// NEWLINE : '\r'? '\n' { {System.out.println("Newlines so far: " + (++counter)); } };
NL: CR? LF -> skip;

// Grammar rules

coordinatesDecl: (coordinateDecl | WS coordinateDecl)* EOF;

// Either: X|Y|X = <decimal/>
// Or just: <decimal/>, coupled with positional semantics
coordinateDecl
    : IDENTIFIER (OP_EQ CoordinatePart)?
    | CoordinatePart
    ;

identifier: IdentifierPart;

fragment IdentifierPart: X | Y | Z;

fragment FractionalPart: '.' DIGIT*;
fragment CoordinatePart: (DIGIT+ FractionalPart?) | FractionalPart;

My test case to start with is:

"10000 10000\r\n"

Just to see what I might get. The parser appears to succeed, but if I am reading the context correctly, I do not find any values, and context is indicating something like "1 0 0 0 0 1 0 0 0 0", which at least it is scanning through the input correctly, so there's that. And with breakpoints set, definitely only hitting one set of enter exit coordinate edges there.

But at some level, my goal is to see two coordinate edges, enter and exit, indicating "1 0 0 0 0" in each case.

And forward from there.

What am I missing?

[mwpowellhtx]

Hmm okay so it is actually throwing Antlr4.Runtime.InputMismatchException on the first coordinateDecl grouping. In either a 10000 case, or X=10000 case, for instance.

[mwpowellhtx]

I thought perhaps the WS rule was gobbling up spaces and ignoring the separator, but that did not correct the InputMismatchException.

SEP: ' ' | '\t';
// ...
coordinatesDecl: (coordinateDecl | SEP coordinateDecl)* EOF;

[mwpowellhtx]

Ah 💡 I think I see, maybe it is working, and I just need to connect some dots in the listener now. In my unit tests I verify against the top level coordinates rule, instead of the inward coordinate rule.

parser.AddParseListener(listener);
IParseTree tree = parser.coordinatesDecl();

I also tweaked the grammar above, I think I see an issue there.

coordinatesDecl: coordinateDecl (SEP coordinateDecl)* EOF;

[mwpowellhtx]

Also I dare say, that was good thinking allowing subscribers the opportunity to extend a base class into the partial listener classes, for instance. Especially if some common baseline infrastructure is required extending into the listener override contexts, i.e.

// As furnished by the Antlr4 code generation
public partial class MyGrammarParserBaseListener : IMyGrammarParserListener
{
}

// ...

// Sliding infrastructure service base underneath the listener
public partial class MyGrammarParserBaseListener : MyGrammarServiceBase
//                   Service based infrastructure: ^^^^^^^^^^^^^^^^^^^^
{
    // ...
}

And extending into the actual listener with overrides...

public class MyGrammarParserListener : MyGrammarParserBaseListener
{
    public override void EnterMyRule([NotNull] MyGrammarParser.MyRuleContext context)
    {
        // ...
    }

    public override void ExitMyRule([NotNull] MyGrammarParser.MyRuleContext context)
    {
        // ...
    }
}

I do not know if that was by design, necessarily, or just a happy accident, but will take the win. 🍻

[mwpowellhtx]

Under some xunit testing, I have breakpoints on each of my rule enter/exit pairs, and I seemed to have it working, briefly, for a moment, rules were more or less reporting their enter/exit edges, but with this bit of adjustment, does not appear to be the case. I get a lines enter/exit pair, and that's it. Whatever might have been produced seems to be flattened to the lines rule, which does show my test case data, "10000 10000\r\n" in context, and that's it. I do not see any line, location, etc, enter/exit pairs, whatsoever. Wondering, maybe I have too much lexer going on here, or not enough ruleage, not hundred percent sure.

Definitely in the unit test, my listener.Locations is empty, which is at least one leading edge indicator the location rules are neither entering nor exiting, etc.

By comparison, if I think in terms of a Boost Spirit, let's say, I would like for each of the pairs to be producing something, that which they should be producing, or at least offer an opportunity to roll things up into said production, if that makes any sense.

grammar ScumParser;

// Lexer rules

DIGIT: [0-9];
X: [xX];
Y: [yY];
Z: [zZ];

DOT: '.';

SEP: ' ' | '\t';

DECIMAL: ( DIGIT+ DOT? ) | ( DIGIT* DOT DIGIT+ );

OP_EQ: '=';

// Carriage Return, Line Feed.
fragment CR: '\r';
fragment LF: '\n';
// Space.
fragment SP: ' ';
// Tab, or Horizontal Tab.
fragment HT: '\t';
// Form Feed, or Vertical Tab.
fragment VT: '\f';

fragment IdentPart: X | Y | Z;
fragment DecimalPart: DECIMAL;

// Whitespace
WS: ( SP | HT | VT )+ -> skip;

// NEWLINE : '\r'? '\n' { {System.out.println("Newlines so far: " + (++counter)); } };
NL: CR? LF -> skip;

// Grammar rules

linesDecl: lineDecl+;

// TODO: we may also want to consider simple comments, allowing event admins more flexibility conveying sets of coordinates
// Location is hundred percent optional in this instance i.e. lines may be blank
lineDecl: location? NL;

location: ( locationDecl | identLocationDecl );

locationDecl: coordinateDecl ( SEP coordinateDecl )*;

identLocationDecl: identCoordinateDecl ( SEP identCoordinateDecl )*;

// The case where coordinate may appear without identifier i.e. <decimal/>
coordinateDecl: DecimalPart;

// The case where coordinate may be axis labeled i.e. X|Y|Z = <decimal/>
identCoordinateDecl: IdentPart SEP OP_EQ SEP DecimalPart;

[mwpowellhtx]

Ah I think I see what happened, had a snafu somewhere in one of the productions. I simplified and it flows cleanly now. 😄

[mwpowellhtx]

Curious though... why would "1 2 3\r\n" not parse, but "10 20 30\r\n" does.

Looking at the lexer parts...

DECIMAL: ( DIGIT+ DOT? ) | ( DIGIT* DOT DIGIT+ );

+ is one or more, correct? But ignores # apparently? Is fine for ## or more?

Similarly Kleene * is zero or more, correct?

[mwpowellhtx]

In a test case such as this: W=1 Y=2 Z=3, I'd actually like to ignore the "first" axis coordinate loosely so called, W=1 pairing, and let the Y and Z coordinate cases fall through. It seems likely to me that perhaps I need more of an IDENTIFIER lexer construct, perhaps, with appropriate listener Enum interpretation, let's say.

fragment X: [xX];
fragment Y: [yY];
fragment Z: [zZ];

AXIS: X | Y | Z;

But it seems like to me it is tripping up over W=, skipping over all that, and attempting to see 1 rather as a non-axis coordinate only. I'm not sure how that's possible, considering I think I have those cases isolated in separate rules.

location: coordinate ( SEP+ coordinate )*;

axesLocation: axisCoordinate ( SEP+ axisCoordinate )*;

// The case where coordinate may appear without identifier i.e. <decimal/>
coordinate: DECIMAL_LITERAL;

// The case where coordinate may be axis labeled i.e. X|Y|Z = <decimal/>
axisCoordinate: AXIS SEP* OP_EQ SEP* DECIMAL_LITERAL;

Along similar lines, cases such as asdasd 2 3 may reject the whole thing, or perhaps allow Y=2 Z=3 cases to call through.

[mwpowellhtx]

...decided to drop the horizontal tab from the definition. This was causing angst and not just for Antlr, but also xUnit, in ways I did not want to think about. Favored simple space separation, which seems to have cleared things up.

Still not very happy that it seems to ignore "invalid" Axis use cases and land in the raw coordinate use cases. I'm not sure how to avert that, necessarily. I ended up throwing an internal exception when I discovered the corresponding AxisType? value happened to yield null. Was the best I could do, and seems somewhat idiomatic in the grand parser-listener scheme of things. Would be better, perhaps, if we had a way of cross referencing parser from listener and building an exception from one of the Antlr base classes, but this will work for the time being.

[mwpowellhtx]

This thought occurred to me. Instead of fighting the alternate use case, what I really need, I feel, are two separate grammars. On feeding parser of the axis flavored coordinates. Another featuring the raw set of coordinates. Done and dusted. I think that's probably the best case scenario. No problem then parsing one, followed by the other, stitched up and self contained. Now if there is also a way to include common elements, we'll see if that is also a possibility. Cheers 🍻