Code actions: pushMode/popMode/...

[Lotes]

Antlr4 has this feature 'lexer modes', where you can set, push and pop a mode as extra state. This enables the user to define tokens for WITHIN a string or a comment, and at the end switch back to the previous mode.

You can have it somehow in Langium when you implement your own TokenBuilder.

My question is: Should Langium also have this option (new code actions)? Is there a need by Langium's users?

//I am also asking to get this idea out of my head :3 tehee

[montymxb]

Are you thinking about adding this kind of support to the syntax of Langium? If so I've thought about it as well, similar to how Monarch structures which tokens can be recognized by state. By default the behavior would just be the current state, but it could get tricky if a user wants multiple states to still recognize the same token.

[msujew]

I have two thoughts on this, one in favor of this idea and one against it:

• [+] It would make information about lexer modes statically available. That is both easier to read and easier to statically verify. Currently, if you build a multi mode lexer using the TokenBuilder, you only know whether it works correctly (or works at all), once you start parsing code. With static availability, we can already verify the lexer behavior when running langium generate or maybe even earlier in the grammar validation phase in the editor.
• [-] It increases complexity of the grammar language. My original idea was to improve the API of the TokenBuilder to facilitate easier multi mode lexing instead of adding more complexity to the grammar language.

@spoenemann Your thoughts on that? Do you think the feature is worth the complexity/effort?

[Lotes]

I have 4 proposals…

1. take the same approach as Antlr4 and introduce code actions pushMode(name) and popMode. The consequence is that you have to introduce regions of tokens. Each mode has a set of tokens. Potentially duplicating existing tokens from other modes. This is a very pragmaatic approach.

2. you can create the same mechanism without polluting the syntax of Langium. By adding a custom TokenBuilder. Where you have to again, define your modes of tokens and care about key tokens that switch between modes using push and pop semantics.

3.1) introducing annotations like in Java to add metadata to your rules. Within the token builder, you can lookup this data inside of a Langium grammar instance.
3.2) you can also introduce JavaDoc as annotation. Within the token builder you can lookup the modes using reflection and setup modes again.

4. The actual goal is to mix grammars. To do that, you have to switch the lexer mode/token set via push/pop. What I actually would love to see is:

grammar Xml;

import Dtd; //the name of a different grammar!!!

entry XmlDocument: decl=XmlDecl dtd+=Dtd* root=XmlElement;
//I hope the DTD position is correct

XmlDecl…

So when XmlDecl ends and a grammar call might be expected, we could push the mode to the DTD lexer. When the DTD parser reaches EOF, we pop the mode.

[spoenemann]

The parsing approach that we're using separates lexing and parsing into separate phases, so we can't mix lexer mode switches into parser rules.

Of all the options you described, I currently like 1. the most. Something like this should work?

lexermode M1
terminal T1: {push mode M1} ...;
terminal T2: {pop mode} ...;

[Lotes]

4.) could work when we request tokens from a tokenizer using JS generators. But I am not sure if that is sufficient in cases where it has to fetch enough tokens to make a decision. It would indeed complicate Langium.

1.) is the good and way that is already tested by ANTLR4. And yes, you would have...

mode A:
terminal X: ...

mode B:
terminal Y: ...
terminal X: ...

Suggestion: We could put the terminals for one mode into parentheses mode XXX{ terminal A:...} to have it in a more clean way.
In Antlr4 you can mix parser and terminal rule, which can be overwhelming. But that's just details...

[spoenemann]

4.) could work when we request tokens from a tokenizer using JS generators.

I think Chevrotain doesn't support that:

export interface IMultiModeLexerDefinition {
  modes: MultiModesDefinition
  defaultMode: string
}

export type TokenTypeDictionary = { [tokenName: string]: TokenType }

export declare type TokenVocabulary =
  | TokenTypeDictionary
  | TokenType[]
  | IMultiModeLexerDefinition

Suggestion: We could put the terminals for one mode into parentheses mode XXX{ terminal A:...} to have it in a more clean way.

Some terminals should be available in multiple modes, so the modes should be a property of a terminal?

[Lotes]

Some terminals should be available in multiple modes, so the modes should be a property of a terminal?

That is also an idea. It was already proposed with 3.1 and 3.2 ...

One has to provide a method that introduces a n:m mapping between terminals and modes, where terminals must be in a specific order (a keyword can be shadowed by ID, when ID is defined first).

So terminals can be in multiple modes, and a mode contains multiple terminals. Annotating the terminals with their modes as a list of strings can be a nice solution. Maybe I dropped too much information, to make my proposals fully understandable?!

I think Chevrotain doesn't support that

Ok, let's do not reinvent the wheel. I mean in theory you can do so much, but when Chevrotain does not provide such a on-the-fly token generation, let's live with it...