Annotations on rules and other concepts of the Langium grammar

[Lotes]

Hi!

I am about to write a grammar that uses multimode lexing. We have no equivalent to pushMode/popMode like it is done in ANTLR and it also might be better to keep that actions out of the Langium grammar specification (lower complexity of Langium).

But I had a second idea that might help to write other parts like TokenBuilder, ScopeProvider and other dependents of the grammar: Annotations on rules and other concepts.

Annotations could look like this:

@Annotation(target='Rule', multiple=true)
interface Mode {
  name: string;
}

@Mode(name='Normal')
@Mode(name='AnotherMode')
entry TranslationUnit: stmts+=Statement+;

Inside of the TokenBuilder you would be able to filter the rules by mode!

One downside I see is Langium's additional complexity of handling values. You certainly need a concept for constants to not duplicate StringLiterals all over the place. And we might also need some first StandardLibrary for the @annotation annotation...

//...

const ModeNormal = 'Normal';

@Mode(name=ModeNormal)
entry TranslationUnit: stmts+=Statement+;

@Mode(name=ModeNormal)
Statement: ...;

The big advantage is that you could annotate any constant values, and you would not have to extend Langium grammar further, because it is possible to cover additional features with a mix of annotations and smart extensions (default implementation overrides).

Another thing is the syntax of the feature. We have to think well about what we want. Entire JSONs, primitive types, forbid lambda for complexity reasons. Bli Bla Blub

WDYT?

[spoenemann]

These Java-style annotations always remind me of things that should have been part of the core language syntax (like @Override) or frameworks that build heavily on reflection (like Spring and many other server technologies). But that's just a syntactic view, the more important thing is what we want to do with that info.

I didn't get why you want to annotate parser rules: the lexing phase works only with the terminal rules (and keywords). It would be cool to see how a custom TokenBuilder using annotations of terminal rules could look like.

[svallory]

Hey @spoenemann

My concern about changing the grammar is that the additional syntax would have absolutely no effect unless you use it in your custom code.

That's exactly my goal with #818 !

I want my grammar file (but not necessarily the grammar) to be the only source of truth for all things impacted by changes in the language. Like, TextMate Syntax grammars. Right now, the process of creating tmLanguage files is nerve-racking. Automatic language documentation generation could be another use.

But ultimately, I think a mechanism to add metadata to tokens would open up a world of possibilities for third-party tools to complement the Langium ecosystem

[msujew]

#866 will add JSDoc support including tags as proposed in #818.

[svallory]

Awesome, @msujew! Will the JSDoc be parsed and available programmatically? Will it be a strict JSDoc implementation or can custom annotations (e.e @scope) be added?

[msujew]

@svallory It's a pretty flexible implementation and doesn't really care about the actual tag values. Unless it's @link which has special handling. You can see the API here and some testing usage here.

[svallory]

Looks like I'll be working on automatic TextMate syntax highlighting generation soon :)