C-style Preprocessor includes and simple macro expansion?

[pheller]

Hi there,

I'm working to create a grammar for an existing language with Langium. This language supports preprocessor include and simple macro exapansion that can occur basically anywhere in the language where there is a statement separator.

In previous work on a lex/yacc-based compiler for this language, includes are easily handled in the lexer by maintaining a stack of file pointers, pushing onto the stack and opening a new file stream when encountering an include directive, and popping on EOF to resume the prior input stream. I have not yet undertaken work to implement the define directive in the lex/yacc toolchain yet, but I was looking into use of the error token, YYBACKUP, and unput in the lexer to push the expansion into the token stream.

Any suggestions how to accomplish this in Langium?

--phil

[msujew]

Hey @pheller,

Similar to the lex/yacc toolchain you can implement a custom lexer that takes care of special requirements like these. The lexer is by default tightly coupled to Chevrotain, but you can supply your own implementation and convert the tokens to the interface that Chevrotain expects.

I imagine that such a feature massively mess with how the location-based LSP features work (basically all of them), due to locations in the source text (i.e. the vscode editor document) and the actual text (after performing the preprocessing) not being the same. You can probably work around this by selectively reimplementing such services.