Printing and AST into text

[coolya]

Hi,

is there a way to serialise an AST back into its concrete syntax representation?
For example a command line tool that parses a file, applies some modifications to AST it has parsed and then saves them back into their textual representation.

What I have seen right now is that the formatter only work if a AST node has a CST node attached to it. Otherwise the formatters completely ignore them.

Cheers,
Kolja

[spoenemann]

Hi, a serializer does not exist yet for Langium, but there are thoughts about creating such a project. We could do this in a separate package langium-stringify (separate because Langium's base functionality does not depend on this rather complex feature).

Current solution until the new package is available: hand-write an AST-to-text generator in the same way as you would write other code generators.

CC @sailingKieler @msujew @danieldietrich @pluralia @insafuhrmann

[coolya]

Do you have concrete plans to work on it? Otherwise I might be able to step in and build such functionality. It doesn't seem too hard to derive a AST-to-text serialiser from the grammar.

[spoenemann]

It is actually a very hard problem that requires complex analysis of the grammar. You're welcome to team up with the persons I mentioned in CC above, who already expressed interest in this topic before.

[Lotes]

It was explained to me why it is a hard problem... but I forgot it tbh... we could add an issue for that problem that also contains the reasons why a generic approach is that hard. I can do it if s.o. gives me a summary.

From my naive thinking a serialization is just a model-to-text transformation... but it gets more difficult when you want to be stable from the previous state of the file... Docs... Ascii art... whitespace management...
Was that the core of the difficulty?

[msujew]

Some reasons why serialization can be a difficult problem:

1. As mentioned by @Lotes, how do we incorporate existing hidden-nodes (comments, etc) into a newly generated model?
2. For whitespace management, we would need a default formatter that can deal with most common grammars without mangling the output.
3. Some AST elements don't have clear rule association [1]. How do we determine which rules have to be used for serialization?
4. How do we serialize unassigned data? Keywords aren't an issue, but how do we deal with unassigned rules/fragments or terminals such as [2], where we don't know how to reconstruct it?
5. How do we efficiently deal with incorrectly constructed AST that the user wants to serialize?

[1]:

Add: Mult ({Binary.left=current} op='+' right=Mult)*;
Mult: Prim ({Binary.left=current} op='*' right=Prim)*;

[2]:

Rule: Term data=ID;
terminal Term: /x+/;