Make parse tree serializable and interchangeable between runtimes

[dimmik]

Scenario:
Python runtime.
Relatively complex language grammar, and really complex translation code (from language to ASM).

The grammar is parsed in several seconds in python.
Even though translation code is complex, the translation runs in ~0.3s in python.
So, tree parsing is consuming ~90% of processing time.
(tried) The same grammar parsing is 3 times faster in c#, and (guess, didn't really try) 10+x faster in C.

Devs, implementing the translation code, are familiar with python, not C/C#.

Suggestion:
It would have been nice if the parse tree, that is result of parsing, was produced by C code, then serialized, then loaded into python code, and then used for translation.
That requires the parse tree to be serializable, and understandable by both (ideally all) runtimes.

That would unblock both performance (that is way better in C) and dev speed (that is better in python).

[jimidle]

Are you translating directly from the parse tree? It would be much easier for you to generate an ast and serialize that. Then you have full control of it all. Also, how efficient is your grammar? Tree walking should be fast, but if your grammar is ambiguous then you well lose performance there. Python isn’t the correct choice for a fast parse though, as you have noticed.

…

On Fri, Aug 23, 2024 at 01:16 dimmik ***@***.***> wrote: *Scenario:* Python runtime. Relatively complex language grammar, and really complex translation code (from language to ASM). The grammar is parsed in several seconds in python. Even though translation code is complex, the translation runs in ~0.3s in python. So, tree parsing is consuming ~90% of processing time. (tried) 3 times faster in c#, (guess, didn't really try) 10+x faster in C. Devs, implementing the translation code, are familiar with python, not C/C#. *Suggestion:* It would have been nice if the parse tree, that is result of parsing, was produced by C code, then serialized, then loaded into python code, and then used for translation. That requires the parse tree to be serializable, and understandable by both (ideally all) runtimes. That would unblock both performance (that is way better in C) and dev speed (that is better in python). — Reply to this email directly, view it on GitHub <#4680>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMEXQR7OYHK7QQ54D3DZS3OUZAVCNFSM6AAAAABM7TCZL2VHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGA4DKNRRGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dimmik]

Definitely anything can be implemented, and any problem can be addressed in this or another way.
Request is not about my problem, request is about making parse tree serializable.
The scenario is just an illustration of when it might be useful.

[jimidle]

It is unlikely to be implemented. While the data may be the same, the generators do things differently, including static initialization. Take a look at all the different generated code. This seems like an XY question

…

On Sat, Aug 31, 2024 at 04:02 dimmik ***@***.***> wrote: Definitely anything can be implemented, and any problem can be addressed in this or another way. Request is not about my problem, request is about making parse tree serializable. The scenario is just an illustration of when it might be useful. — Reply to this email directly, view it on GitHub <#4680 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMBNUM2UN7TXI22RWSLZUGIDXAVCNFSM6AAAAABM7TCZL2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJQGYZDOMA> . You are receiving this because you commented.Message ID: ***@***.***>