scan_string functionality?

[aanastasiou]

Hello

I have been using pyparsing for parsing but a grammar I had to deal with recently is getting increasingly more complex and Lark's definition and "automatic" AST transformation look more attractive.

However, there is one thing I have not been able to replicate with Lark and that is the scan_string() functionality.

scan_string takes the grammar and a string to parse and returns the fragments (as begin, end pairs) of the original string that matched the grammar as well as the parsed results of the matched fragments.

What I am working on is similar to a template language: Text that contains markup that delineates expressions of a DSL.

With pyparsing I simply scan the text, get the fragments and then substitute the paragraph span with the result from the evaluation of the DSL.

But, with Lark, I have to somehow express the text surrounding the markup so that the whole text gets parsed. Simple rules such as:

PAR_OPEN: "$$"
PAR_CLOSE:"$$"
GENERIC_TEXT: /.*?/ PAR_OPEN

dsl_exp: . . . 

segment: GENERIC_TEXT PAR_OPEN dsl_exp PAR_CLOSE

start: segment+

Do not work, as the parser stops at PAR_OPEN.

I have also tried ugly constructions such as trying to parse a string from location k onwards and upon success, advancing k to the end of the parsed segment, upon failure advancing k to the next character....But I don't like where this is going. I have also tried capturing the generic text via an on_error function (a variant of the above solution) but it results in equally ugly code.

Is there a better practice to deal with this problem using Lark?

EDIT:

OR split the string first on the markup marker and then evaluate the even index strings ... If there is a way of handling everything via Lark though it would be nice to learn about it.