Possible to generate text mate grammar with option to change the strings?

[dhrubomoy]

Currently the text mate grammar generated by langium generator has the strings rule's escape hardcoded (I think). In the DSL I am working on strings are escaped differently:

• for single quote '' (e.g. 'ab''cd' )
• double quote by "" (e.g. "ab""cd" )
• square bracket by ]] (e.g. [ab]]cd]] )

So the string rules should look like following:

{
  "patterns": [
    {
      "name": "string.quoted.double.my-dsl",
      "begin": "\"",
      "end": "\"",
      "patterns": [
        {
          "include": "#string-character-escape-double"
        }
      ]
    },
    {
      "name": "string.quoted.single.my-dsl",
      "begin": "'",
      "end": "'",
      "patterns": [
        {
          "include": "#string-character-escape-single"
        }
      ]
    },
    {
      "name": "string.quoted.bracket.my-dsl",
      "begin": "\\[",
      "end": "\\]",
      "patterns": [
        {
          "include": "#string-character-escape-bracket"
        }
      ]
    }
  ],
  "repository": {
    "string-character-escape-double": {
      "name": "constant.character.escape.my-dsl",
      "match": "\"\""
    },
    "string-character-escape-single": {
      "name": "constant.character.escape.my-dsl",
      "match": "''"
    },
    "string-character-escape-bracket": {
      "name": "constant.character.escape.my-dsl",
      "match": "\\]\\]"
    }
  }
}

It would be great to add the option to define the string rules (may be in the language-config.json), unless there is already a way and I am missing it 😄

[msujew]

Hey @dhrubomoy,

1. Can you share the string terminal regexp you have defined? The generator uses a heuristic to find which characters are valid start- and end-characters for the purposes of generating the textmate grammar. With a slight adjustment, the heuristic might be better at finding those characters.
2. The generated highlighting grammars (as said above) mostly generated through heuristics, because a 1:1 transformation of the Langium grammar to a textmate grammar isn't possible. Therefore, we recommend adopters to stop using the generated textmate/monarch grammars at some point and write a custom grammar based off of that. Often times we use a script that collects all of the keywords of the grammar and merges them with a handwritten grammar.

[dhrubomoy]

Thank a lot @msujew for the reply. Here are the string terminal rules

terminal fragment SINGLE_QUOTED_STRING: /'(?:[^']|'')*'/;
terminal fragment DOUBLE_QUOTED_STRING: /"(?:[^"]|"")*"/;
terminal BRACKET_STRING returns string: /\[(?:[^\]]|(?:\]\]))*\]/;
terminal STRING returns string: SINGLE_QUOTED_STRING | DOUBLE_QUOTED_STRING | BRACKET_STRING;

Sounds good. I have also followed a similar approach. First generating the textmate grammar then wrote a script to patch those string changes to the textmate grammar.