Express Unicode codepoints over 0xFFFF

[iacore]

On page http://www.unicode.org/faq/blocks_ranges.html:
Unicode characters may be encoded at any code point from U+0000 to U+10FFFF.

So how do I express that in Ohm?

[pdubroy]

(Caveat: I'm bumping up against the limits of my Unicode knowledge here, but I wanted to respond quickly so I can understand a bit more about your use case before I do some more reading.)

It depends a bit on what exactly you're trying to do. You can embed those characters directly in a terminal expression, and that should work. E.g.:

ohm.grammar('G { start = "😀" }').match('😀').succeeded() // true

You can also encode it as a surrogate pair (same as in ES5 and JSON). E.g., assume you have the following grammar in g.ohm:

G {
  start = "\uD83D\uDE00"
}

..then this will work:

ohm.grammar(fs.readFileSync('g.ohm', 'utf-8')).match('😀').succeeded() // true

What we don't yet support properly is ranges involving such code points. E.g., the following is not possible (but should be):

G {
  start = "😁".."🥴"
}

I've opened up #355 for that.

If you can give me some more details about exactly what you need, we can see if any of these options would cover your use case, or if there's still something we're missing.

[iacore]

I'm actually trying to parse Koka, where the text-form of the language need three independent passes.

1. Text->Tokens
2. Layout (turn indent into braces and insert semicolons)
3. Tokens->AST

I tried using tree-sitter, but it only accepts a stream of unicode codepoints as input. The first step (tokenize) does not need utf-8 decoding; the second step need a stack to remember indentation, and the thrid step is can be parsed if the input is a stream of tokens.

There is no existing tool that I know that allow me to write the turn PEG (in first and thrid step) into a parser.

[pdubroy]

Thanks for the details. In that case, I wonder if this might already be possible to do in Ohm by exclusion — e.g.,

notIgnored = "\x00".."\x80"
highSurrogate = "\uD800".."\uDBFF"
ignored = ~notIgnored highSurrogate any
        | ~notIgnored any 

Maybe something like that would work. Regardless, I'll still work on #355 and see if I can maybe get a fix in the next few days.

[pdubroy]

I've just published [email protected] with experimental support for ES2015-style Unicode code point escapes (\u{...}) which are also supported in range expressions. @locriacyber it would be great if you can give it a try and let me know if you run into any problems!

Example:

ohm.grammar(String.raw`G { start = "\u{10000}".."\u{10FFFF}" }`).match('🤠').succeeded() // true

[iacore]

@pdubroy thank you for your work! I'm only using ohm as a prototyping tool, since my final language parser will not ship with Node.js.

Few problems:

1. How to disable built-in rules like whitespace?
2. When a complex rule may be empty, the error message just says the rule may be empty, without telling which branch may be empty.
3. The editor cannot display emoji even if I have emoji font installed. (see below)

https://ohmlang.github.io/editor/?ohm-js=16.3.0-dev.unicode-code-point-escape#0a9a649c3c630fd0a470ba6cb75393fe

Test input: var x = "😻"; I have no idea why this emoji exist though.

[pdubroy]

Unfortunately, the last one is a known issue in the editor: ohmjs/ohm-editor#52. In fact, thanks to your original question, I think I know what the problem is now...we are likely rendering each 16-bit code unit individually, rather than rendering the entire code point.

[pdubroy]

How to disable built-in rules like whitespace?

What do you mean by this? There is no way to disable built-in rules — but you can choose to not use them. The spaces and space rules are an exception, as they are invoked implicitly. You can find more details here.

If you want to disable implicit space skipping entirely, you can use only lexical rules (i.e., rules that begin with a lowercase letter) in your grammar as you've done here. However, typically what you'd want to do is redefine the space rule for your language. E.g., take a look at how our ES5 grammar does it.

How to exclude a few character from a range of characters? This is useful for string literals, where only a few characters are invalid between quotes

You can use negative lookahead for this purpose. See Patterns & Pitfalls > Delimited strings for an example.

When a complex rule may be empty, the error message just says the rule may be empty, without telling which branch may be empty.

Can you attach the error that you're encountering and I can try to help you debug it?

[iacore]

I've already debugged it myself by checking each rule. However, I think the compiler should say which rule might be empty, since people work on a grammar step by step.

Example:

Step 1: (no error so far)

tokens = token_a | token_b | token_c | token_d
token_a = "a"
token_b = "b"
token_c = "c"
token_d = ""

Step 2:

program = tokens* // error here
tokens = token_a | token_b | token_c | token_d
token_a = "a"
token_b = "b"
token_c = "c"
token_d = ""

This example is trivial to debug, but a complex language grammar have deeply nested rules.