Incorrect highlighting when colon is used inside yaml literal style

[aeschli]

From @tnikodym on October 12, 2016 14:34

• VSCode Version: 1.6.0
• OS Version: OS X El Capitan (10.11.6)

See the screenshot below. When a colon is used inside literal content (indicated by | character), the entire line up to the colon is highlighted as a property.

test: |
    note that this is blue and it should't:
    when there's no colon, it works correctly
    same problem here: but after the colon it's OK

test: |
    this works correctly

Note: I have not seen this issue in the previous version of VSCode.

Copied from original issue: microsoft/vscode#13601

[infininight]

This issue and the other four just closed as duplicates all are tied to the same underlying issue. The string rule is not being properly entered in #block-scalar due to how \G is being interpreted during a line re-parse. For many contexts it can appear to be working and it will work when a file is initially read but not as being entered or upon editing the first line.

This is a complicated issue to fix, working on how to do so now.

[ValentinFunk]

Is there anything we can do to help? Would more test cases help?

[infininight]

There is not, this is a current limitation of the grammar engine and I don't believe it is possible to fix on the grammar side. Unless I figure out an alternate method this will need to wait for a fix on the parser side of things.

[ValentinFunk]

@aeschli seeing the answer above, would it make sense then to reopen the vscode issue microsoft/vscode#13601 since it can't be fixed in the grammar and actually needs a change on the vscode side as well?

[infininight]

@kamshak So a better description of the issue since I haven't talked it about it yet in these issues:

\G was added as a zero width assertion to match the end of the parent begin that started the active pattern array. This is a non-standard extension to the regex parser by co-opting an already existing but unused assertion. TextMate grammars are line-based which means that each line is parsed separately, and changes on a line only necessitate repaying that single line unless it causes a new state to be caused which will cause latter lines to be re-parsed as well.

The current limitation is that the state that \G uses is not saved from a previous line when a line is re-parsed due to editing. This is why there is inconsistency noticed, the initial parsing runs the line through in order and the state is saved. Due to the line based nature of the parser this isn't really so much a bug as just a limitation of the parser, a exception would have to be made in this case to save the state for later re-use.

If we decide to make this change it will need to be updated in VSCode and GitHub as well, I'll open issues with both if this is the course taken. The alternative is to find another solution to workaround the limitation, but so far I have not been successful in finding an alternative, without being able to save the state it's hard to not prematurely exit the rule.

[trondhindenes]

Any movement on this? It's a very annoying bug :-(

[ValentinFunk]

Since @infininight has kindly explained the issue in a very detailed way, maybe someone familiar with VSCode would be willing to take a look at it :) It seems we're stuck without some help

Sorry for tagging you out of the blue @alexandrudima @bpasero, is there a chance you could take a quick look at this issue (or tag someone who could)? Would be very much appreciated! ❤️

[thislooksfun]

You mentioned that it was a parsing issue. I just wanted to chime in and say that multiline string highlighting works perfectly if the string is enclosed in quotes.

test:
  "note that this green as it should be:
  when there's no colon, it also works correctly
  same solution here: and after the colon it's OK"

test: |
  but when using a pipe, it breaks again:
  when there's no colon, it works correctly
  same problem here: but after the colon it's OK
    
test: >
  when using a right arrow it also breaks:
  when there's no colon, it works correctly
  same problem here: but after the colon it's OK

test:
  "this also works correctly"

test: |
  as does this
  
test: >
  and this

test:
  and also this

Hopefully this helps narrow down the issue and/or potential solutions.

[phrounz]

It seems to detect comments used inside yaml multiline value too, which is also wrong.

Also it seems to depend whether the first line of the multiline value is an empty line or not.

though, as far as I know, both are valid YAML (There is no reason a multiline could not start by a line return):

$ perl -e 'use YAML qw/LoadFile/;use Data::Dumper;print Dumper(LoadFile("test.yaml"));'
$VAR1 = {
          'my_test2' => 'dsq
# test2
',
          'my_test1' => '
 dsq
 # test1
'
        };

(Color is back to normal if I put two spaces instead of the empty line, but VSCode tends to remove them afterwards.)

My VSCode version:

$ code --version
1.67.2
c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5
x64

$ cat /etc/fedora-release 
Fedora release 36 (Thirty Six)