More precise completions and namespace member completions

[minestarks]

Feature overview

This PR enables:

1. Much more precise keyword completions where we only show the relevant syntax at a particular cursor location. For example:

within { } | 

At the cursor location indicated by |, we should only get the keyword apply .

2. More precise completions for names where we take the syntactic context into account. For example:

operation Foo(x : | )

Here, we should only get type names.

3. Namespace member completions. For example:

import Std.Diagnostics.|

The completions here should return only items that exist under Std.Diagnostics.

Code changes

Parser completions module

Introduce the completion module in the parser.

• possible_words_at_offset_in_source - a new API to return completions at a particular offset . Uses the parser directly as the best source of knowledge about what tokens to expect.
• Introduce the WordKinds flags to define the different kinds of word tokens that the parser can expect during parsing. (See compiler/qsc_parse/src/completion/word_kinds.rs for longer description)
• See compiler/qsc_parse/src/completion/collector.rs for a description of how this all works.
• Sprinkle expect()s at all possible locations in the parser where we expect a word.

Error recovery - AST changes

• Introduce PathKind::Ok and PathKind::Err .
  • This new approach allows for a valid AST node to exist even when path parsing fails for a partial path. This is good for syntax such as "open Std." where we need:
    • The identifier nodes preceding . to be valid, so we can use them to look up possible completions.
    • The open statement to also be a valid node, so we can tell that the context requires a namespace (and not, say, a type name).
  • Most existing code will continue to only process valid paths.
• Remove Idents struct and replace it with a trait implemented by various types such as Path and Ident slices. This should make the convenience methods easier to use without cloning in/out of the different types we need.

Error recovery - parser changes

• Recovering struct parsing, so that e.g. new Foo. is still parsed as a valid struct expression node, allowing us to access the Foo ident from the AST
• Recovering open statement parsing so that open Foo. can be recognized as an open item in the AST.

Language service

In language_service is essentially a complete rewrite of completions.rs, so I suggest reviewing the new code only and ditching the diff.

Other parser changes

• Move implicit namespace parsing a bit so that it works better with the completion mechanism.
  • This also fixes the spurious "expected item, found eof" error when document only contains one token
• Delete the dead path path_field_op
• Add some missing keywords to item parsing recovery (export, import)

Other notes

• Locals completions exclude imports from locals (they conflict with globals already added to completion) (fix in resolve.rs)
• language-configuration.json : and removed wonky word pattern in playground and VS Code because we don't need it anymore now that we have proper . completions! Also update trigger characters.
• Only include the samples snippets in empty documents, because now with the more concise completion list, it gets quite intrusive for them to show up at every word when typing code.
• Update integration test to account for the fact that completions are more accurate now.

Not in this PR (coming next)

• Struct and UDT field access completions.

[github-actions]

Change in memory usage detected by benchmark.

Memory Report for 6fbbec5

Test	This Branch	On Main	Difference
compile core + standard lib	17940700 bytes	17932132 bytes	8568 bytes

[minestarks]

Change in memory usage detected by benchmark.

Memory Report for 6fbbec5

Test This Branch On Main Difference
compile core + standard lib 17940700 bytes 17932132 bytes 8568 bytes

I ran some tests to narrow it down. This increase is attributable entirely to the changes in the AST. A bunch of Path or Idents fields have been replaced with PathKind, a Span field has been added to ImportOrExportItem. It adds up to a net increase in memory.

I can't think of a way to shrink the AST any further, we are storing more information in there after all. So I think the 0.04% increase in memory usage is a worthwhile tradeoff for the feature.

[github-actions]

Change in memory usage detected by benchmark.

Memory Report for c119d44

Test	This Branch	On Main	Difference
compile core + standard lib	17945380 bytes	17936812 bytes	8568 bytes

[github-actions]

Change in memory usage detected by benchmark.

Memory Report for a9e7e76

Test	This Branch	On Main	Difference
compile core + standard lib	17945380 bytes	17936812 bytes	8568 bytes

[ScottCarda-MS]

partially reviewed, will continue to review

[minestarks]

@idavis found a bug:

Trying to complete

import QIR.in;

causes a panic since in is a keyword and that's a parse tree I did not expect to have to handle....

Working on the fix.

[minestarks]

@idavis found a bug:

Trying to complete

import QIR.in;

causes a panic since in is a keyword and that's a parse tree I did not expect to have to handle....

Working on the fix.

Fixed with my latest commit.

[github-actions]

Change in memory usage detected by benchmark.

Memory Report for 96002f0

Test	This Branch	On Main	Difference
compile core + standard lib	17945380 bytes	17936812 bytes	8568 bytes

[minestarks]

Fixed with my latest commit.

Okay, I regressed things pretty badly with this fix, as @ScottCarda-MS discovered through testing (thanks!).

The following piece of code should produce a mostly valid AST (save for the error on line 3).

namespace Sample {
    operation BellStates() : (Result, Result)[] {
        Std.
        let bellStateTuples = [];

        for (label, prepare) in bellStateTuples {}
        return measurements;
    }

    /// doc comment
    operation PreparePhiPlus() : Unit {
    }
}

With my latest commit, there's no AST at all.

cc reviewers @idavis @swernli . I'm going to have to go back to the drawing board with the "keyword after dot" fix that handles QIR.in and similar paths. Expect more parser changes :/

[minestarks]

I pushed what I think is my final update. The parser change to fix the last issue ended up being minimal!

Please sign off if no more concerns.

[ScottCarda-MS]

Discussed offline that the merge conflict is minor. Everything else looks good, and I tested out the feature manually.

[github-actions]

Change in memory usage detected by benchmark.

Memory Report for 145b6e6

Test	This Branch	On Main	Difference
compile core + standard lib	17945380 bytes	17936812 bytes	8568 bytes