Feature Request: Produce fewer semantic tokens if the client uses augmentsSyntaxTokens

[frankplow]

rust-analyzer produces a lot of semantic tokens, including those for simple syntax features such as parentheses and semicolons. Most other LSP servers do not produce such a large set of tokens and so perform better than rust-analyzer for me when semantic tokens are enabled.

I believe the intended use of the augmentsSyntaxTokens client capability is to indicate to the server that these simple tokens which do not require semantic analysis will be highlighted independently by the client and do not need to be provided by the server.

[Veykril]

That is one confusing capability, the documentation basically states nothing about what it means at all 😕

[frankplow]

@Veykril Yes it's not very helpful really. The use case described in my OP is the only reason I can see the client needing to communicate this to the server, so I've made this presumption. There's the other issue of a very fuzzy line between what's better served by static grammar and what's better served by semantic analysis as well (strings jump to mind). I think the feature is still worth trying to implement though - the difference in performance compared to, for example, clangd is substantial (to the extent where I turn off semantic highlighting when editing Rust).

[Veykril]

The question really is, what semantic tokens are relevant and what aren't for this. To me this capability sounds borderline useless, though I guess just not emitting semantic tokens for stupid punctuation would be a decent start. Instead of the capability this really sounds like the server should just offer configurations for disabling certain kinds of semantic tokens though (like we have a setting for disabling string highlighting).

Regarding performance, are you referring to the client not handling semantic tokens well if there are a lot of them?

[flodiebold]

@Veykril I think it'd be fine to leave out purely syntax-based tokens, but err on the side of sending slightly too much. Having ways to disable additional token types would be helpful on top of that.

Regarding performance, in my testing our semantic highlighting was pretty unusable in Emacs for example, so I think this would be great to have.

[frankplow]

@Veykril Yes by performance I mean the client struggling with that many tokens rather than rust-analyzer itself. There's a slight performance difference between the two in terms of the time taken to respond to the request as well, but this is insignificant compared to the time for the client to apply the tokens.

sounds like the server should just offer configurations for disabling certain kinds of semantic tokens

I've never quite understood how the tokenTypes in the client capabilities is meant to work to be honest. Most servers (rust-analyzer included I believe, though correct me if I'm wrong) just seem to discard this.

Not producing those punctuation tokens in the first place is just as good a solution to the problem for me.

[Veykril]

So, I wonder what exactly we should interpret this flag as. I assume it really just means only sent tokens over that differ from their syntax (like usages of const parameters, which syntactically look like any other one segment path etc). It seems like an odd setting since the server has no idea what the client can differentiate on its own and what not honestly.

[frankplow]

Here is the discussion leading to the addition of the capability.

Yeah it seems that it's intended to indicate that the client is doing some traditional static highlighting, and the server only needs to send tokens which require semantic analysis.