Catch-all grammars using property span-lists

[ohad]

One general way to support arbitrary grammars is to allow pairing source-files with property span-lists with the appropriate syntax highlighting. So the project offers linguist a property span-list file for some source-files, and instead of offering a 3rd party grammar, directly offer the correct highlighting for each file fragment.

Does this kind of design makes sense?

Is linguist the right place for this kind of feature?

Can linguist's architecture accommodate this feature?

[michaelmesser]

I supporting adding semantic tokens to LSIF and allowing users to have a GitHub action generate an LSIF index to provide semantic tokens, jump to definition, search by symbol, etc. This would allow support for any language that could generate an LSIF index.

[lildude]

Is linguist the right place for this kind of feature?

Can linguist's architecture accommodate this feature?

Unfortunately not. Syntax highlighting is performed by a completely separate service. All Linguist does is curate the list of grammars for that service. That service comprises two engines called PrettyLights and Treelights. PrettyLights is the old syntax highlighting engines which uses the TextMate compatible grammars that Linguist curates (it uses the tarball added to each release). Treelights is the new engine that uses tree-sitter grammars which are not curated by Linguist.

This use of tree-sitter grammars also ties in with code navigation on github which is powered by https://github.com/github/semantic