feat: introduce GrammarState

[antfu]

Motivations

Sometimes, we might want to highlight a partial of the code snippet, which might not necessarily be a full language statement.

For example, if we want to highlight a TypeScript type annotation Pick<MyObject, string>[] we might do:

import { codeToHtml } from 'shiki'

const highlighted = await codeToHtml('Pick<MyObject, string>[]', { lang: 'ts', theme: 'github-dark' })

We will get the result like this:

This isn't accurate. This is because from the grammar, Shiki doesn't know it's a type annotation already.

Instead, we could append let a: to our code to hint that:

import { codeToHtml } from 'shiki'

const highlighted = await codeToHtml('let a: Pick<MyObject, string>[]', { lang: 'ts', theme: 'github-dark' })

Which will now render:

While the highlighting is now correct, we have another challenge: removing the "temporary snippet" we just added from the highlighted HTML.

Solutions

This PR introduces a new concept, GrammarState, with a few APIs around it. GrammarState is a special token that holds the grammar context and allows you to highlight from an intermediate grammar state, making it easier to highlight code snippets.

For example, you can get the grammar state with the getLastGrammarState method, and pass into the grammarState option:

import { createHighlighter } from 'shiki'

const shiki = await createHighlighter({ langs: ['ts'], themes: ['github-dark'] })

const stateTypeAnnotation = shiki.getLastGrammarState('let a:', { lang: 'ts', theme: 'github-dark' })

const highlightedType = shiki.codeToHtml(
  'Pick<MyObject, string>[]',
  {
     lang: 'ts', 
     theme: 'github-dark',
     grammarState: stateTypeAnnotation // <--- this
  }
)

Now Shiki would highlight correctly as it knows to start as the type annotation. You can keep that grammar state object for multiple uses as well.

---

For one-off grammar context shifting, we also provide a shorthand by the grammarContextCode option:

const highlightedType = shiki.codeToHtml(
  'Pick<MyObject, string>[]',
  {
     lang: 'ts', 
     theme: 'github-dark',
     grammarContextCode: 'let a:' // same as above, a temporary grammar state is created internally
  }
)

Credits

Thanks to @benjamincanac for the idea and discussions

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.30%. Comparing base (603713d) to head (0953f83).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #712      +/-   ##
==========================================
+ Coverage   96.21%   96.30%   +0.09%     
==========================================
  Files          71       72       +1     
  Lines        6072     6226     +154     
  Branches      805      829      +24     
==========================================
+ Hits         5842     5996     +154     
  Misses        225      225              
  Partials        5        5              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[orta]

Cool idea