"TypeError: onigBinding.UTF8ToString is not a function" after some time / parallel requests in Next.js

[hlubek]

Validations

• Follow our Code of Conduct
• Read the Contributing Guide.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

Describe the bug

We experienced crashes in production when using shikijs in a Next.js 14 application in React Server Components.

Initially the rendering works, but after a few (parallel?) requests we have the following stacktrace and errors are returned in the rendering:

TypeError: onigBinding.UTF8ToString is not a function
    at throwLastOnigError (file:///app/node_modules/@shikijs/core/dist/index.mjs:4677:38)
    at new OnigScanner (file:///app/node_modules/@shikijs/core/dist/index.mjs:4883:13)
    at createOnigScanner (file:///app/node_modules/@shikijs/core/dist/index.mjs:5024:12)
    at Object.createOnigScanner (file:///app/node_modules/@shikijs/core/dist/index.mjs:5358:20)
    at Grammar.createOnigScanner (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:2275:30)
    at new CompiledRule (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:1270:32)
    at RegExpSourceList._resolveAnchors (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:1260:16)
    at RegExpSourceList.compileAG (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:1237:56)
    at IncludeOnlyRule.compileAG (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:728:57)
    at prepareRuleSearch (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:2107:30)
TypeError: onigBinding.UTF8ToString is not a function
    at throwLastOnigError (file:///app/node_modules/@shikijs/core/dist/index.mjs:4677:38)
    at new OnigScanner (file:///app/node_modules/@shikijs/core/dist/index.mjs:4883:13)
    at createOnigScanner (file:///app/node_modules/@shikijs/core/dist/index.mjs:5024:12)
    at Object.createOnigScanner (file:///app/node_modules/@shikijs/core/dist/index.mjs:5358:20)
    at Grammar.createOnigScanner (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:2275:30)
    at new CompiledRule (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:1270:32)
    at RegExpSourceList.compile (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:1219:28)
    at RegExpSourceList.compileAG (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:1225:25)
    at IncludeOnlyRule.compileAG (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:728:57)
    at prepareRuleSearch (file:///app/node_modules/@shikijs/core/dist/textmate.mjs:2107:30)
[Error: An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance
  digest: '25876654'
}
[Error: An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance
  digest: '3589831450'
}

This is our code:

export default async function Code({ title, code, language }: CodeProps) {
  const html = await codeToHtml({
    code,
    language,
  });

  return <div
        dangerouslySetInnerHTML={{ __html: html }}
      />
}

Where our syntax definition comes from https://github.com/networkteam/vscode-neos-fusion/blob/master/syntaxes/fusion.tmLanguage.json .

import { bundledLanguages, getHighlighter } from 'shiki/bundle/web';
import fusion from '../data/fusion.tmLanguage.json';

export const codeToHtml = async ({ code, language }: { code: string; language: string }) => {
  const highlighter = await getHighlighter({
    themes: ['dark-plus'],
    langs: [...Object.keys(bundledLanguages), fusion as any],
  });

  return highlighter.codeToHtml(code, {
    lang: language,
    themes: {
      dark: 'dark-plus',
      light: 'dark-plus',
    },
    colorReplacements: {
      '#1e1e1e': '#282828',
    },
  });
};

We could reproduce the behaviour with different Next.js versions (>=14) and by just refreshing constantly with F5 in the browser.

Our workaround for now is to use unsable_cache to cache the generated highlighted HTML.

Reproduction

https://codesandbox.io/p/devbox/silly-grass-ngrp2h

Contributes

• I am willing to submit a PR to fix this issue
• I am willing to submit a PR with failing tests

[zhangyu1818]

Same error, but I'm using@shikijs/rehype in mdx, and I don't know how to solve it yet.

When I reverted to the code that I could deploy yesterday and redeployed it, it still failed.

I'm not sure if this problem is caused by Next.js or Shiki, but I have seen this error many times in the development environment, and I have to restart the development server every time.

By the way, I don't think this title describes the error very well😂

[slax57]

FTR, I ran into the same issue while trying to use the shiki rehype plugin in a node script, to preprocess numerous markdown files to build the documentation website of our framework.

My investigations tend to show this issue is linked to the memory usage. Indeed the node process memory kept increasing (until around 2.3 GB) until this error was thrown.

By playing with the node_modules/@shikijs/core/dist/index.mjs source file, I also managed to get a RuntimeError: memory access out of bounds with a stacktrace pointing to some wasm code.

Finally, I managed to workaround this issue by splitting the files I had to process into several chunks, spawning a new node process for each chunk.
To me, this also gives credit to the memory management hypothesis.

[Neo-Zhixing]

Ran into the same problem in an application that uses @nuxtjs/mdc when preprocessing a large number of markdown files. I reverted shiki version as far back as 1.6.0 but that didn't make a difference.

[slax57]

@Neo-Zhixing I believe the code responsible for this (at least, according to the stacktrace) was introduced in shiki 1.0.0, so you'd have to downgrade to v0 to have a chance to get it working.
However in my case I couldn't do that because I'm using the rehype plugin, so can't be affirmative this will work.

[antfu]

This down to two problems

• UTF8ToString was missing during throwing the error, where the error is fail to memory allocation
  • fixed in 0c89157
• Memory leak was due to getHighlighter being called too many times, where the instance should be cached
  • improved the docs and added a runtime warning in 412fe10

[Neo-Zhixing]

Thanks! That was fast.

[slax57]

Thanks for the update!

For those using the @shikijs/rehype plugin just like me, note that you can pass a getHighlighter singleton instance to the plugin by following the instructions from the Fine-grained Bundle doc section:

import { unified } from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import rehypeShikiFromHighlighter from '@shikijs/rehype/core'

import { getHighlighterCore } from 'shiki/core'

const highlighter = await getHighlighterCore({
  themes: [
    import('shiki/themes/vitesse-light.mjs')
  ],
  langs: [
    import('shiki/langs/javascript.mjs'),
  ],
  loadWasm: import('shiki/wasm')
})

const raw = await fs.readFile('./input.md')
const file = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeShikiFromHighlighter, highlighter, {
    //                             ^^^^^^^^^^^ singleton instance
    themes: {
      light: 'vitesse-light',
      dark: 'vitesse-dark',
    }
  })
  .use(rehypeStringify)
  .processSync(raw) // it's also possible to process synchronously

Btw, maybe it would be worth mentioning this good practice in the plugin's doc page (https://shiki.style/packages/rehype) too @antfu , wdyt?

[antfu]

Yeah sure, PR welcome. I also work on #702 to make it less confusing.

[Gerrit0]

Memory leak was due to getHighlighter being called too many times, where the instance should be cached

This raises the question... is there a way to ask Shiki to dispose of that memory?

[antfu]

Found a proper way to dispose the instance #707, will landed in v1.8