Issue with monaco workers when bundling `monaco-languageclient` and the monaco editor in a npm package

[vemonet]

Hi, I am trying to create a npm package that bundles the monaco-editor and a custom wasm compiled language server (qlue-ls)

It aims to be modular, and is 2 levels deep:

• Yasqe contains the query editor (monaco-languageclient + qlue-ls wasm language client)
• Yasgui is a configurable UI containing the editor (Yasqe) + additional stuff around
• The Yasgui package is then deployed to a HTML page compiled with vite

The 2 can be deployed simply with a call to a JS class given an HTML element and a JSON config:

import Yasgui from "@sib-swiss/yasgui";

const yasgui = new Yasgui(document.getElementById("yasgui"), {/* config */})

I managed to make this work in dev, but I am facing monaco workers import issues when building and deploying it in a HTML vite-compiled app.

For the vite config, I took inspiration from your own root config, and a config from an app that was working for bundling monaco + the language client directly, but the behavior seems different when building a lib. When building a lib it seems like the workers are compiled as base64 instead of having .wasm files (when building apps with similar vite config) bundled in dist/assets

• Our Yasqe config
• Our demo app config

I am using the configureDefaultWorkerFactory, cf. editor setup here, and import importMetaUrlPlugin from "@codingame/esbuild-import-meta-url-plugin"; in our vite configs.

You can see check the demo with the described issue deployed here: https://vemonet.github.io/Yasgui

At initial load the monaco editor is shown, but the syntax highlighting is not triggered, we are getting this error in the console:

TypeError: Error resolving module specifier “./main-BaMT9ISm.js”. data:dmFyIENoYXJDb2RlOwooZnVuY3Rpb24gKENoYXJDb2RlKSB7CiAgICBDaGFyQ29kZVtDaGFyQ29kZVsiTnVsbCJdID0gMF0
Uncaught (in promise) TypeError: Error resolving module specifier “./main-BaMT9ISm.js”. extension.api-DoXXMoHF-DDgp3BwH-B4ANaycD.js:27887:24
TypeError: Error resolving module specifier “./main-BaMT9ISm.js”.

The error in extension.api.###.js points to code in the class WebWorkerProtocol in vsCode workers related code

As well as a warning related to sourceMap URLs of workers:

Source map error: Error: URL constructor:  is not a valid URL.
Stack in the worker:resolveSourceMapURL@resource://devtools/client/shared/source-map-loader/utils/fetchSourceMap.js:56:22
getOriginalURLs@resource://devtools/client/shared/source-map-loader/source-map.js:74:24
workerHandler/</<@resource://devtools/client/shared/worker-utils.js:115:52
workerHandler/<@resource://devtools/client/shared/worker-utils.js:113:13

Resource URL: wasm:http://localhost:4173/Yasgui/assets/main-Dx4Qdq1s-Qpe4Nb0D-C6BcJyJa.js%20line%2031%20%3E%20WebAssembly.instantiate
Source Map URL: null

As soon as we enter a character in the editor then:

• proper syntax highlighting kicks in (maybe some monaco editor service fallback?)
• the qlue-ls language server autocomplete properly works

This seems to point at the fact that the workers are actually bundled in the libs and app, they just have issues with monaco workers imports at init

Click here to see how to test it locally

git clone https://github.com/vemonet/Yasgui
cd Yasgui
npm i

Dev works:

npm run dev

But build and deploying the demo gives the described error:

npm run build && npm run demo:preview

My questions are:

• Do you have recommendation on how to define vite.config.ts to enable proper bundling of the monaco-languageclient in a reusable lib? Or comments on some bad config I might have set? I have tried to apply everything I could find in your docs, but could not find much about lib bundling.
• Any pointers to code of npm packages that managed to properly bundle monaco and the monaco-languageclient and simplify its distribution?
  • monaco-languageclient seems to have managed to do it, but I could not find any vite.config.ts for the libs in the packages folder, there is just a vite.config.ts at the root, but it is not clear how you managed to properly packaged the monaco editor in libs
• Is this even a good idea to try to bundle monaco and the monaco-languageclient? Or am I trying to do something for which the current ESM import system and vite is not yet ready for? (bundling monaco workers in ES modules)

This issue is a follow up to #950 after migrating from monaco-editor-wrapper to the latest monaco-languageclient version

[CGNonofr]

Do you have recommendation on how to define vite.config.ts to enable proper bundling of the monaco-languageclient in a reusable lib? Or comments on some bad config I might have set? I have tried to apply everything I could find in your docs, but could not find much about lib bundling.

I have the feeling vite target is final projects. To build libraries, I would chose a simpler bundler (like rollup directly) to have more control over the result

Any pointers to code of npm packages that managed to properly bundle monaco and the monaco-languageclient and simplify its distribution?

What do you mean by bundle? what is the point compared to just be depending on it?

[vemonet]

I have the feeling vite target is final projects. To build libraries, I would chose a simpler bundler (like rollup directly) to have more control over the result

vite is using rollup for production build, with custom rollupOptions just for build: https://vite.dev/guide/build#customizing-the-build so migrating to rollup will not change anything, everything I can do in rollup I can do through rollupOptions

The tricky part is finding the right rollupOptions so that monaco workers imports are properly bundled for the lib build

There is this plugin for esbuild that fixes imports: import importMetaUrlPlugin from "@codingame/esbuild-import-meta-url-plugin";

Could it be that rollup needs something similar?

What do you mean by bundle? what is the point compared to just be depending on it?

So that devs who wants to integrate the editor can just do npm i --save @sib-swiss/Yasgui and call it this way:

import Yasgui from "@sib-swiss/yasgui";

const yasgui = new Yasgui(document.getElementById("yasgui"), {/* config */})

Without having to think about the monaco editor setup, and without having to maintain it (which would greatly increase complexity to use it)

It is to build a reusable component to enable maintainers of publicly available databases (most of the time for research or public services) to deploy a query interface for their DB on their frontend. Ease of deployment and maintenance is important, they don't have an army of experienced web devs available at all time to make sure all the components are properly linked in their website just to deploy a text editor

[CGNonofr]

vite is using rollup for production build, with custom rollupOptions just for build: https://vite.dev/guide/build#customizing-the-build so migrating to rollup will not change anything, everything I can do in rollup I can do through rollupOptions

It will change that you won't have all the vite configuration/plugins configured by default

The tricky part is finding the right rollupOptions so that monaco workers imports are properly bundled for the lib build

If you don't bundle it, you don't have to care about it

So that devs who wants to integrate the editor can just do npm i --save @sib-swiss/Yasgui and call it this way:

I'm not sure how bundling it would help 🤔

[CGNonofr]

I guess you want it to work out of the box with any bundler 🤔 so maybe bundling the workers can help, so they can be referenced as assets. but bundling the rest of the code is pointless

[kaisalmen]

Hi @vemonet a possible way forward could be to bundle the library and all required workers. You need some wrapper index that re-exports all sub-exports from monaco-languageclient and contains its own monacoWorkerFactory referring to the bundled workers, but this may still be error-prone and @CGNonofr has a point here that bundling the library may not solve the issue.

You could also pre-configure monaco-languageclient or expose certain functionality and then users building applications can use documented bundling instructions for dev or production. We deploy the examples without need for backend to GitHub Pages and use these two vite configurations (https://github.com/TypeFox/monaco-languageclient/blob/main/packages/examples/vite.production.base.ts and https://github.com/TypeFox/monaco-languageclient/blob/main/packages/examples/vite.config.deploy.ts). That just works with editor and textmate worker, but no library is created/bundled in the process.

You could embed a monacoWorkerFactory referring to pre-bundled workers in you package and then export a base configuration to users and thereby mitigating the problem.

Btw, use rolldown or vite 8 Beta. The speed improvements are really impressive (10x or even faster bundling times).

[kaisalmen]

@vemonet I have a working solution using vite:
Turbopack / next 16 still gives me a hard time to make all worker work with it, but a by-product I created was to test pre-bundling of the whole monaco-languageclient configuration and all workers. This does what you need and proves that is possible, see this here:
https://github.com/TypeFox/monaco-languageclient/tree/nextjs-16/verify/next

Run npm i and npm run dev:vite afterwards. Don't bother about next. The important files are vite.config.bundle.ts, vite.config.editorWorker.ts, vite.config.extHostWorker.ts and vite.config.textmateWorker.ts. The dev server uses the default vite config file. This file is bundled. This is the application and the index.html needs to contain the css from the bundle. Hope this helps. 🙂

[vemonet]

Thanks a lot @kaisalmen ! Your pointers will be really helpful in this investigation, that seems to match what I am trying to do

It is crazy it requires so many different config files to make vite/rollup/rolldown just capable of importing wasm workers... Feels like going back to those long webpack configs 10 years ago 😅

guess you want it to work out of the box with any bundler 🤔 so maybe bundling the workers can help, so they can be referenced as assets. but bundling the rest of the code is pointless

I am not an expert in JS bundling (I have some experience with npm packages, but not for complex setup such as the monaco service and its army of workers, usually the default vite config is good enough). I don't really want to bundle everything, I just want all dependencies to be properly installed when doing npm i --save @sib-swiss/Yasgui, and imports to be successfully resolved (including the one that uses workers, which does not seems to be the case by default...)

Logically I think that it would be preferable if my code just point to "we need to get this packages/versions from npm", and they get installed from their own npm package (instead of bundling everything in my npm package). After all that's what every package manager should be doing (and even browsers are doing it since they all support direct ESM imports now), and that was the whole point behind ES modules.

But it does not seems like wasm web workers are playing out well yet with this kind of package management

It's weird to see that web workers and ESM came around about the same time, but the 2 don't work well together.