Investigate ES6 modules as our internal components

[kripken]

(Forked off from #5794 (comment) )

As a long-term goal, it would be nice if emscripten output

• Fit in with node.js packaging in a natural way.
• Fit in with ES6 build system tools.
• Were overall as modular as possible.

Those might be achievable by splitting all or most of the JS we currently emit into ES6 modules. So the GL code might be such a module, etc. Less obvious how the runtime would, but could be possible.

This is not something we can do right now (in particular, not all browsers support ES6 modules, and wasm integration there is farther out), but in the linked issue above we discuss various refactorings, and it could be useful to keep the ES6 modular goal in mind, so that we are working towards that.

We'll need to do a bunch of investigation along the way, including

• Code and compile time implications of such changes. In theory compile-time tools can merge ES6 modules and remove the overhead (of imports etc.), but we should verify that (e.g., it might remove imports but leave more JS objects or IIFEs around as "namespaces"), and we'll need to see how fast those things are.
• How such modularization would work with our various build flag options (separate module for GL vs GL emulation? or at compile time like now? etc.).
• How it would fit with our EM_ASM and js-library options.
• Look at Rust's new approach to how JS and wasm integrate. Still early there, but we should see what they do and how it goes.

[kripken]

One possible way towards this may be

• Allow JS libraries to be ES6 modules.
• Move more things into JS libraries.

[lukewagner]

That sounds like the right first incremental steps. Additional incremental steps could be:

• Create some notion of "package" that is a collection of .h, .cpp, and .js files that can be explicitly depended on by other packages.
  • The .js files would be ES modules, importable by the .cpp files.
  • Static linking of multiple packages would simply collect all the .js files from all the packages and optionally merge them into a single .js module with a tool like webpack or rollup.
• Allow packages to be compiled in a "clean" environment in which the only code they can #include and link against are packages they've explicitly included as dependencies.
  • Like in npm does for ES modules with the node_modules folder
• Break up all standard Emscripten runtime/libraries/glue-code into fine-granularity packages
  • Provide a compatibility mode/flag/command that implicitly depends on all the standard packages, thereby preserving today's behavior
• Build up a workflow for creating packages, publishing packages to a package manager (e.g., npm), and installing external packages

Following this to its logical conclusion (which may require some of the breaking changes described in #5794 to avoid otherwise-unpackage-able JS glue code) would I think lead to the end state described in my #5984 comment.

[saschanaz]

I think we may use ES2015 modules (and even any ES2015+ syntax) before browser support with Babel+Webpack dependency. Both depends on Node.js we already have so probably okay to have them together.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[caspervonb]

I've been playing around with this lately in a standalone helper script for LLVM, here's what I've learned so far.

I've done a couple of passes on it, earlier iterations handled archives in a special way allowing .js files to be embedded in archives but the simplest approach turned out to be embedding the source in custom sections. Primarily it allows for most things to be left as-is including linker flag as an archive is now just an archive which may or may not have ECMAScript modules embedded within it.

Essentially I allow ECMAScript modules to be input files just as C source files would be. They're preprocessed with cpp -E -P ... before being embedded into .ll files which contain the source in NUL terminated strings in a custom section named esm.

At this stage it makes sense to also mark ECMAScript exports as extern_weak symbols in the .ll file but without having type annotations in the ECMAScript code this is non-trivial and requires additional parsing so for now I've skipped doing this.

Clang/LLVM then does it's thing with the modified arguments, wasm-ld concatenates the custom sections into a single section during linker hence the need for a delimiter.

Then wasm-ld spits out a module I extract each fragment of the esm section into numbered files (Each fragment is written to it's own file to maintain scope).

Once every partial file is extracted a top level env.js is generated which exports every export from the fragments that were extracted.

Finally entry point/loader is generated by looking at the exports of the module. I generate the importsObject by explicitly naming each member I want to be kept as it lets compilers/bundlers do a much better job at eliminating code with everything being depended on explicitly.

Code and compile time implications of such changes. In theory compile-time tools can merge ES6 modules and remove the overhead (of imports etc.), but we should verify that (e.g., it might remove imports but leave more JS objects or IIFEs around as "namespaces"), and we'll need to see how fast those things are.

Depends on the bundler but Rollup leaves everything in a nice single flat namespace, Webpack is more versatile but slower and noisier output. Closure Compiler produces the smallest bundles but is also the slowest by far; an empty project takes about 6 seconds to boot.

How such modularization would work with our various build flag options (separate module for GL vs GL emulation? or at compile time like now? etc.).

If emulated features can be designed to just go away when not being called that would be the best way.

Otherwise aim for GL{N} and GLES{N} libraries?

How it would fit with our EM_ASM and js-library options.

As for EM_ASM and EM_JS, while it's possible to do with pre-processing it would be easier with upstream support from LLVM.

This can be done trough an attribute, I found that rust's bindgen tools often uses an attribute to embed things like typescript typings in custom sections.

__attribute__((section("esm"))
const char* foo = "inline";

Or inline assembly; not a huge fan of doing with this approach but saw there has been some work on the parser side in LLVM already and tried to implement it this way but seems the patch hasn't landed.

__asm__ {
.custom_section esm
  ...
}

Upstream LLVM support in any direction would go along way here, section attributes would probably be the most useful across toolchains. Rust uses them to embed arbitrary data in various tools but they have their own front-end attribute for it.

As a side effect, this would resolve #9366 as the section can easily be removed by wasm-ld, wasm-opt etc.

The -js-library option would go away with time as ECMAScript fragments would be embedded inside of objects which are inside an archive so it would become just -larchive or -lshared_module. The fact that some parts are in JavaScript just becomes an implementation detail the same way having fragments of Objective C source code in a library made up of mostly C is just a detail.

[kripken]

cc @jgravelle-google , see last comment.

[caspervonb]

Couldn't find the exact issue when writing the comment earlier but managed to dig it up now; Rust went with a similar approach to embedding data in custom sections with a wasm_custom_section attribute in their frontend which was merged in rust-lang/rust#48883 and later merged into their link_section attribute which I assume is their equivalent to the section attribute found in Clang.