diff --git a/README.md b/README.md index 6b4092ad..29be37fb 100644 --- a/README.md +++ b/README.md @@ -15,64 +15,18 @@ ## About this repo -**Introduction**: Run your JavaScript on WebAssembly. Javy takes your JavaScript code, and executes it in a WebAssembly embedded JavaScript runtime. Javy can create _very_ small Wasm modules in the 1 to 16 KB range with use of dynamic linking. The default static linking produces modules that are at least 869 KB in size. +**Introduction**: Run your JavaScript on WebAssembly. Javy takes your JavaScript +code, and executes it in a WebAssembly embedded JavaScript runtime. Javy can +create _very_ small Wasm modules in the 1 to 16 KB range with use of dynamic +linking. The default static linking produces modules that are at least 869 KB in +size. -## Runtime requirements +## Installation -When running the official Javy binary on Linux, `glibc` 2.31 or greater must be available. You may need to update the version of your operating system if you are using an older version of `glibc`. +Pre-compiled binaries of the Javy CLI can be found on [the releases +page](https://github.com/bytecodealliance/javy/releases). -## Extending Javy - -If you would like to use Javy for your own project but need to make some changes, read the [extending Javy documentation](docs/extending.md) for one approach to consider. - -## Contributing - -We welcome feedback, bug reports and bug fixes. We're also happy to discuss feature development but please discuss the features in an issue before contributing. - -Read our [contribution documentation](docs/contributing.md) for additional information on contributing to Javy. - -## Requirements to build - -- On Ubuntu, `sudo apt-get install curl pkg-config libssl-dev clang` -- [rustup](https://rustup.rs/) -- Stable Rust, installed via `rustup install stable && rustup default stable` -- wasm32-wasi, can be installed via `rustup target add wasm32-wasi` -- cmake, depending on your operating system and architecture, it might not be - installed by default. On MacOS it can be installed with `homebrew` via `brew - install cmake`. On Ubuntu, `sudo apt-get install cmake`. -- Rosetta 2 if running MacOS on Apple Silicon, can be installed via - `softwareupdate --install-rosetta` - -## Development requirements - -- wasmtime-cli, can be installed via `cargo install wasmtime-cli` (required for - `cargo-wasi`) -- cargo-wasi, can be installed via `cargo install cargo-wasi` -- cargo-hack, can be installed via `cargo +stable install cargo-hack --locked` - -## How to build - -Inside the Javy repository, run: -``` -$ cargo build -p javy-core --target=wasm32-wasi -r -$ cargo build -p javy-cli -r -``` - -Alternatively if you want to install the Javy CLI globally, inside the Javy repository run: -``` -$ cargo build -p javy-core --target=wasm32-wasi -r -$ cargo install --path crates/cli -``` - -If you are going to recompile frequently, you may want to prepend `CARGO_PROFILE_RELEASE_LTO=off` to cargo build for the CLI to speed up the build. - -## Using Javy - -Pre-compiled binaries of the Javy CLI can be found on [the releases page](https://github.com/bytecodealliance/javy/releases). - -Javy supports ES2023 JavaScript. Javy does _not_ provide support for NodeJS or CommonJS APIs. - -### Compiling to WebAssembly +## Example Define your JavaScript like: @@ -144,163 +98,6 @@ $ echo '{ "n": 2, "bar": "baz" }' | wasmtime index.wasm {"foo":3,"newBar":"baz!"}% ``` -If you have a lot of JavaScript and you want to reduce compile times, try using the `--no-source-compression` flag. It will skip compressing the JavaScript source code when generating the Wasm module but will result in the Wasm module being larger. - -```bash -javy build index.js -o destination/index.wasm -C source-compression=n -``` - -### Exporting functions - -To export exported JavaScript functions, you can pass a WIT file and WIT world when running `javy build`. Only ESM exports are supported (that is, Node.js/CommonJS exports are _not_ supported). For each exported JavaScript function, Javy will add an additional function export to the WebAssembly module. Exported functions with arguments and generators are not supported. Return values will also be dropped and not returned. The Wasm module generated is a core Wasm module, **not** a Wasm component. +## Documentation -An example looks like: - -`index.js`: -```javascript -export function foo() { - console.log("Hello from foo!"); -} - -console.log("Hello world!"); -``` - -`index.wit`: -``` -package local:main; - -world index-world { - export foo: func(); -} -``` - -In the terminal: -```bash -$ javy build index.js -C wit=index.wit -C wit-world=index-world -o index.wasm -$ wasmtime run --invoke foo index.wasm -Hello world! -Hello from foo! -``` - -The WIT package name and WIT world name do not matter as long as they are present and syntactically correct WIT (that is, it needs to be two names separated by a `:`). The name of the WIT world (that is, the value after `world` and before `{`) must be passed as the `-n` argument. The `-n` argument identifies the WIT world in the WIT file for the Wasm module generated by `javy build`. - -#### Exports with multiple words - -Exported function names with multiple words have to written in kebab-case in the WIT file (that is a restriction imposed by WIT), they are exported from the Wasm module as kebab-case to match the WIT, and Javy will match the WIT export to a JS export with the same name but in camel-case. - -`index.js`: -```javascript -export function fooBar() { - console.log("In foo-bar"); -} -``` - -`index.wit`: -``` -package local:main; - -world index { - export foo-bar: func(); -} -``` - -In the terminal: -```bash -$ javy build index.js -C wit=index.wit -C wit-world=index -o index.wasm -$ wasmtime run --invoke foo-bar index.wasm -In foo-bar -``` - -#### Exporting a default function - -Exporting a function named `default` in the WIT world exports a function named `default` on the Wasm module and corresponds to either an exported default function or exported default arrow function in JS. - -`index.js`: -```javascript -export default function () { - console.log("In default"); -} -``` - -`index.wit`: -``` -package local:main; - -world index { - export default: func(); -} -``` - -In the terminal: -```bash -$ javy build index.js -C wit=index.wit -C wit-world=index -o index.wasm -$ wasmtime run --invoke default index.wasm -In default -``` - -You can also export a default function by writing: -```javascript -export default () => { - console.log("default"); -} -``` - -### Invoking Javy-generated modules programatically - -Javy-generated modules are by design WASI only and follow the [command pattern](https://github.com/WebAssembly/WASI/blob/snapshot-01/design/application-abi.md#current-unstable-abi). Any input must be passed via `stdin` and any output will be placed in `stdout`. This is especially important when invoking Javy modules from a custom embedding. - -In a runtime like Wasmtime, [wasmtime-wasi]( -https://docs.rs/wasmtime-wasi/latest/wasmtime_wasi/struct.WasiCtx.html#method.set_stdin) -can be used to set the input and retrieve the output. - -To embed Javy in a Node.js application see this [example](docs/nodejs-embedding.md). - -### Creating and using dynamically linked modules - -An important use for Javy is for when you may want or need to generate much smaller Wasm modules. Using the `-d` flag when invoking Javy will create a dynamically linked module which will have a much smaller file size than a statically linked module. Statically linked modules embed the JS engine inside the module while dynamically linked modules rely on Wasm imports to provide the JS engine. Dynamically linked modules have special requirements that statically linked modules do not and will not execute in WebAssembly runtimes that do not meet these requirements. - -To successfully instantiate and run a dynamically linked Javy module, the execution environment must provide a `javy_quickjs_provider_v2` namespace for importing that links to the exports provided by the `javy_quickjs_provider.wasm` module. Dynamically linked modules **cannot** be instantiated in environments that do not provide this import. - -Dynamically linked Javy modules are tied to QuickJS since they use QuickJS's bytecode representation. - -#### Obtaining the QuickJS provider module - -The `javy_quickjs_provider.wasm` module is available as an asset on the Javy release you are using. It can also be obtained by running `javy emit-provider -o ` to write the module into ``. - -#### Creating and running a dynamically linked module on the CLI - -``` -$ echo 'console.log("hello world!");' > my_code.js -$ javy build -C dynamic -o my_code.wasm my_code.js -$ javy emit-provider -o provider.wasm -$ wasmtime run --preload javy_quickjs_provider_v2=provider.wasm my_code.wasm -hello world! -``` - -## Releasing - -1. Update the root `Cargo.toml` with the new version -2. Create a tag for the new version like `v0.2.0` -``` -git tag v0.2.0 -git push origin --tags -``` -3. Create a new release from the new tag in github [here](https://github.com/bytecodealliance/javy/releases/new). -4. A GitHub Action will trigger for `publish.yml` when a release is published ([i.e. it doesn't run on drafts](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#:~:text=created%2C%20edited%2C%20or%20deleted%20activity%20types%20for%20draft%20releases)), creating the artifacts for downloading. - -## Testing locally - -1. Clone submodules -``` -git submodules init -git submodules update -``` -2. Install cargo hack -``` -cargo +stable install cargo-hack --locked -``` -3. Run tests, eg: -``` -cargo +stable install cargo-hack --locked -``` -CARGO_TARGET_WASM32_WASI_RUNNER="wasmtime --dir=." cargo hack wasi test --workspace --exclude=javy-cli --exclude=javy-config --each-feature -- --nocapture +Read the documentation [here](./docs/index.md) diff --git a/docs/contributing-architecture.md b/docs/contributing-architecture.md deleted file mode 100644 index 60a9f331..00000000 --- a/docs/contributing-architecture.md +++ /dev/null @@ -1,111 +0,0 @@ -# Architecture of Javy - -This document is intended to provide an overview of the crates and NPM packages in Javy. - -## Crates - -```mermaid -flowchart TD - javy-cli --> wasm - subgraph wasm[javy_core.wasm / javy_quickjs_provider.wasm] - javy-core --> javy - javy --> rquickjs - end -``` - -We anticipate most changes will be to the `javy-cli` and `javy` crates. - -### `javy` - -The entrypoint for working with Javy as a library for third parties. This crate is intended to compile to `wasm32-wasi` and provide ergonomic APIs for configuring a QuickJS-based runtime. If there is a configuration option for QuickJS that would be helpful, this is the place to add it. - -#### Important concepts - -- `javy::Runtime` - a configurable QuickJS runtime. -- `javy::Config` - a configuration for the runtime. - -#### Example of a change - -This is a contrived example of how to make a change. If I want to add a configuuration to set a global variable called `javy_rocks` to `true`, I would do the following: - -In `crates/javy/src/config.rs`: - -```diff - /// A configuration for [`Runtime`](crate::Runtime). - #[derive(Debug)] - pub struct Config { -+ pub(crate) set_javy_rocks: bool, - } - - impl Default for Config { - /// Creates a [`Config`] with default values. - fn default() -> Self { - Self { -+ set_javy_rocks: false, - } - } - } - - impl Config { -+ /// Sets `globalThis.javy_rocks` to `true`. -+ pub fn set_javy_rocks(&mut self) -> &mut Self { -+ self.set_javy_rocks = true; -+ self -+ } - } -``` - -We require creating a method to set the property and it should return `&mut Self` so it can be chained. - -In `crates/javy/src/runtime.rs`: - -```diff - impl Runtime { - /// Creates a new [`Runtime`]. - pub fn new(config: Config) -> Result { - let context = JSContextRef::default(); -+ if config.set_javy_rocks { -+ context -+ .global_object()? -+ .set_property("javy_rocks", context.value_from_bool(true)?)?; -+ } - Ok(Self { context }) - } -``` - -Read the `config` and call the appropriate methods on `context` to apply the configuration. - -#### When to add a Cargo feature - -You should consider gating your feature by a Cargo feature when: - -- Your feature would materially increase the size of the produced Wasm module. - -These are guidelines and we're willing to discuss if a feature needs to be gated by a Cargo feature on a case-by-case basis. - -### `javy-cli` - -The CLI for compiling JS to Wasm. This isn't intended to be a CLI that accommodates all uses for all users but rather to provide a useful base of functionality. This is kind of similar to how Wasmtime ships with a crate and a CLI and doing non-generic things with Wasmtime requires writing your own CLI around the Wasmtime crate. - -#### When to add a cargo feature - -You should gate your feature with a cargo feature if: - -- It's not commonly going to be used and it would complicate the CLI options to include enabling it. For example, printing the WAT of a dynamic module is not something users would want 99.9% of the time and including it as an option on the CLI would make the `--help` output harder for most users to understand. -- You want to have integration tests in the `javy-cli` crate that should only run when the `javy-core` crate is built with a non-default configuration (that is, with different cargo features enabled). For example, we introduced the `experimental_event_loop` cargo feature in the `javy-cli` crate since we test for different expected outputs when using a promise when the `experimental_event_loop` cargo feature is enabled on the `javy_core` crate compared to when that cargo feature is disabled. - -### `javy-core` - -Gets compiled to `javy_core.wasm` and `javy_quickjs_provider.wasm` for use by the CLI and in environments for running dynamically linked modules. This isn't intended to be used as a code library by third parties. Contains logic for driving the `javy` crate for Wasm modules generated by `javy-cli`. - -#### When to add a cargo feature - -You should gate your feature with a cargo feature if: - -- You want to support building a Wasm module with an experimental configuration of the runtime. We do this for the event loop because the current implementation has not been thoroughly vetted. We also need a build of Javy with event loop support to run a number of web platform tests for text encoding. - -## NPM packages - -### `javy` - -A JS library providing ergonomic helpers around the lower level APIs for I/O exposed by the `javy` crate. diff --git a/docs/contributing.md b/docs/contributing.md deleted file mode 100644 index 7df4da37..00000000 --- a/docs/contributing.md +++ /dev/null @@ -1,29 +0,0 @@ -# Contributing - -## Architecture - -See our [architecture document](contributing-architecture.md) for more information about how the project is organized. - -## Adding additional JS APIs - -We will only add JS APIs or accept contributions that add JS APIs that are potentially useful across multiple environments and do not invoke non-[WASI](https://wasi.dev/) hostcalls. If you wish to add or use JS APIs that do not meet these criteria, please use the `rquickjs` crate directly. We may revisit how we support importing and exporting custom functionality from Javy once [the Component Model](https://github.com/WebAssembly/component-model) has stabilized. - -## Versioning for library crates - -The library crate, `javy`, uses the versioning system described in [Rust for Rustaceans](https://rust-for-rustaceans.com/) in the _Unreleased Versions_ section in the _Project Structure_ chapter. The underlying motivation is that the version in the crate's `Cargo.toml` is important between releases to ensure Cargo does not reuse a stale version if a project relies on a version of the crate that has not yet been published to crates.io and the version required by that project is updated to a version with new additive or breaking changes. - -### The system - -After publishing a release, immediately update the version number to the next patch version with an `-alpha.1` suffix. The first time an additive change is made, reset the patch version to `0` and increment the minor version and reset the suffix to `-alpha.1`. When making additional additive changes, increment the number in the suffix, for example `-alpha.2`. The first time a breaking change is made, reset the patch version and minor version to `0` and increment the major version and reset the suffix to `-alpha.1`. When making additional breaking changes, increment the number in the suffix, for example `-alpha.2`. - -When releasing, remove the suffix and then publish. - -For example, let's say the last published version of `javy` is `2.0.0`, so the current version in the Cargo.toml file is `2.0.0-alpha.1`. If you add a new public function, you would change the version to `2.1.0-alpha.1`. This is because adding a new public function is considered an additive change. After merging those changes, if you add a new public function, you would change the version to `2.1.0-alpha.2`. This is because adding another new function is an additional additive change. Now if you were to make a function that was public, private, you would change the version to `3.0.0-alpha.1`. This is because removing a public function is considered a breaking change. After merging that change, if you were to then add a new public function, then you would increment the version to `3.0.0-alpha.2` because this is making an additional additive change. It's not necessary to increment the minor version in this case because version `3.0.0` has not been published yet so version `3.0.0` can contain a mixture of additive and breaking changes from the last `2.x.x` version published. - -## cargo vet - -We use [cargo vet](https://mozilla.github.io/cargo-vet/) to audit dependencies for the project. If you need to change or add dependencies, please try to use a dependency that has been audited by one one of the audits we import or is published by one of the authors we trust (sunfishcode, dtolnay, Amanieu, cuviper). This is preferable to adding new exemptions for the project. Do not add new audits for crates that are not in this project. - -## Web platform tests (WPT) - -We run a subset of the web platform test suite during continuous integration. We recommend reading our suite's [WPT readme](../wpt/README.md) for tips on how to add tests to the suite and what to do when tests won't pass. diff --git a/docs/docs-contributing-architecture.md b/docs/docs-contributing-architecture.md new file mode 100644 index 00000000..57d05ab5 --- /dev/null +++ b/docs/docs-contributing-architecture.md @@ -0,0 +1,130 @@ +# Architecture of Javy + +This document is intended to provide an overview of the architecture of `javy`. + +## Crates + +```mermaid +flowchart TD + javy-cli --> wasm + subgraph wasm[javy_core.wasm / javy_quickjs_provider.wasm] + javy-core --> javy + javy --> rquickjs + end +``` + +We anticipate most changes will be to the `javy-cli` and `javy` crates. + +### `javy` + +The entrypoint for working with Javy as a library for third parties. This crate +is intended to compile to `wasm32-wasi` and provide ergonomic APIs for +configuring a QuickJS-based runtime. If there is a configuration option for +QuickJS that would be helpful, this is the place to add it. + +#### Important concepts + +- `javy::Runtime` - a configurable QuickJS runtime. +- `javy::Config` - a configuration for the runtime. + +#### Example + +This is a contrived example of how to make a change. If I want to add +a configuuration to set a global variable called `javy_rocks` to `true`, I would +do the following: + +In `crates/javy/src/config.rs`: + +```diff + /// A configuration for [`Runtime`](crate::Runtime). + #[derive(Debug)] + pub struct Config { ++ pub(crate) set_javy_rocks: bool, + } + + impl Default for Config { + /// Creates a [`Config`] with default values. + fn default() -> Self { + Self { ++ set_javy_rocks: false, + } + } + } + + impl Config { ++ /// Sets `globalThis.javy_rocks` to `true`. ++ pub fn set_javy_rocks(&mut self) -> &mut Self { ++ self.set_javy_rocks = true; ++ self ++ } + } +``` + +In `crates/javy/src/runtime.rs`: + +```diff ++ context.with(|cx| { ++ if cfg.set_javy_rocks { ++ let globals = cx.globals(); ++ globals.set("javy_rocks", true); ++ } ++ }); ++ +``` + + +Read the `config` and call the appropriate methods on `context` to apply the +configuration. + +#### When to add a `cargo` feature + +You should consider gating your feature by a Cargo feature when your feature +would materially increase the size of the produced Wasm module. + +These are guidelines and we're willing to discuss if a feature needs to be gated +by a Cargo feature on a case-by-case basis. + +### `javy-cli` + +The CLI for compiling JS to Wasm. This isn't intended to be a CLI that +accommodates all uses for all users but rather to provide a useful base of +functionality. + +#### When to add a `cargo` feature + +You should gate your feature with a cargo feature if your feature/change: + +- Is not commonly going to be used and it would complicate the CLI options to + include enabling it. For example, printing the `.wat` of a dynamic module is not + something users would want 99.9% of the time and including it as an option on + the CLI would make the `--help` output harder for most users to understand. + +- You want to have integration tests in the `javy-cli` crate that should only + run when the `javy-core` crate is built with a non-default configuration (that + is, with different cargo features enabled). For example, we introduced the + `experimental_event_loop` cargo feature in the `javy-cli` crate since we test + for different expected outputs when using a promise when the + `experimental_event_loop` cargo feature is enabled on the `javy_core` crate + compared to when that cargo feature is disabled. + +### `javy-core` + +Gets compiled to `javy_core.wasm` and `javy_quickjs_provider.wasm` for use by +the CLI and in environments for running dynamically linked modules. This isn't +intended to be used as a code library by third parties. Contains logic for +driving the `javy` crate for Wasm modules generated by `javy-cli`. + +#### When to add a `cargo` feature + +You should gate your feature with a cargo feature if you want to support +building a Wasm module with an experimental configuration of the runtime. We do +this for the event loop because the current implementation has not been +thoroughly vetted. We also need a build of Javy with event loop support to run +a number of web platform tests for text encoding. + +## `npm` packages + +### `javy` + +A JS library providing ergonomic helpers around the lower level APIs for I/O +exposed by the `javy` crate. diff --git a/docs/docs-contributing-building.md b/docs/docs-contributing-building.md new file mode 100644 index 00000000..a1dde2e3 --- /dev/null +++ b/docs/docs-contributing-building.md @@ -0,0 +1,31 @@ +# Build requirements + +- On Ubuntu, `sudo apt-get install curl pkg-config libssl-dev clang` +- [rustup](https://rustup.rs/) +- Stable Rust, installed via `rustup install stable && rustup default stable` +- wasm32-wasi, can be installed via `rustup target add wasm32-wasi` +- cmake, depending on your operating system and architecture, it might not be + installed by default. On MacOS it can be installed with `homebrew` via `brew + install cmake`. On Ubuntu, `sudo apt-get install cmake`. +- Rosetta 2 if running MacOS on Apple Silicon, can be installed via + `softwareupdate --install-rosetta` + +# How to build + +In the repository root, run: + +``` +$ cargo build -p javy-core --target=wasm32-wasi -r +$ cargo build -p javy-cli -r +``` + +Alternatively if you want to install the `javy` binary globally, at the +repository root, run: +``` +$ cargo build -p javy-core --target=wasm32-wasi -r +$ cargo install --path crates/cli +``` + +If you are going to recompile frequently, you may want to prepend +`CARGO_PROFILE_RELEASE_LTO=off` to cargo build for the CLI to speed up the +build. diff --git a/docs/complex-data-types-in-wasm-functions.md b/docs/docs-contributing-complex-data-types.md similarity index 76% rename from docs/complex-data-types-in-wasm-functions.md rename to docs/docs-contributing-complex-data-types.md index 5ed1de07..1e87e9bf 100644 --- a/docs/complex-data-types-in-wasm-functions.md +++ b/docs/docs-contributing-complex-data-types.md @@ -1,10 +1,24 @@ # Using complex data types in Wasm functions -Core WebAssembly currently only supports using numbers for arguments and return values for exported and imported functions. This presents a problem when you want to pass strings, byte arrays, or structured data to and from imported/exported functions. The WebAssembly Component Model provides one approach to solving this problem but we have not yet added support for producing WebAssembly components to Javy. This document will provide an overview for an approach using Core WebAssembly to consider. - -At a high level, byte arrays can be passed using a pair of integers with the first integer representing the address of the start of the byte array in the instance's linear memory and the second integer representing the length of the byte array. Strings can be passed by encoding the string into a UTF-8 byte array and using the previous solution to pass the byte array. Structured data can be encoded to a JSON string and that string can be passed by encoding it into a UTF-8 byte array and using the previous solution. Other serialization formats can also be used to encode the structured data to a byte array. - -The examples below use Rust and Wasmtime to on the host however any programming language and WebAssembly runtime should support using the same approach. +Core WebAssembly currently only supports using numbers for arguments and return +values for exported and imported functions. This presents a problem when you +want to pass strings, byte arrays, or structured data to and from +imported/exported functions. The WebAssembly Component Model provides one +approach to solving this problem but we have not yet added support for producing +WebAssembly components to Javy. This document will provide an overview for an +approach using Core WebAssembly to consider. + +At a high level, byte arrays can be passed using a pair of integers with the +first integer representing the address of the start of the byte array in the +instance's linear memory and the second integer representing the length of the +byte array. Strings can be passed by encoding the string into a UTF-8 byte array +and using the previous solution to pass the byte array. Structured data can be +encoded to a JSON string and that string can be passed by encoding it into +a UTF-8 byte array and using the previous solution. Other serialization formats +can also be used to encode the structured data to a byte array. + +The examples below use Rust and Wasmtime to on the host however any programming +language and WebAssembly runtime should support using the same approach. ## For exported functions @@ -45,9 +59,12 @@ fn call_the_export(bytes: &[u8], instance: wasmtime::Instance, store: &mut wasmt } ``` -You can export the `canonical_abi_realloc` function by enabling the `export_alloc_fns` feature in the `javy` crate. +You can export the `canonical_abi_realloc` function by enabling the +`export_alloc_fns` feature in the `javy` crate. -In the WebAssembly instance when receiving a byte array in the exported function, you can use the `std::slice::from_raw_parts` function to get the slice. +In the WebAssembly instance when receiving a byte array in the exported +function, you can use the `std::slice::from_raw_parts` function to get the +slice. ```rust #[export_name = "your_fn"] @@ -66,7 +83,9 @@ Given an exported WebAssembly function that returns a byte array and looks like: ) ``` -To return a byte array from that exported function in a WebAssembly instance, you need to leak the byte array and we recommend using a static wide pointer for storing the pointer and length. +To return a byte array from that exported function in a WebAssembly instance, +you need to leak the byte array and we recommend using a static wide pointer for +storing the pointer and length. ```rust static mut BYTES_RET_AREA: [u32; 2] = [0; 2]; @@ -82,7 +101,9 @@ pub unsafe extern "C" fn your_fn() -> *const u32 { } ``` -On the host, you can use `memory.read` to populate a vector with the byte array. WebAssembly uses little-endian integers so we read 32-bit integers using `from_le_bytes`. +On the host, you can use `memory.read` to populate a vector with the byte array. +WebAssembly uses little-endian integers so we read 32-bit integers using +`from_le_bytes`. ```rust fn get_slice(instance: wasmtime::Instance, store: &mut wasmtime::Store) -> Result> { @@ -106,7 +127,8 @@ fn get_slice(instance: wasmtime::Instance, store: &mut wasmtime::Store) -> Resul ## For imported functions -Given an imported WebAssembly function that receives a byte array as an argument and looks like: +Given an imported WebAssembly function that receives a byte array as an argument +and looks like: ```wat (module @@ -114,7 +136,8 @@ Given an imported WebAssembly function that receives a byte array as an argument ) ``` -When passing a byte array to the host from the WebAssembly instance, we pass the pointer and length to the imported function: +When passing a byte array to the host from the WebAssembly instance, we pass the +pointer and length to the imported function: ```rust use anyhow::Result; @@ -129,7 +152,8 @@ fn call_the_import(bytes: &[u8]) -> Result<()> { } ``` -When receiving a byte array from the WebAssembly instance on the host, we use `memory.read` along with the pointer and length to get the byte array: +When receiving a byte array from the WebAssembly instance on the host, we use +`memory.read` along with the pointer and length to get the byte array: ```rust use anyhow::Result; @@ -169,7 +193,12 @@ Given an imported WebAssembly function that returns a byte array and looks like: ) ``` -When returning a byte array from the host, things get a little more complicated. Below we use a wide pointer to return the byte array. This requires two memory allocations in the instance, one for the byte array and one for the wide pointer, and using `memory.write` to place the array and wide pointer into the allocated memory. Since the byte array is copied into the instance's memory, there is no need to leak the original byte array. +When returning a byte array from the host, things get a little more complicated. +Below we use a wide pointer to return the byte array. This requires two memory +allocations in the instance, one for the byte array and one for the wide +pointer, and using `memory.write` to place the array and wide pointer into the +allocated memory. Since the byte array is copied into the instance's memory, +there is no need to leak the original byte array. ```rust fn setup(linker: &mut wasmtime::Linker) -> Result<()> { @@ -221,9 +250,12 @@ fn setup(linker: &mut wasmtime::Linker) -> Result<() } ``` -You can export the `canonical_abi_realloc` function by enabling the `export_alloc_fns` feature in the `javy` crate. +You can export the `canonical_abi_realloc` function by enabling the +`export_alloc_fns` feature in the `javy` crate. -When reading a returned byte array from the host, we extract the pointer and length from the wide pointer and then use the pointer and length to read a slice from memory: +When reading a returned byte array from the host, we extract the pointer and +length from the wide pointer and then use the pointer and length to read a slice +from memory: ```rust #[link(wasm_import_module = "host")] diff --git a/docs/docs-contributing-developing.md b/docs/docs-contributing-developing.md new file mode 100644 index 00000000..ca0288bc --- /dev/null +++ b/docs/docs-contributing-developing.md @@ -0,0 +1,6 @@ +# Development requirements + +- `wasmtime-cli`, can be installed via `cargo install wasmtime-cli` (required for + `cargo-wasi`) +- `cargo-wasi`, can be installed via `cargo install cargo-wasi` +- `cargo-hack`, can be installed via `cargo +stable install cargo-hack --locked` diff --git a/docs/docs-contributing-releasing.md b/docs/docs-contributing-releasing.md new file mode 100644 index 00000000..c965078a --- /dev/null +++ b/docs/docs-contributing-releasing.md @@ -0,0 +1,18 @@ +# CLI releases + +1. Update the root `Cargo.toml` with the new version, following semver. +2. Create a tag for the new version like `v0.2.0`. + +``` +git tag v0.2.0 +git push origin --tags +``` + +3. Create a new release from the new tag in github + [here](https://github.com/bytecodealliance/javy/releases/new). +4. A GitHub Action will trigger for `publish.yml` when a release is published + ([i.e. it doesn't run on + drafts](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#:~:text=created%2C%20edited%2C%20or%20deleted%20activity%20types%20for%20draft%20releases)), + creating the artifacts for downloading. + + diff --git a/docs/docs-contributing-testing-locally.md b/docs/docs-contributing-testing-locally.md new file mode 100644 index 00000000..08eef7dc --- /dev/null +++ b/docs/docs-contributing-testing-locally.md @@ -0,0 +1,19 @@ +## Testing locally + +1. Clone submodules + +``` +git submodules init +git submodules update +``` + +2. Install cargo hack + +``` +cargo +stable install cargo-hack --locked +``` + +3. Run tests, eg: +``` +CARGO_TARGET_WASM32_WASI_RUNNER="wasmtime --dir=." cargo hack wasi test --workspace --exclude=javy-cli --exclude=javy-config --each-feature -- --nocapture +``` diff --git a/docs/docs-contributing.md b/docs/docs-contributing.md new file mode 100644 index 00000000..3cf32568 --- /dev/null +++ b/docs/docs-contributing.md @@ -0,0 +1,70 @@ +# Contributing + +## Architecture + +See our [architecture documentation](./docs-contributing-architecture) for more +information about how the project is organized. + +## Adding additional JS APIs + +We will only add JS APIs or accept contributions that add JS APIs that are +potentially useful across multiple environments and do not invoke +non-[WASI](https://wasi.dev/) hostcalls. If you wish to add or use JS APIs that +do not meet these criteria, please use the `rquickjs` crate directly. We may +revisit how we support importing and exporting custom functionality from Javy +once [the Component Model](https://github.com/WebAssembly/component-model) has +stabilized. + +## Versioning for library crates + +The library crate, `javy`, uses the versioning system described in [Rust for +Rustaceans](https://rust-for-rustaceans.com/) in the _Unreleased Versions_ +section in the _Project Structure_ chapter. The underlying motivation is that +the version in the crate's `Cargo.toml` is important between releases to ensure +Cargo does not reuse a stale version if a project relies on a version of the +crate that has not yet been published to crates.io and the version required by +that project is updated to a version with new additive or breaking changes. + +### The system + +After publishing a release, immediately update the version number to the next +patch version with an `-alpha.1` suffix. The first time an additive change is +made, reset the patch version to `0` and increment the minor version and reset +the suffix to `-alpha.1`. When making additional additive changes, increment the +number in the suffix, for example `-alpha.2`. The first time a breaking change +is made, reset the patch version and minor version to `0` and increment the +major version and reset the suffix to `-alpha.1`. When making additional +breaking changes, increment the number in the suffix, for example `-alpha.2`. + +When releasing, remove the suffix and then publish. + +For example, let's say the last published version of `javy` is `2.0.0`, so the +current version in the Cargo.toml file is `2.0.0-alpha.1`. If you add a new +public function, you would change the version to `2.1.0-alpha.1`. This is +because adding a new public function is considered an additive change. After +merging those changes, if you add a new public function, you would change the +version to `2.1.0-alpha.2`. This is because adding another new function is an +additional additive change. Now if you were to make a function that was public, +private, you would change the version to `3.0.0-alpha.1`. This is because +removing a public function is considered a breaking change. After merging that +change, if you were to then add a new public function, then you would increment +the version to `3.0.0-alpha.2` because this is making an additional additive +change. It's not necessary to increment the minor version in this case because +version `3.0.0` has not been published yet so version `3.0.0` can contain +a mixture of additive and breaking changes from the last `2.x.x` version +published. + +## cargo vet + +We use [cargo vet](https://mozilla.github.io/cargo-vet/) to audit dependencies +for the project. If you need to change or add dependencies, please try to use +a dependency that has been audited by one one of the audits we import or is +published by one of the authors we trust (sunfishcode, dtolnay, Amanieu, +cuviper). This is preferable to adding new exemptions for the project. Do not +add new audits for crates that are not in this project. + +## Web platform tests (WPT) + +We run a subset of the web platform test suite during continuous integration. We +recommend reading our suite's [WPT readme](../wpt/README.md) for tips on how to +add tests to the suite and what to do when tests won't pass. diff --git a/docs/docs-using-dynamic-linking.md b/docs/docs-using-dynamic-linking.md new file mode 100644 index 00000000..cdcef821 --- /dev/null +++ b/docs/docs-using-dynamic-linking.md @@ -0,0 +1,40 @@ +# Dynamically linked modules + +An important use for Javy is for when you may want or need to generate much +smaller Wasm modules. Using the `-C dynamic` flag when invoking `javy build` will create +a dynamically linked module which will have a much smaller file size than +a statically linked module. Statically linked modules embed the JS engine inside +the module while dynamically linked modules rely on Wasm imports to provide the +JS engine. Dynamically linked modules have special requirements that statically +linked modules do not and will not execute in WebAssembly runtimes that do not +meet these requirements. + +To successfully instantiate and run a dynamically linked Javy module, the +execution environment must provide a `javy_quickjs_provider_v` namespace for +importing that links to the exports provided by the `javy_quickjs_provider.wasm` +module. Dynamically linked modules **cannot** be instantiated in environments +that do not provide this import. + +Dynamically linked Javy modules are tied to QuickJS since they use QuickJS's +bytecode representation. + + +#### Obtaining the provider module + +The `javy_quickjs_provider.wasm` module is available as an asset on the Javy +release you are using. + +It can also be obtained by running `javy emit-provider -o +` to write the module into ``. + +#### Creating and running a dynamically linked module througy the CLI + +Run: + +``` +$ echo 'console.log("hello world!");' > my_code.js +$ javy build -C dynamic -o my_code.wasm my_code.js +$ javy emit-provider -o provider.wasm +$ wasmtime run --preload javy_quickjs_provider_v3=provider.wasm my_code.wasm +hello world! +``` diff --git a/docs/docs-using-exports.md b/docs/docs-using-exports.md new file mode 100644 index 00000000..405346e3 --- /dev/null +++ b/docs/docs-using-exports.md @@ -0,0 +1,114 @@ +# Exporting JavaScript functions through WebAssembly + +To export exported JavaScript functions, you can pass a `.wit` file and `wit` world +when running `javy build`. + +Only ESM exports are supported (that is, Node.js/CommonJS exports are _not_ +supported). For each exported JavaScript function, Javy will add an additional +function export to the WebAssembly module. Exported functions with arguments and +generators are not supported. Return values will also be dropped and not +returned. The Wasm module generated is a core Wasm module, **not** a Wasm +component. + +An example looks like: + +`index.js`: +```javascript +export function foo() { + console.log("Hello from foo!"); +} + +console.log("Hello world!"); +``` + +`index.wit`: +``` +package local:main; + +world index-world { + export foo: func(); +} +``` + +Run: + +```bash +$ javy build index.js -C wit=index.wit -C wit-world=index-world -o index.wasm +$ wasmtime run --invoke foo index.wasm +Hello world! +Hello from foo! +``` + +The `wit` package name and `wit` world name do not matter as long as they are +present and syntactically correct (that is, it needs to be two names +separated by a `:`). The name of the `wit` world (that is, the value after `world` +and before `{`) must be passed via `-C wit-world` argument. The `-C wit-world` argument +identifies the `wit` world in the `wit` file for the Wasm module generated by `javy +build`. + +## Exports with multiple words + +Exported function names with multiple words have to written in kebab-case in the +`.wit` file (a restriction imposed by `wit`), they are exported from the Wasm +module as kebab-case to match the WIT, and Javy will match the WIT export to +a JavaScript export with the same name but in camel-case. + +`index.js`: +```javascript +export function fooBar() { + console.log("In foo-bar"); +} +``` + +`index.wit`: +``` +package local:main; + +world index { + export foo-bar: func(); +} +``` + +Run: + +```bash +$ javy build index.js -C wit=index.wit -C wit-world=index -o index.wasm +$ wasmtime run --invoke foo-bar index.wasm +In foo-bar +``` + +## Exporting a default function + +Exporting a function named `default` in the WIT world exports a function named +`default` on the Wasm module and corresponds to either an exported default +function or exported default arrow function in JS. + +`index.js`: +```javascript +export default function () { + console.log("In default"); +} +``` + +`index.wit`: +``` +package local:main; + +world index { + export default: func(); +} +``` + +In the terminal: +```bash +$ javy build index.js -C wit=index.wit -C wit-world=index -o index.wasm +$ wasmtime run --invoke default index.wasm +In default +``` + +You can also export a default function by writing: +```javascript +export default () => { + console.log("default"); +} +``` diff --git a/docs/docs-using-extending.md b/docs/docs-using-extending.md new file mode 100644 index 00000000..ac460a22 --- /dev/null +++ b/docs/docs-using-extending.md @@ -0,0 +1,28 @@ +# Extending + +If you want to use Javy for your own project, you may find that the existing +code is not sufficient since you may want to offer custom APIs or use different +branding for the CLI. The approach we'd recommend taking is to create your own +version of the `javy-cli` and `javy-core` crates (you could fork these if you +would like) and depend on the upstream version of the `javy` crate. You can add +your own implementations of custom JS APIs in your fork of the `javy-core` crate +or in a different crate that you depend on in your `javy-core` fork. If you find +that something is missing in the `javy` crate that you require to implement +something in your fork, we would appreciate it if you would open a GitHub issue +and consider making the change upstream instead of in your fork so all users of +the `javy` crate can benefit. + +See our documentation on [using complex data types in Wasm +functions](./contributing-complex-data-types.md) for how to support Wasm +functions that need to use byte arrays, strings, or structured data. + +For a visual representation of how we expect forks to consume our crates: + +```mermaid +flowchart TD + your-cli --> wasm + subgraph wasm[your_core.wasm] + your-core --> javy[upstream javy] + javy --> rquickjs + end +``` diff --git a/docs/docs-using-invoking.md b/docs/docs-using-invoking.md new file mode 100644 index 00000000..d64f81dd --- /dev/null +++ b/docs/docs-using-invoking.md @@ -0,0 +1,14 @@ +### Invoking modules programatically + +Javy-generated modules are by design WASI only and follow the [command +pattern](https://github.com/WebAssembly/WASI/blob/snapshot-01/design/application-abi.md#current-unstable-abi). + +Any input must be passed via `stdin` and any output will be placed in `stdout`. +This is especially important when invoking Javy modules from a custom embedding. + +In a runtime like Wasmtime, [wasmtime-wasi]( +https://docs.rs/wasmtime-wasi/latest/wasmtime_wasi/struct.WasiCtx.html#method.set_stdin) +can be used to set the input and retrieve the output. + +To embed Javy in a Node.js application see this +[example](./docs-using-nodejs.md). diff --git a/docs/docs-using-js-api-support.md b/docs/docs-using-js-api-support.md new file mode 100644 index 00000000..48c081cc --- /dev/null +++ b/docs/docs-using-js-api-support.md @@ -0,0 +1,32 @@ +# JavaScript API Support + +Javy by default supports ES2023, plus partial support for additional APIs. +NodeJS APIs are not supported. + +This document aims to give an overview of the additional APIs provided by Javy. + +In general the ultimate goal of Javy is to provide a [WinterCG Common +API](https://common-min-api.proposal.wintercg.org/#api-index) compatible +implementation, however, currently some APIs are not fully +compliant and therefore are provided under a custom `Javy` namespace or +explicitly marked as partially supported in the table below. + +## Compatibility table + +|API|Support|Comments| +|:-:|:-:|:-:| +|`JSON`|✅| Improved performace through SIMD JSON, when using the `-J simd-json-builtins` flag| +|`TexDecoder`|🚧| Partial support, not fully compliant| +|`TextEncoder`|🚧| Partial support, not fully compliant| +|`TextEncoder`|🚧| Partial support, not fully compliant| +|`console`|🚧| Partial support, `console.log` and `console.error`| + +Javy provides a custom `Javy` namespace, which includes the following +functionality: + +* `IO`: provides `readSync` and `writeSync`, analogous to [Node's `fs` + API](https://nodejs.org/api/fs.html). + +* `JSON`: provides `fromStdin()` and `toStdout()`. Which are helpers to read or + write from and to a file descriptor when working with `JSON`. + diff --git a/docs/nodejs-embedding.md b/docs/docs-using-nodejs.md similarity index 97% rename from docs/nodejs-embedding.md rename to docs/docs-using-nodejs.md index e4b59c8a..63c66d2d 100644 --- a/docs/nodejs-embedding.md +++ b/docs/docs-using-nodejs.md @@ -13,7 +13,7 @@ This example shows how to use a dynamically linked Javy compiled WASM module. We 1. The first step is to compile the `embedded.js` with Javy using dynamic linking: ```shell -javy compile embedded.js -d -o embedded.wasm +javy build -C dynamic -o embedded.wasm embedded.js ``` 2. Next emit the Javy provider ```shell @@ -149,7 +149,7 @@ async function runJavy(providerModule, embeddedModule, input) { wasi.getImportObject(), ); const instance = await WebAssembly.instantiate(embeddedModule, { - javy_quickjs_provider_v2: providerInstance.exports, + javy_quickjs_provider_v3: providerInstance.exports, }); // Javy provider is a WASI reactor see https://github.com/WebAssembly/WASI/blob/main/legacy/application-abi.md?plain=1 diff --git a/docs/docs-using-runtime-requirements.md b/docs/docs-using-runtime-requirements.md new file mode 100644 index 00000000..10d0d39a --- /dev/null +++ b/docs/docs-using-runtime-requirements.md @@ -0,0 +1,5 @@ +## Runtime requirements + +When running the official Javy binary on Linux, `glibc` 2.31 or greater must be +available. You may need to update the version of your operating system if you +are using an older version of `glibc`. diff --git a/docs/extending.md b/docs/extending.md deleted file mode 100644 index dd23e048..00000000 --- a/docs/extending.md +++ /dev/null @@ -1,16 +0,0 @@ -# Extending Javy - -If you want to use Javy for your own project, you may find that the existing code is not sufficient since you may want to offer custom APIs or use different branding for the CLI. The approach we'd recommend taking is to create your own version of the `javy-cli` and `javy-core` crates (you could fork these if you would like) and depend on the upstream version of the `javy` crate. You can add your own implementations of custom JS APIs in your fork of the `javy-core` crate or in a different crate that you depend on in your `javy-core` fork. If you find that something is missing in the `javy` crate that you require to implement something in your fork, we would appreciate it if you would open a GitHub issue and consider making the change upstream instead of in your fork so all users of the `javy` crate can benefit. - -See our documentation on [using complex data types in Wasm functions](complex-data-types-in-wasm-functions.md) for how to support Wasm functions that need to use byte arrays, strings, or structured data. - -For a visual representation of how we expect forks to consume our crates: - -```mermaid -flowchart TD - your-cli --> wasm - subgraph wasm[your_core.wasm] - your-core --> javy[upstream javy] - javy --> rquickjs - end -``` diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..e6dbec15 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,22 @@ +# Table of Contents + +## Usage + +* [Runtime requirements](./docs-using-runtime-requirements.md) +* [Extending](./docs-using-extending.md) +* [Exporting functions](./docs-using-exports.md) +* [Invoking modules](./docs-using-invoking.md) +* [JavaScript API Support](./docs-using-js-api-support.md) + +## Contributing + +We welcome feedback, bug reports and bug fixes. We're also happy to discuss +feature development but please discuss the features in an issue before +contributing. + +* [General guidelines](./docs-contributing.md) +* [Architecture](./docs-contributing-architecture.md) +* [Build requirements](./docs-contributing-building.md) +* [Development requirements](./docs-contributing-developing.md) +* [Running tests locally](./docs-contributing-testing-locally.md) +* [Releasing](./docs-contributing-releasing.md)