feat: add binary type support for host functions by simongdavies · Pull Request #40 · hyperlight-dev/hyperlight-js

simongdavies · 2026-03-06T00:20:28Z

Adds Uint8Array/Buffer support for host function arguments and returns.

Architecture:

New hyperlight-js-common crate: shared wire-format constants, binary sidecar encode/decode, FnReturn enum, DecodeError type
Guest extracts Uint8Array from QuickJS VM into binary sidecar
Host dispatches via register() (typed serde) or register_js() (JS bridge)
NAPI layer creates native Node.js Buffers via C API (no base64)
Tagged return format (0x00=JSON, 0x01=binary, 0x02=JSON with binary sidecar) for return path — supports returning binary data embedded in structured JSON responses, not just standalone buffers

Key changes:

Single CallHostJsFunction entry point (removed legacy JSON-only path)
Native Buffer marshalling in NAPI (JsArg/JsReturn types)
Depth limits on all recursive JSON tree traversals (MAX_JSON_DEPTH=64 defined once in hyperlight-js-common, used by both guest and host)
Trailing data rejection in sidecar decoder
Typed register() rejects binary args with clear error message
Comprehensive test coverage (Rust unit + integration + JS vitest)
Updated README with Binary Data section and wire protocol docs
Updated CI publish order for new common crate

Closes #38

…n path QuickJS stores JSON-parsed numbers as doubles internally, so value_to_json_with_binaries was emitting 42.0 instead of 42 for whole numbers. This caused serde deserialization failures on the host side when typed host functions expected i32/i64 args. Fix: when a float has no fractional part and fits in i64, emit it as an integer to match JSON.stringify behaviour. Also adapts user_module_can_import_host_function test to use the new typed register() API (register_raw was removed in PR hyperlight-dev#40).

dblnz

This looks good to me, although I am not very familiar to the js-host-api side of things.

One thought I have is, it seems to me that this serialization/deserialization to/from guest would benefit from something similar to what we do in hyperlight-wasm with the WIT world bindings generation, right?
Practically this boils down to having shared types definitions between host and guest downstream.

docs/release.md

src/hyperlight-js-common/src/lib.rs

src/hyperlight-js-runtime/src/host_fn.rs

simongdavies · 2026-03-19T21:43:43Z

This looks good to me, although I am not very familiar to the js-host-api side of things.

One thought I have is, it seems to me that this serialization/deserialization to/from guest would benefit from something similar to what we do in hyperlight-wasm with the WIT world bindings generation, right? Practically this boils down to having shared types definitions between host and guest downstream.

Yes I think so , if you have an idea of what this could/should look like maybe you could open an issue and describe it? Thanks !

Copilot

Pull request overview

This PR adds first-class binary data support (Uint8Array/Node.js Buffer) for host function calls across the guest runtime, Rust host dispatch, and the Node.js NAPI bridge by introducing a shared binary sidecar wire format and tagged return encoding.

Changes:

Introduces hyperlight-js-common to centralize wire-format constants, sidecar encode/decode, return tagging, and shared JSON depth limits.
Switches host function invocation to a unified binary-capable entry point (CallHostJsFunction) and adds FnReturn to support JSON, binary, and JSON-with-binaries returns.
Updates NAPI marshalling to create/detect native Node.js Buffer values (no base64), plus adds extensive Rust + JS test coverage and documentation updates.

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/js-host-api/tests/host-functions.test.js	Adds JS integration tests for Buffer/Uint8Array args and returns (including nested and empty cases).
src/js-host-api/src/lib.rs	Implements native Buffer marshalling in NAPI and wires host module registration through binary-capable `register_js`.
src/js-host-api/lib.js	Wraps `HostModule.register` to normalize top-level Buffer/Uint8Array returns for NAPI detection.
src/js-host-api/README.md	Updates docs for new wire protocol and adds a Binary Data section.
src/js-host-api/Cargo.toml	Adds dependency on `hyperlight-js-common`.
src/hyperlight-js/tests/host_functions.rs	Adds end-to-end Rust tests covering binary args/returns and nested binary (JsonWithBinaries).
src/hyperlight-js/src/sandbox/proto_js_sandbox.rs	Updates the host entry point registration to accept args JSON + binary sidecar and return tagged bytes.
src/hyperlight-js/src/sandbox/host_fn.rs	Introduces unified host-function dispatch (`Typed` vs `JsBridge`) and enforces typed rejection of binary args.
src/hyperlight-js/src/lib.rs	Re-exports `FnReturn` for the NAPI bridge signature.
src/hyperlight-js/Cargo.toml	Adds dependency on `hyperlight-js-common`.
src/hyperlight-js-runtime/src/main/hyperlight.rs	Switches guest registration to use binary-capable host function registration and improves error message sizing.
src/hyperlight-js-runtime/src/lib.rs	Renames/reshapes host function registration APIs and adds `register_binary_host_function`.
src/hyperlight-js-runtime/src/host_fn.rs	Adds guest-side extraction of `Uint8Array` into sidecar, tagged return decoding, and JSON depth limiting.
src/hyperlight-js-runtime/Cargo.toml	Adds dependency on `hyperlight-js-common`.
src/hyperlight-js-common/src/lib.rs	New shared crate defining sidecar framing, tagged return encoding/decoding, and `MAX_JSON_DEPTH`.
src/hyperlight-js-common/Cargo.toml	New crate manifest for `hyperlight-js-common`.
docs/release.md	Documents crates.io publish order for the new dependency chain.
Cargo.toml	Adds `hyperlight-js-common` to the workspace and workspace dependencies.
Cargo.lock	Updates lockfile for new crate and dependency graph changes.
.github/workflows/CreateRelease.yml	Publishes `hyperlight-js-common` before dependent crates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/hyperlight-js-runtime/src/host_fn.rs

src/js-host-api/src/lib.rs

src/js-host-api/lib.js

src/js-host-api/README.md

src/hyperlight-js-runtime/src/host_fn.rs

Adds Uint8Array/Buffer support for host function arguments and returns. Architecture: - New hyperlight-js-common crate: shared wire-format constants, binary sidecar encode/decode, FnReturn enum, DecodeError type - Guest extracts Uint8Array from QuickJS VM into binary sidecar - Host dispatches via register() (typed serde) or register_js() (JS bridge) - NAPI layer creates native Node.js Buffers via C API (no base64) - Tagged return format (0x00=JSON, 0x01=binary) for return path Key changes: - Single CallHostJsFunction entry point (removed legacy JSON-only path) - Native Buffer marshalling in NAPI (JsArg/JsReturn types) - Depth limits on all recursive JSON tree traversals (MAX_JSON_DEPTH=64) - Trailing data rejection in sidecar decoder - Typed register() rejects binary args with clear error message - Comprehensive test coverage (Rust unit + integration + JS vitest) - Updated README with Binary Data section and wire protocol docs - Updated CI publish order for new common crate Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/hyperlight-js/src/sandbox/host_fn.rs

src/js-host-api/README.md

src/js-host-api/src/lib.rs

src/hyperlight-js-runtime/src/lib.rs

src/hyperlight-js-runtime/src/host_fn.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js-host-api/src/lib.rs

src/hyperlight-js/src/sandbox/proto_js_sandbox.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js-host-api/src/lib.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

simongdavies added the kind/enhancement New feature or improvement label Mar 6, 2026

simongdavies force-pushed the allow-binary-types-for-host-funcs branch 4 times, most recently from db7d6ed to 4ba03cd Compare March 9, 2026 12:14

simongdavies force-pushed the allow-binary-types-for-host-funcs branch 2 times, most recently from 9e70be3 to bfc51b5 Compare March 10, 2026 14:31

dblnz reviewed Mar 11, 2026

View reviewed changes

docs/release.md Outdated Show resolved Hide resolved

src/hyperlight-js-common/src/lib.rs Outdated Show resolved Hide resolved

src/hyperlight-js-common/src/lib.rs Outdated Show resolved Hide resolved

src/hyperlight-js-runtime/src/host_fn.rs Outdated Show resolved Hide resolved

simongdavies force-pushed the allow-binary-types-for-host-funcs branch from b4e41a8 to 0ade7b4 Compare March 17, 2026 18:37

simongdavies force-pushed the allow-binary-types-for-host-funcs branch from 0ade7b4 to bc413f8 Compare March 19, 2026 22:06

simongdavies force-pushed the allow-binary-types-for-host-funcs branch from bc413f8 to 32c2a72 Compare March 27, 2026 14:14

simongdavies requested a review from Copilot March 27, 2026 14:16

Copilot started reviewing on behalf of simongdavies March 27, 2026 14:17 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

simongdavies force-pushed the allow-binary-types-for-host-funcs branch from 32c2a72 to 4ac2aed Compare March 27, 2026 15:21

simongdavies requested a review from Copilot March 27, 2026 15:21

Copilot started reviewing on behalf of simongdavies March 27, 2026 15:21 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Apply suggestions from code review

4b7e31d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

simongdavies force-pushed the allow-binary-types-for-host-funcs branch from 0f3f5b2 to 4b7e31d Compare March 27, 2026 15:51

simongdavies requested a review from Copilot March 27, 2026 16:12

Copilot started reviewing on behalf of simongdavies March 27, 2026 16:13 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

src/js-host-api/src/lib.rs Show resolved Hide resolved

src/hyperlight-js/src/sandbox/proto_js_sandbox.rs Outdated Show resolved Hide resolved

Apply suggestions from code review

96dd47a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

simongdavies force-pushed the allow-binary-types-for-host-funcs branch from b406caf to 96dd47a Compare March 27, 2026 19:14

simongdavies requested a review from Copilot March 27, 2026 19:34

Copilot started reviewing on behalf of simongdavies March 27, 2026 19:35 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

src/js-host-api/src/lib.rs Outdated Show resolved Hide resolved

Update src/js-host-api/src/lib.rs

a74cf50

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Conversation

simongdavies commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dblnz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simongdavies commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simongdavies commented Mar 6, 2026 •

edited

Loading

dblnz left a comment •

edited

Loading