Skip to content

Commit 471eb11

Browse files
nilslicenilslice
authored
feat: add overview on generating bindings (#70)
1 parent 286160a commit 471eb11

File tree

2 files changed

+194
-48
lines changed

2 files changed

+194
-48
lines changed

‎README.md

Lines changed: 166 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22

33
[![crates.io](https://img.shields.io/crates/v/extism_pdk.svg)](https://crates.io/crates/extism-pdk)
44

5-
This library can be used to write [Extism Plug-ins](https://extism.org/docs/concepts/plug-in) in Rust.
5+
This library can be used to write
6+
[Extism Plug-ins](https://extism.org/docs/concepts/plug-in) in Rust.
67

78
## Install
89

@@ -18,7 +19,9 @@ Add the library from [crates.io](https://crates.io/crates/extism-pdk).
1819
cargo add extism-pdk
1920
```
2021

21-
Change your `Cargo.toml` to set the crate-type to `cdylib` (this instructs the compiler to produce a dynamic library, which for our target will be a Wasm binary):
22+
Change your `Cargo.toml` to set the crate-type to `cdylib` (this instructs the
23+
compiler to produce a dynamic library, which for our target will be a Wasm
24+
binary):
2225

2326
```toml
2427
[lib]
@@ -27,7 +30,9 @@ crate_type = ["cdylib"]
2730

2831
### Rustup and wasm32-unknown-unknown installation
2932

30-
Our example below will use the `wasm32-unknown-unknown` target. If this is not installed you will need to do so before this example will build. The easiest way to do this is use [`rustup`](https://rustup.rs/).
33+
Our example below will use the `wasm32-unknown-unknown` target. If this is not
34+
installed you will need to do so before this example will build. The easiest way
35+
to do this is use [`rustup`](https://rustup.rs/).
3136

3237
```bash
3338
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
@@ -39,11 +44,15 @@ Once `rustup` is installed, add the `wasm32-unknown-unknown` target:
3944
rustup target add wasm32-unknown-unknown
4045
```
4146

42-
4347
## Getting Started
4448

45-
The goal of writing an [Extism plug-in](https://extism.org/docs/concepts/plug-in) is to compile your Rust code to a Wasm module with exported functions that the host application can invoke. The first thing you should understand is creating an export. Let's write a simple program that exports a `greet` function which will take a name as a string and return a greeting string. For this, we use the `#[plugin_fn]` macro on our exported function:
46-
49+
The goal of writing an
50+
[Extism plug-in](https://extism.org/docs/concepts/plug-in) is to compile your
51+
Rust code to a Wasm module with exported functions that the host application can
52+
invoke. The first thing you should understand is creating an export. Let's write
53+
a simple program that exports a `greet` function which will take a name as a
54+
string and return a greeting string. For this, we use the `#[plugin_fn]` macro
55+
on our exported function:
4756

4857
```rust
4958
use extism_pdk::*;
@@ -54,39 +63,54 @@ pub fn greet(name: String) -> FnResult<String> {
5463
}
5564
```
5665

57-
Since we don't need any system access for this, we can compile this to the lightweight `wasm32-unknown-unknown` target instead of using the `wasm32-wasi` target:
66+
Since we don't need any system access for this, we can compile this to the
67+
lightweight `wasm32-unknown-unknown` target instead of using the `wasm32-wasi`
68+
target:
5869

5970
```bash
6071
cargo build --target wasm32-unknown-unknown
6172
```
6273

6374
> **Note**: You can also put a default target in `.cargo/config.toml`:
75+
6476
```toml
6577
[build]
6678
target = "wasm32-unknown-unknown"
6779
```
6880

69-
This will put your compiled wasm in `target/wasm32-unknown-unknown/debug`.
70-
We can now test it using the [Extism CLI](https://github.com/extism/cli)'s `run`
81+
This will put your compiled wasm in `target/wasm32-unknown-unknown/debug`. We
82+
can now test it using the [Extism CLI](https://github.com/extism/cli)'s `run`
7183
command:
7284

7385
```bash
7486
extism call target/wasm32-unknown-unknown/debug/my_plugin.wasm greet --input "Benjamin"
7587
# => Hello, Benjamin!
7688
```
7789

78-
> **Note**: We also have a web-based, plug-in tester called the [Extism Playground](https://playground.extism.org/)
90+
> **Note**: We also have a web-based, plug-in tester called the
91+
> [Extism Playground](https://playground.extism.org/)
7992
8093
### More About Exports
8194

82-
Adding the [plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html) macro to your function does a couple things. It exposes your function as an export and it handles some of the lower level ABI details that allow you to declare your Wasm function as if it were a normal Rust function. Here are a few examples of exports you can define.
95+
Adding the
96+
[plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html)
97+
macro to your function does a couple things. It exposes your function as an
98+
export and it handles some of the lower level ABI details that allow you to
99+
declare your Wasm function as if it were a normal Rust function. Here are a few
100+
examples of exports you can define.
83101

84102
### Primitive Types
85103

86-
A common thing you may want to do is pass some primitive Rust data back and forth.
87-
The [plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html) macro can map these types for you:
104+
A common thing you may want to do is pass some primitive Rust data back and
105+
forth. The
106+
[plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html)
107+
macro can map these types for you:
88108

89-
> **Note**: The [plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html) macro uses the [convert crate](https://github.com/extism/extism/tree/main/convert) to automatically convert and pass types across the guest / host boundary.
109+
> **Note**: The
110+
> [plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html)
111+
> macro uses the
112+
> [convert crate](https://github.com/extism/extism/tree/main/convert) to
113+
> automatically convert and pass types across the guest / host boundary.
90114
91115
```rust
92116
// f32 and f64
@@ -118,9 +142,10 @@ pub fn process_string(input: String) -> FnResult<String> {
118142

119143
### Json
120144

121-
We provide a [Json](https://docs.rs/extism-pdk/latest/extism_pdk/struct.Json.html) type that allows you to pass structs
122-
that implement serde::Deserialize as parameters and serde::Serialize
123-
as returns:
145+
We provide a
146+
[Json](https://docs.rs/extism-pdk/latest/extism_pdk/struct.Json.html) type that
147+
allows you to pass structs that implement serde::Deserialize as parameters and
148+
serde::Serialize as returns:
124149

125150
```rust
126151
#[derive(serde::Deserialize)]
@@ -165,8 +190,9 @@ pub fn add(add: Add) -> FnResult<Sum> {
165190

166191
### Raw Export Interface
167192

168-
[plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html) is a nice macro abstraction but there may be times where you want more control. You can code directly to the raw ABI interface of export functions.
169-
193+
[plugin_fn](https://docs.rs/extism-pdk/latest/extism_pdk/attr.plugin_fn.html) is
194+
a nice macro abstraction but there may be times where you want more control. You
195+
can code directly to the raw ABI interface of export functions.
170196

171197
```rust
172198
#[no_mangle]
@@ -181,7 +207,8 @@ pub unsafe extern "C" fn greet() -> i32 {
181207
## Configs
182208

183209
Configs are key-value pairs that can be passed in by the host when creating a
184-
plug-in. These can be useful to statically configure the plug-in with some data that exists across every function call. Here is a trivial example:
210+
plug-in. These can be useful to statically configure the plug-in with some data
211+
that exists across every function call. Here is a trivial example:
185212

186213
```rust
187214
#[plugin_fn]
@@ -191,7 +218,8 @@ pub fn greet() -> FnResult<String> {
191218
}
192219
```
193220

194-
To test it, the [Extism CLI](https://github.com/extism/cli) has a `--config` option that lets you pass in `key=value` pairs:
221+
To test it, the [Extism CLI](https://github.com/extism/cli) has a `--config`
222+
option that lets you pass in `key=value` pairs:
195223

196224
```bash
197225
extism call my_plugin.wasm greet --config user=Benjamin
@@ -202,7 +230,10 @@ extism call my_plugin.wasm greet --config user=Benjamin
202230

203231
Variables are another key-value mechanism but it's a mutable data store that
204232
will persist across function calls. These variables will persist as long as the
205-
host has loaded and not freed the plug-in. You can use [var::get](https://docs.rs/extism-pdk/latest/extism_pdk/var/fn.get.html) and [var::set](https://docs.rs/extism-pdk/latest/extism_pdk/var/fn.set.html) to manipulate them.
233+
host has loaded and not freed the plug-in. You can use
234+
[var::get](https://docs.rs/extism-pdk/latest/extism_pdk/var/fn.get.html) and
235+
[var::set](https://docs.rs/extism-pdk/latest/extism_pdk/var/fn.set.html) to
236+
manipulate them.
206237

207238
```rust
208239
#[plugin_fn]
@@ -216,7 +247,12 @@ pub fn count() -> FnResult<i64> {
216247

217248
## Logging
218249

219-
Because Wasm modules by default do not have access to the system, printing to stdout won't work (unless you use WASI). Extism provides some simple logging macros that allow you to use the host application to log without having to give the plug-in permission to make syscalls. The primary one is [log!](https://docs.rs/extism-pdk/latest/extism_pdk/macro.log.html) but we also have some convenience macros named by log level:
250+
Because Wasm modules by default do not have access to the system, printing to
251+
stdout won't work (unless you use WASI). Extism provides some simple logging
252+
macros that allow you to use the host application to log without having to give
253+
the plug-in permission to make syscalls. The primary one is
254+
[log!](https://docs.rs/extism-pdk/latest/extism_pdk/macro.log.html) but we also
255+
have some convenience macros named by log level:
220256

221257
```rust
222258
#[plugin_fn]
@@ -243,13 +279,17 @@ extism call my_plugin.wasm log_stuff --log-level=info
243279
2023/09/30 11:52:17 An error!
244280
```
245281

246-
> *Note*: From the CLI you need to pass a level with `--log-level`. If you are running the plug-in in your own host using one of our SDKs, you need to make sure that you call `set_log_file` to `"stdout"` or some file location.
282+
> _Note_: From the CLI you need to pass a level with `--log-level`. If you are
283+
> running the plug-in in your own host using one of our SDKs, you need to make
284+
> sure that you call `set_log_file` to `"stdout"` or some file location.
247285
248286
## HTTP
249287

250288
Sometimes it is useful to let a plug-in make HTTP calls.
251289

252-
> **Note**: See [HttpRequest](https://docs.rs/extism-pdk/latest/extism_pdk/struct.HttpRequest.html) docs for more info on the request and response types:
290+
> **Note**: See
291+
> [HttpRequest](https://docs.rs/extism-pdk/latest/extism_pdk/struct.HttpRequest.html)
292+
> docs for more info on the request and response types:
253293
254294
```rust
255295
#[plugin_fn]
@@ -261,17 +301,22 @@ pub fn http_get(Json(req): Json<HttpRequest>) -> FnResult<Vec<u8>> {
261301

262302
## Imports (Host Functions)
263303

264-
Like any other code module, Wasm not only let's you export functions to the outside world, you can
265-
import them too. Host Functions allow a plug-in to import functions defined in the host. For example,
266-
if you host application is written in Python, it can pass a Python function down to your Rust plug-in
267-
where you can invoke it.
304+
Like any other code module, Wasm not only let's you export functions to the
305+
outside world, you can import them too. Host Functions allow a plug-in to import
306+
functions defined in the host. For example, if you host application is written
307+
in Python, it can pass a Python function down to your Rust plug-in where you can
308+
invoke it.
268309

269-
This topic can get fairly complicated and we have not yet fully abstracted the Wasm knowledge you need
270-
to do this correctly. So we recommend reading out [concept doc on Host Functions](https://extism.org/docs/concepts/host-functions) before you get started.
310+
This topic can get fairly complicated and we have not yet fully abstracted the
311+
Wasm knowledge you need to do this correctly. So we recommend reading out
312+
[concept doc on Host Functions](https://extism.org/docs/concepts/host-functions)
313+
before you get started.
271314

272315
### A Simple Example
273316

274-
Host functions have a similar interface as exports. You just need to declare them as `extern` on the top of your `lib.rs`. You only declare the interface as it is the host's responsibility to provide the implementation:
317+
Host functions have a similar interface as exports. You just need to declare
318+
them as `extern` on the top of your `lib.rs`. You only declare the interface as
319+
it is the host's responsibility to provide the implementation:
275320

276321
```rust
277322
#[host_fn]
@@ -280,20 +325,23 @@ extern "ExtismHost" {
280325
}
281326
```
282327

283-
> **Note**: Under the hood this macro turns this into an interface that passes a pointer as an argument
284-
> and a pointer as a return. If you want to pass raw, dereferenced wasm values see the raw interface documentation below.
328+
> **Note**: Under the hood this macro turns this into an interface that passes a
329+
> pointer as an argument and a pointer as a return. If you want to pass raw,
330+
> dereferenced wasm values see the raw interface documentation below.
285331
286-
To declare a host function in a specific namespace, pass the module name to the `host_fn` macro:
332+
To declare a host function in a specific namespace, pass the module name to the
333+
`host_fn` macro:
287334

288335
```rust
289336
#[host_fn("extism:host/user")]
290337
```
291338

292-
> **Note**: The types we accept here are the same as the exports as the interface also uses the [convert crate](https://docs.rs/extism-convert/latest/extism_convert/).
293-
294-
To call this function, we must use the `unsafe` keyword. Also note that it automatically wraps the
295-
function return with a Result in case the call fails.
339+
> **Note**: The types we accept here are the same as the exports as the
340+
> interface also uses the
341+
> [convert crate](https://docs.rs/extism-convert/latest/extism_convert/).
296342
343+
To call this function, we must use the `unsafe` keyword. Also note that it
344+
automatically wraps the function return with a Result in case the call fails.
297345

298346
```rust
299347
#[plugin_fn]
@@ -305,8 +353,10 @@ pub fn hello_from_python() -> FnResult<String> {
305353

306354
### Testing it out
307355

308-
We can't really test this from the Extism CLI as something must provide the implementation. So let's
309-
write out the Python side here. Check out the [docs for Host SDKs](https://extism.org/docs/concepts/host-sdk) to implement a host function in a language of your choice.
356+
We can't really test this from the Extism CLI as something must provide the
357+
implementation. So let's write out the Python side here. Check out the
358+
[docs for Host SDKs](https://extism.org/docs/concepts/host-sdk) to implement a
359+
host function in a language of your choice.
310360

311361
```python
312362
from extism import host_fn, Plugin
@@ -324,7 +374,7 @@ def a_python_func(input: str) -> str:
324374
```
325375

326376
Now when we load the plug-in we pass the host function:
327-
377+
328378
```python
329379
manifest = {"wasm": [{"path": "/path/to/plugin.wasm"}]}
330380
plugin = Plugin(manifest, functions=[a_python_func], wasi=True)
@@ -340,17 +390,85 @@ python3 app.py
340390

341391
## Raw Import Interface
342392

343-
Like exports, with imports we do some magic to turn the parameters and returns into pointers for you.
344-
In some rare situations, you might wish to pass raw wasm values to the host (not pointers).
345-
If you do, you need to drop down into a raw interface.
346-
E.g, imagine an interface that sums two i64s
393+
Like exports, with imports we do some magic to turn the parameters and returns
394+
into pointers for you. In some rare situations, you might wish to pass raw wasm
395+
values to the host (not pointers). If you do, you need to drop down into a raw
396+
interface. E.g, imagine an interface that sums two i64s
347397

348398
```rust
349399
extern "C" {
350400
fn sum(a: i64, b: i64) -> i64;
351401
}
352402
```
353403

354-
### Reach Out!
404+
## Generating Bindings
405+
406+
It's often very useful to define a schema to describe the function signatures
407+
and types you want to use between Extism SDK and PDK languages.
408+
409+
[XTP Bindgen](https://github.com/dylibso/xtp-bindgen) is an open source
410+
framework to generate PDK bindings for Extism plug-ins. It's used by the
411+
[XTP Platform](https://www.getxtp.com/), but can be used outside of the platform
412+
to define any Extism compatible plug-in system.
413+
414+
### 1. Install the `xtp` CLI.
415+
416+
See installation instructions
417+
[here](https://docs.xtp.dylibso.com/docs/cli#installation).
418+
419+
### 2. Create a schema using our OpenAPI-inspired IDL:
420+
421+
```yaml
422+
version: v1-draft
423+
exports:
424+
CountVowels:
425+
input:
426+
type: string
427+
contentType: text/plain; charset=utf-8
428+
output:
429+
$ref: "#/components/schemas/VowelReport"
430+
contentType: application/json
431+
# components.schemas defined in example-schema.yaml...
432+
```
433+
434+
> See an example in [example-schema.yaml](./example-schema.yaml), or a full
435+
> "kitchen sink" example on
436+
> [the docs page](https://docs.xtp.dylibso.com/docs/concepts/xtp-schema/).
437+
438+
### 3. Generate bindings to use from your plugins:
439+
440+
```
441+
xtp plugin init --schema-file ./example-schema.yaml
442+
1. TypeScript
443+
2. Go
444+
> 3. Rust
445+
4. Python
446+
5. C#
447+
6. Zig
448+
7. C++
449+
8. GitHub Template
450+
9. Local Template
451+
```
452+
453+
This will create an entire boilerplate plugin project for you to get started
454+
with:
455+
456+
```rust
457+
// returns VowelReport (The result of counting vowels on the Vowels input.)
458+
pub(crate) fn count_vowels(input: String ) -> Result<VowelReport, Error> {
459+
todo!("Implement count_vowels")
460+
}
461+
```
462+
463+
Implement the empty function(s), and run `xtp plugin build` to compile your
464+
plugin.
465+
466+
> For more information about XTP Bindgen, see the
467+
> [dylibso/xtp-bindgen](https://github.com/dylibso/xtp-bindgen) repository and
468+
> the official
469+
> [XTP Schema documentation](https://docs.xtp.dylibso.com/docs/concepts/xtp-schema).
470+
471+
## Reach Out!
355472

356-
Have a question or just want to drop in and say hi? [Hop on the Discord](https://extism.org/discord)!
473+
Have a question or just want to drop in and say hi?
474+
[Hop on the Discord](https://extism.org/discord)!

0 commit comments

Comments
 (0)