Adding WASM support to Knative Functions

[aslom]

What would be the simplest way to enable running WASM with Knative? Enable with different WASM runtimes?

[No9]

Hey @aslom I'm not sure I know the "simplest" way but I'll share my work so far:

At an abstract level I see two approaches to running WASM in knative.

1. App Hosted - Provide a base hosting application that runs .wasm file provided by the developer. Use kn func and the build systems to bundle the .wasm with the host for deployment in a standard linux container.

2. OCI Hosted - Target WASI runtimes that provide network capabilities and OCI integration and bundle the wasm code and into a Scratch container ready for deployment.

Here is a matrix that compares the two approaches. It's not exhaustive but what I have been considering.

I have ordered them based on conversations I am having with 3rd Parties but other use cases may give a different priority.

Feature	App Hosted	OCI Hosted
Provides the smallest container size	🔴	🟢
Container portable across architectures	🔴	🟢
Consistent developer experience with kn func	🟡	🟢
Simple extensibility model	🟡	🟢
Language Support	🟢	🟡
Smaller CVE count for image	🔴	🟢
Requires Additional K8s configuration	🟢	🔴
Possible to run on Public Cloud xKS managed services	🟢	🟢
WASI standards compliant	🟢	🔴
Provides the fastest startup time	🔴	🟢
Utilises the sandbox	🟢	🔴
Operationally Consistent	🟢	🟡
Lowest Maintenance for kn func	🔴	🟢

Given the priorities I have I looked at the OCI hosted option using WASMedge specifically in the following area:

configuring a knative environment

WASMEdge provides guidance on using the runtime with Kubernetes and knative

https://wasmedge.org/book/en/use_cases/kubernetes.html

https://wasmedge.org/book/en/use_cases/kubernetes/kubernetes/knative.html

I've taken that guidance and packaged it into a deployment system for IKS and OpenShift to validate that it can be performed on public cloud providers with both Knative and K8s workloads. I have a bit of experience with providing host configuration on public cloud and the pattern should be repeatable for most other cloud providers.

https://github.com/uirlis/wasi-crun-deployer

code sample

To validate the deployment I also created a running example of the knative docs rust code mapped to WASMEdge OCI approach.
Original Code https://github.com/knative/docs/tree/main/code-samples/community/serving/helloworld-rust

WASMEdge Example https://github.com/uirlis/helloworld-rust-wasi

template

I have started to look at what would be required to create a kn func template project for both serving and eventing scenarios based on the templates

Original Code https://github.com/knative/func/tree/1a6c42f01baff54f68a92a5fd59ed8f339837338/templates/rust

WASMEdge Example https://github.com/uirlis/func-rust-wasi

The majority of the code is in place however a build system has not been created but I would expect that to be achievable.

cloud events

To support the template cloudevents project I have opened a PR to land hyper-wasi support into the rust cloud events sdk cloudevents/sdk-rust#202

other observations:

• Given the status of standards in WASI and WASMEdges current language support it's likely that an App Hosted version will be required for certain langauages.

• Current network support in WASMEdge seems to be limited to JS, Rust and C https://wasmedge.org/book/en/write_wasm.html

• Standard wasi-sockets seems to be some time away. status is available here - https://github.com/WebAssembly/wasi-sockets

• Garbage Collection may also be a blocker for wide spread language support.

• Observability in terms of runtime tracing and core dumps still needs to be understood.

• TLS support from the runtime is not available.

• Test runners seem to be a little flaky and could cause noise in the func project

• The app hosted option to merge code with a base server would have to be well considered as it wasn't as smooth as it could be when it was tried it over in appsody although I haven't considered it deeply in this specific scenario so I'd be delighted to be wrong on this.

• crun supports also supports wasmtime, wasmer I haven't performed any investigations into these runtimes as they don't seem to have network support available. But I do wonder if it's feasible to provide the app hosted approach but at the OCI level.
  [Edit] It would seem as though it might be possible at the CRI level https://github.com/containerd/runwasi

next items

• How much of a performance improvement is there running OCI Hosts over Traditional Containers?

  I plan to look at running benchmarks with https://github.com/IBM/knative-quarkus-bench and I will update this discussion as I work on it.

• Investigate requirements for modifying the build packs so the templates can work end to end

• Understand the available options for observability

Hopefully this is useful information and if the WASMEdge samples or templates are of interest I'd be happy to PR them.

[codefromthecrypt]

I'm a little surprised that enabling wasm in a FaaS implies that wasm should control network sockets. Maybe I'm misunderstanding, but I think usually a FaaS does that part on behalf of the function, which makes the function impls a lot simpler. Specific to wasm, if network control is put out of scope it is a lot more portable, just like any other wasm FaaS. Am I missing something?

[aslom]

I'm a little surprised that enabling wasm in a FaaS implies that wasm should control network sockets. Maybe I'm misunderstanding, but I think usually a FaaS does that part on behalf of the function, which makes the function impls a lot simpler. Specific to wasm, if network control is put out of scope it is a lot more portable, just like any other wasm FaaS. Am I missing something?

I think for knative/func networking (and other WASM runtime) would be handled by a buildpack - it seems we may need more than one for different WASM runtimes and languages supported?

[aslom]

@No9 thank for detailed description of your experiments. Based on your experience what do you think is the simplest way to enable running WASM with Knative using buildpacks for knative/func?

[matejvasek]

/cc @cardil

[No9]

Hey @aslom
I have an initial iteration of a buildpack available here
https://github.com/knawd/buildpack
It's bash based and uses a custom stack to keep the run image down to a minimum.
I haven't optimized the layering and I am not happy with the launch.toml but the main bits are in place so it would be great to get some feedback and if it's of interest understand how we incubate it in to func/knative.
I also haven't ran this "end to end" to understand the DX yet from a create/deploy perspective so there are likely some rough edges.
[Edit] Once I have ran it end to end I'll be able to answer your question more completely. I'm on compassionate leave from today but should get a chance to look at it next week.

@codefromthecrypt Love this line of questioning.
My initial work is based on the current architecture and developer flow of kn which, as @aslom mentioned, is container and buildpack driven.
Hence looking at the App Hosted vs OCI hosted and I agree with him that we are probably going to need different approaches for networked and non-network capable runtimes/languages.

That said we should look at the likes of runwasi and krustlet to understand how kn func could provide tooling in that space. The challenge there is that it's likely to require invasive changes upstream which the current build pack approach doesn't require.

[No9]

Hey @aslom Update on the E2E Scenario
The build pack now generates a WASI container that can be executed in a crun podman by over-riding the launcher.
e.g.

git clone https://github.com/knawd/buildpack

cd buildpack 

pack build helloworld-rust-wasi --path ./apps/rust/helloworld-rust-wasi --buildpack ./buildpacks/rust \
--builder  quay.io/knawd/builder:rust --docker-host=inherit -v --trust-builder

podman run --rm --annotation module.wasm.image/variant=compat-smart -p 8080:8080 \
--entrypoint /layers/knawd_rust/wasi/hellorust.wasm helloworld-rust-wasi

Here is a list of outstanding issues to get a WASMEdge enabled crun integrated with kn func

1. Blocker - Custom builders are blocked in func so I can't provide an end to end that I can share easily. It would be good if @lance has an update on this comment Provide a way to specify a lifecycle or remove the hardcoded value during build #1379 (comment)

2. Currently the wasi project template follows the embedded rust template project structure. Both of these templates fail with

   Building function image
   Error: the 'rust' runtime defines no default 's2i' builder image
   

   This is due to Unable to build pack images that don't also have s2i builders when s2i is the default #1492 although a workaround of --builder pack is available.

3. As the .wasm file is hosted in crun The launcher layer isn't required and should not be added to the final image but it doesn't seem to be possible to override.

4. I can't get the launch.toml to point to the .wasm file so the --entrypoint always needs to be overridden.

5. A way to add annotations such as module.wasm.image/variant=compat-smart to the generated image so that podman run is less verbose or maybe enhancing kn func run maybe a solution.

6. The container carries the SBOM and the Launcher which means the image size is ~1Gb instead of 4Mb This impairs one of the key benefits of using OCI WASI.

7. A mechanism for deploying the custom crun runtime from kn may need to be built.

It should be noted that the majority of these issues are caused by adopting the OCI approach outlined my first post above.
Depending on the appetite to change some of the issues it might be simpler to go for the hosted app approach in kn func and let other dev tooling target the OCI approach?

[codefromthecrypt]

I'm surprised a wasi base image is so massive (1GB). wazero in docker is about 5-7MB total I think (from SCRATCH). Can you link the image discussed so that it is more transparent what's making it huge?

[No9]

I think it's the SBOM and the Launcher layers (most likely the launcher). I would be very very surprised if it's the WASI layers.
If I understand it correctly the additional layers are added in the post build

1. Here is the 1Gb image

2. And this is the original

This is the two stage Dockerfile used to build the second image.

[Edit]
As per point 3

The launcher layer isn't required and should not be added to the final image but it doesn't seem to be possible to override.

[codefromthecrypt]

I suppose the main thing here would be to vet the point 3: what is not overridable by intent vs what's not overridable due to no need to do that yet. In any case, glad to hear wasi isn't adding heft as it shouldn't really be orders of even megabytes above a runtime!

[aslom]

@No9 I was looking to solve the limitation you mentioned ("WASMEdge seems to be limited to JS, Rust and C") and it seems that today the easiest way to host any language will be to use WAGI as all needed in runtime is for the binary wasm to be compiled with WASI and read input form environment variables and produce output to standard out (WAGI is inspired by CGI). I will join today Func WG to talk about it

[No9]

@aslom Seems like a reasonable conclusion - I think my goal of a scratch based image that could be used in kn and at edge was cutting across too many previous decisions. Also the WAGI approach will give the ability to support multiple wasm runtimes which will be good for the ecosystem as well.
Good luck with it and thanks to all for following along.
[Edit]
For the moment I've moved to cargo-generate for a templated project approach.
https://github.com/knawd/gen-templates/blob/main/README.md

[aslom]

I have got it now to work with WAGI: #1551 (comment)

[aslom]

As Wasm is evolving we should update it in 2024. In particular to take advantage of the latest developments such as support for outbound HTTP and CLI options in WASI previeew 2: https://github.com/WebAssembly/WASI/tree/main/preview2

[lkingland]

I think we have some new information to add to this discussion now, @aslom?

[aslom]

We had quick discussion in Tuesday meeting - here is gist of it:

@lkingland mentioned that initial function code deployment for new users is major hurdle, in particular getting container registry configured and working

After discusssion we think that one use case could be ultra fast Knative func deployments - as WASM is already a lightweight container format we could have a shell container that is deployed in dev mode and then WASM code is located into it. No need for users for container registry as the shell container is provided and WASM blob could be send directly to the container to run

Other use case/scenrio could be to do it for edge scenarios where WASM code is deployed and a web page is available. This takes advantage of WASM to generate code once and run it anywhere without need to recompile for what hardware is used on edge.

[lkingland]

Thanks for writing it up. I am excited about the benefit both of these avenues could bring. I am not sure how the first would work (I am pretty sure a registry is needed for when a new instance of a function would be scheduled on a different node), but perhaps there's a way (or maybe it only works in dev mode?).

The second would require a new method signature added to our base set (HTTP, CloudEvent) that would be specifically only for functions executing in an edge environment. I don't think that's a problem, however, because of how unique the context is.

[aslom]

Yes about the first - I would focus on maximum speed so no registry and directly streaming wasm as you compile it into Knative func container that hosts WASM in dev mode.

About the second: we should identify two (more?) target devices such as Android smartphone and Raspberry PI?
Quick search about WASM and Raspberry Pi I found this form 2022: https://wasmlabs.dev/articles/modern-websites-pi-zero/

[aslom]

One way to accomplish very quick deployments for dev version is to use container that has sshd running and then sftp or rsync files from local dir to remote container over kubectl port-forward tunnel.

[lkingland]

I remember discussing this as a future development option quite some time in the past. It will of course add complexity, but the ability to skip some of the slower steps would be very nice from a user experience angle. I would like to know how much faster this would be than using the "Host" builder, because it's fairly optimized once the base container layer exists on the target nodes (it does not download a build container).

[aslom]

My goal is to see what is the ultimate speed - really small updates instantly visible in Knative function without any delays. I think WASM function can be compiled to be really small and only included code necessary then dynamically linking to WASI or other libraries when running in remote container.

[aslom]

Today in WG meeting we talked about options to get code form local dir into remote container. One is to use SSHD but without any admin privileges configured just for one user. One port exposed to do ssh over kubectl port-forward to copy files and also ssh or kubectl to forward port for debugging - IDE would open connection to the port and allow remote breakpoints.

Other option mentioned by @matejvasek is to use stdin and stdout to connect to remote container that may include sending tarball that is then extracted remotely. (my understanding of it is limited)

[matejvasek]

@aslom
Yes, for general cluster networking in Go we have NewInClusterDialer. We use it for instance for func invoke where function is not exposed. Also we use it for OCI image upload to internal OpenShift cluster. Advantage is that it does not require higher privileges as opposed to port-forward. In my experiece we had trouble to do port-forward as ordinary user, at least when port forwarding foreign namespace (e.g. OpenShift image registry).

For data upload we have UploadToVolume it uploads tar stream to chosen PVC. It uploads data to PVC using ephemeral pod. But it could be used as inspiration for code uploading data into a pod via k8s/exec call.

Another inspiration could be oc cp sub-command it works similarly to the UploadToVolume.

[matejvasek]

It's akin to doing something like this in shell

tar cf - src-dir-or-file | kubectl exec pod/our-pod -i -- tar fx - -C /target/dir/in/pod