Wrap the Cloudflare Worker runtime APIs

[armanbilge]

Many of these are very similar to existing Web APIs, but slightly different. So we should probably facade and wrap them ourselves :(

• Cache
• Durable Objects
• Encoding
• Fetch
• FetchEvent
• HTMLRewriter
• KV
• Web Crypto
• WebSockets

Typescript available here:
https://github.com/cloudflare/workers-types/tree/master/overrides

[zetashift]

What about using ScalablyTyped to generate the Scala defs as an initial start?
Even the cloudflare/workers-types repo says those types are generated as well.
(And then making it typelevel-friendly)

[armanbilge]

As an initial start, we can grab most of these from existing places. I've already done some in #4. Fetch, WebCrypto, and WebSockets are available via scala-js-dom.

(And then making it typelevel-friendly)

My preference would be not to expose the facades at all if possible. I think that's what I was doing in #4, but it's been a while.

Would be great if you'd like to pick this up :) my main concern was figuring out a way to actually test this.

[zetashift]

I don't know much about Feral, or Cloudflare Workers :P.

But for testing, am I misunderstanding something? We should be able to test with https://developers.cloudflare.com/workers/wrangler/ right?

[armanbilge]

I don't know much about Feral, or Cloudflare Workers :P.

Sounds like a great learning opportunity! :P

But for testing, am I misunderstanding something? We should be able to test with https://developers.cloudflare.com/workers/wrangler/ right?

Yes, actually I've had my eye on https://miniflare.dev/ for testing. But wiring it up is non-trivial :) I started a branch doing that here:
armanbilge/scala-js-env-jsdom-nodejs@7f6081c

[zetashift]

I read up on Miniflare recently and from Cloudflare's blog: https://blog.cloudflare.com/workerd-open-source-workers-runtime/

workerd is also designed to facilitate realistic local testing of Workers. Up until now, this has been achieved using Miniflare, which simulated the Workers API within a Node.js environment. Miniflare has worked well, but in a number of cases its behavior did not exactly match Workers running on Cloudflare. With the release of workerd, Miniflare and the Wrangler CLI tool will now be able to provide a more accurate simulation by leveraging the same runtime code we use in production.

[zetashift]

I'll check out the #4 PR, but from a cursory glance it looks hella intimidating :P.

So it uses facades, but provides a very functional-programming-y interface on top of it using cats effect and fs2?

[armanbilge]

Interesting! Well I guess we should toss that miniflare branch then 😂

Thanks for sharing, that's pretty cool. I'll have to look at it more closely to figure out how we might use it.

It also reminded me that cloudflare workers supports WASM, and Scala Native will support WASM in the next major release. So that's something to keep in mind as well 🤔

---

So it uses facades, but provides a very functional-programming-y interface on top of it using cats effect and fs2?

Yes, that's the idea. http4s-dom does a similar thing, as does the History API in armanbilge/calico#68. In fact, fs2-io JVM does the exact same thing for JDK APIs (instead of JS facades). So it's a very common pattern.

Feel free to comment directly on the PR with questions!

[zetashift]

I asked in the scala-native discord channel about Scala Native wasm examples, but I didn't know it was slated for next release, thought it was in already haha.

I guess the hard part of contributing to that PR is, the fact that I've never written/worked on a FP library, I've only consumed them!
I'll read more into it, thank you for the help.

[armanbilge]

Yes, check out https://github.com/shadaj/scala-native-wasm

I didn't know it was slated for next release, thought it was in already haha.

Yep it was already merged into the 0.5.x branch, in theory you can publish a snapshot and start playing it.

I guess the hard part of contributing to that PR is, the fact that I've never written/worked on a FP library,

Personally I find it quite fun to figure out how to wrap some (god awful) non-FP API with CE3/FS2 😃

[zetashift]

Personally I find it quite fun to figure out how to wrap some (god awful) non-FP API with CE3/FS2

It does sound quite fun, I just have never done it :P. So let's try this out!
Here's what I have in mind, let me know if I'm close:

• rebase the PR?
• integrate wrangler instead of miniflare with the current code
• test and break stuff
• fix broken stuff by using fs2/functional programming things?

[armanbilge]

@zetashift personally I merge instead of rebase 😜 but yes!! 🚀

integrate wrangler instead of miniflare with the current code

I'm quite interested about this part too :)

[zetashift]

• integrate wrangler instead of miniflare with the current code

So this seems to be a bit of a rabbithole. This is as far I know the current situation.

• Miniflare was a simulator for testing Cloudflare Workers locally.
• workerd is the new runtime and is the future as well.
• Miniflare v3(as of writing in beta) wraps workerd and provides additional friendly libraries for Cloudflare API's like R2, KV.
• Miniflare says the following: https://github.com/cloudflare/miniflare/releases/tag/v3.0.0-next.1

Miniflare now uses Cloudflare's open-source Workers runtime, workerd, to run your code! tada This is a massive change, and should mean your code runs locally almost-exactly as in production.

Breaking Changes

Miniflare's CLI has been removed. We're still discussing whether it makes sense to keep this, given wrangler dev has additional features such as automatic bundling and TypeScript support. For now, use wrangler dev --experimental-local.

So I think we should still use Miniflare, but for the developer experience that will mean actually using wrangler.

[armanbilge]

So I think we should still use Miniflare

I agree.

but for the developer experience that will mean actually using wrangler

I'm less sure about this. I wonder if instead of working with the wrangler CLI, we can directly use the Miniflare JS API. This is essentially what I was exploring in armanbilge/scala-js-env-jsdom-nodejs@7f6081c.

[zetashift]

I'm less sure about this. I wonder if instead of working with the wrangler CLI, we can directly use the Miniflare JS API. This is essentially what I was exploring in armanbilge/scala-js-env-jsdom-nodejs@7f6081c.

Ah yea! You posted this link before, but I couldn't really grasp the implications.I'll look into the Miniflare API more (the docs seem solid from a first glance) and figure out what this whole scala-js-env-jsdom-nodejs (what a naming :P) exactly does.

[armanbilge]

Right, I cheated by starting with a fork of this project because it does a similar thing and trying to modify it for miniflare:

https://github.com/scala-js/scala-js-env-jsdom-nodejs

Basically, JSDOM is a thing that lets you emulate browser APIs inside of Node.js.

https://github.com/jsdom/jsdom

Similarly, Miniflare is a thing that lets you emulate Cloudflare APIs inside of Node.js. (Or at least it used to be, not sure if workerd has changed that.)

The scala-js-env-jsdom-nodejs lets you run your Scala.js tests inside of the JSDOM environment.

So in theory seems like we should be able to do basically the same thing except with Miniflare.

[zetashift]

Right that clears a lot up, thank you!
So Miniflare (the v3) currently using workerd, which is a custom runtime and I don't think using scala-js-env-jsdom-nodejs will cut it?

Look at miniflare's code it, it just seems to talk to a local workerd runtime https://github.com/cloudflare/miniflare/blob/v3.0.0-next.2/packages/tre/src/index.ts

Wouldn't it be better to have bindings to Miniflare(which does all the hard work of talking to workerd) then that are used for running tests locally?

EDIT: I guess that's what you did in your scala-js-env-jsdom-node-js fork, but it still seems somewhat redundant? Like it's a nodejs process emulating a DOM, that calls a miniflare library talking to a workerd runtime

[armanbilge]

Like it's a nodejs process emulating a DOM, that calls a miniflare library talking to a workerd runtime

scala-js-env-jsdom-node-js is a Node.js process emulating a DOM. My fork removes JSDOM and replaces it with Miniflare, so we end up with a Node.js process emulating Cloudflare (no more DOM).

With the new Miniflare, you are right that we would have a Node.js process calling to Miniflare calling to workerd. Which raises the question, why don't we just build a JSEnv that directly runs on workerd? And I think we've come full circle 😅