Brainstorm Ideas

[frink]

I think you could use the WebSocket implementation in Facil.io to replace WebRTC if you wanted...

You mentioned thinking about this in your Fosdem 2019 presentation...

Obviously, the server piece of this protocol is the most important part in terms of controllable latency. I theory, WebSockets could be used to asynchronously of the synchronous draw loop in the Wayland protocol. Essentially, you could send interface events back via one channel while you are continuously receiving new frames. This may be able to further obscure the latency incurred by the network.

The idea would be that you can essentially split applications between various hosts using the web worker scenario. You have the web worker load wasm so that you can have a very fast drawing library and now you can send draw calls instead of finished frames. Essentially what X11 does now...

Another thing that occurs to me is using the WebRTC protocol to deliver WASM apps to bare metal servers to spin up more processing power. This would be truly cloud computing on the back and front because a call to a web host would then spin instances of worker on raw hosts that live as long as they are needed. Essentially allowing a modest terminal with a nice GPU and hefty backend processes doing all sorts of odd number crunching.

In this sort of scenario the network latency between the "cloud threads" (distributed wasm packages on small containers) having their response come back in a timely manner may or may not happen depending on how the app is built...

Regardless, I like where your head is at here. The opportunities that this technology provides are the closest I've seen to true a neuronet using separate synapses to compute a result simultaneously.

I think it's wise to have both WebRTC and WebSockets to explore every protocol possible. In reality, we really need to be building some of this on UDP directly. But maybe that will be exposed in browsers in the near future. Who knows...

Kudos.

[Zubnix]

Hi Frink,

Current master branch was just recently rewritten to use only WebSockets and drop WebRTC entirely. Current master also uses the establishment of a WebSocket connection with certain HTTP query parameters as a signal to launch a (remote) application.

WebRTC was dropped because WebSockets make it far easier to implement inter-host copy-paste operations (as data source file descriptors can be represented by a WebSocket URL, which makes it easy for the data receiver to connect and receive the data). It also eliminates the need for an intermediate WebRTC signaling server. Another advantage of WebSockets is that it offers far more predictable network behavior which makes it easier to setup in a network environment than WebRTC.

Your ideas are definitely in line what I had in mind. If you or others are interesting in exploring this further, you can always reach me on the #wayland IRC channel on FreeNode.

[frink]

I would really like to flesh out this idea more. I may have some time for a side project in a month or so. Here are several things that may need to be addressed to for a complete ecosystem:

• It would be helpful to create a simple display that provides the browser component and can work as a direct display via VESA or FrameBuffer using the GPU for acceleration. This would give us a full usable desktop environment where we can eat our own dogfood and begin using this tech on a daily basis. That will bring some growing pains necessary to actually begin thinking through the intricacies of this architecture both from a usability and security standpoint.

• For the redraw of the frame it may also be helpful to send only the portion of the frame that needs redrawn instead of the last frame. There are pros and cons to this idea. If a frame gets dropped by the network a window could be malformed. However, transferring only the refresh box will improved throughput since it's smaller in size. May not be enough of an issue to fix. I don't know. But I thought it's worth documenting a decision one way or another.

• The distributed process component across multiple servers is very interesting too. The whole idea of having the client app delivered via web services to run in the browser then connected to the master host who initiates many slave processes as needed (on same or different machines) this all has merit but needs to be fleshed out and proven. Distributed processing is even more convoluted than threaded programing. Massive parallelism with acceptable latency in the hundred millisecond range is very different from concurrent processes on the same machine. The entire paradigm needs shaping before its useful. - See Bare Metal OS for another approach to this...

• Another thing I'm interested in understanding in this whole process is multiple cursors in the interface. With a distributed system like this where there could be a mesh of clients and servers this is already implied. But then with multitouch becoming a standard use case this also becomes more important as well. Making sure that event messaging is not single cursor is going to be an important part of this whole thing.

• Transferring applications between users in a multi-tenant environment is another case that needs to be explored. Since multiple browsers could be theoretically connected to the same machine, one user may have a program open and need to send it to another user to view, edit manipulate. They may also want to share interacting with the same application. Understanding how these things play out is an important consideration.

The big aha for me is realizing that if applications are running in the cloud from everywhere and users are connecting to the same computer from everywhere, the interface paradigm changes to where many applications are multi-tenant and within those seats multi-cursor. The definition of interface changes and the definition of server takes on new meaning as well.

It's a brave new world. This stuff is really pushing the limits...

[Zubnix]

I've though about all these points as well so allow me to address your points in order:

• We probably want WPE with a suitable back-end (mesa), or perhaps a chromium based fork (CEF?).
• Not needed. H264 encoder already takes care of this for us using P-frames (delta frames). A full HD office application updates are usually no more than a few hundred bytes per P-frame for text updates.
• I'm not quite sure what you mean here.
• I think you're making some wrong assumptions here. Multi touch has no relation with multiple pointers. Multiple seats do, which is already natively supported by the Wayland protocol. However this has currently no relation to the project unless we one day decide for real-time application sharing.
• This is the case for real time application sharing. In theory as simple as adding an extra logical seat from the shared user and duplexing the frame output to the shared user. Simple in theory but harder in practice but still very much possible.

All these points (and much more) show the endless potential of this approach. 🌈 🦄 But let's first get the foundation rock solid. There are still about 122 TODOs & FIXMEs in the code, and fixing all of those will probably generate another 100 issues. 🚧

Most of the concrete for the foundation is poured. 🏗️ Let's pour the rest, let it harden and stabilize, and build the most awesome temple you can imagine. 🏛️

[frink]

In point three I'm talking about the distributed nature of programs. What if...

• You access the program from Server A
• It sends you Wasm and starts a connection to it's part running in the background
• Then you do something really intensive like crunching unsolvable math
• The back end recognizes it needs help and spins off daughter processes but in a new way
• The master server sends WASM to slave servers which are standing by to help out
• The then the slaves are able to communicate back to either the master server or the greenfield browser

When we realize that this could happen as both single seat and multi-seat environments there is a lot to think through. Giving each part of the system a way to collate data to reorder stuff that may come in out of order becomes even more important that in simple multi-thread programming. Now you've got threads running on multiple boxes with milliseconds of latency. That's an interesting set of problems to figure out... ANd that's before you add in things like DDOS prevention which becomes even more of a concern in this distributed world.

If you have built castles in the air, your work need not be lost; that is where they should be.
Now put the foundations under them.
Henry David Theroux

I agree 100% - I'll try to carve out some time to help...

[frink]

I think the first thing is to get this to the point of replacing the noVNC and XPRA use cases.
We are basically there but probably need to write a few scripts to make this nice and easy. That's the first thing I'm going to attempt anyway. I'm trying to get everything setup on a VPS here using Greenfield as my UI so that I'm actually forced to endure and fix problems as they come.

//edit I have moved your installation issue to a new ticket #10 -Zubnix