A UI for Wolf

[ABeltramo]

This has been discussed already many times over a few places (#69 (comment) #44) and it's getting more and more clear that we need a UI for Wolf. The main goals are:

• Configuration: We need an easy way to configure Wolf without editing the config file.
• Sharing: Allow users to share a game session with multiple users (instead of making completely isolated sessions
  as we are doing right now).
• Monitoring: Allow users to see what's the status of the game session (ex: "Downloading docker image", "Container
  died", etc.)
• Usability: This GUI has to be controllable by keyboard and mouse, whilst at the same time should be usable in a TV
  screen even just with a gamepad.

Implementation

I think our "dream solution" here would have:

• Ability to run multithreaded directly inside Wolf so that each user will have its own UI state, and we can show what's
  happening even before the app container is started.
• A way to easily communicate bidirectionally with Wolf (status reports, configuration changes, etc.)
  • An easy way to inject mouse, keyboard and joypad events without going through the system
• Expose direct access to the raw framebuffer where the UI is rendered so that we can feed it directly into the video
  encoding pipeline.
• (optionally) An easy way to build the UI, possibly noob friendly like HTML+CSS+JS.

I've been looking around for something like this and I think the best candidate is Sciter, more specifically the windowless version of Sciter: Sciter.Lite
There are some examples:

• render to OpenGL
• render to Bitmap (uses CPU rasterization)

The nice thing is that it seems to tick all the boxes above, the only concern is that it's not open source, and it's not free (albeit just for commercial use which shouldn't concern us). Is there any other alternative that we should consider?

---

Otherwise, we could just run any normal GUI toolkit (GTK, Qt, Electron, etc.) in a separate process inside the Wolf container, hook it up with our Wayland comp, and communicate with it via a socket or something. This would be a lot more complex and
less efficient, but it would be open source and free.

[brine0m]

@ABeltramo, not sure if the above implies this but I would also suggest a rudimentary auth solution with 2 roles, admin and user. User would be able to login and pair devices and this would allow sharing of installed games between paired devices. Admin would do that and the aforementioned config / monitoring.

Also haven't read in depth of Sciter but are you intending for this to be a webapp or desktop app... just first few things I read it looks look as it uses mostly web languages but seems like its to dev desktop app

[ABeltramo]

not sure if the above implies this but I would also suggest a rudimentary auth solution with 2 roles, admin and user. User would be able to login and pair devices and this would allow sharing of installed games between paired devices. Admin would do that and the aforementioned config / monitoring.

Yep proper user management will definitely be added. First we'll have to plug everything together though..

Also haven't read in depth of Sciter but are you intending for this to be a webapp or desktop app... just first few things I read it looks look as it uses mostly web languages but seems like its to dev desktop app

The idea here is that Sciter will allow Wolf to render mutliple GUIs, ideally one per user/session without even having a Desktop and without requiring the user to jump over a browser (which might not be possible on some Moonlight platforms, like TVs/handelds ..).
Everything will be rendered inside Moonlight and controlled with the same mouse/kb/joypad devices.

Ideally this UI will also be easily developed by using the same frameworks as any webapp so that anyone can easily contribute!

[salty2011]

This sounds interesting, look forward to seeing it in action.

[SamuelDudley]

I was digging around a bit looking to see if it would be possible to use electron for the UI to simplify (maybe?) development and contributing and ran into this PR that I'm linking incase anyone finds it useful: electron/electron#42001
Although the PR is marked as closed, it appears to be under active development by the author.

[ABeltramo]

Thanks for the pointer!
I've quickly skimmed thru it, but I can't quite figure out if electron supports running multiple instances from the same process or is it using global state like most of the other solutions? It would be nice to use a fully open source project instead of Sciter, it just seems like such a perfect fit for this use case..

[SamuelDudley]

Thanks for the pointer! I've quickly skimmed thru it, but I can't quite figure out if electron supports running multiple instances from the same process or is it using global state like most of the other solutions? It would be nice to use a fully open source project instead of Sciter, it just seems like such a perfect fit for this use case..

I agree Sciter seems to be a great technical fit and would likely be very performant.

Digging a bit deeper on the electron side it does appear you can create multiple windows from the same process: https://jasonsturges.medium.com/multiple-window-electron-app-9dbffde8ce95

I'll dig into this some more and let you know if it goes anywhere

[austonpramodh]

I am curious, is the "dream solution" going to allow users to stream without moonlight?

This is where I was wondering why we would need Electron or Sciter, Why not just use raw HTML, CSS, and JS?

I might be missing something here I guess.

[SamuelDudley]

I am curious, is the "dream solution" going to allow users to stream without moonlight?

This is where I was wondering why we would need Electron or Sciter, Why not just use raw HTML, CSS, and JS?

I might be missing something here I guess.

Happy to be corrected, but I think it's this point which is steering the Sciter / electron discussion:

Expose direct access to the raw framebuffer where the UI is rendered so that we can feed it directly into the video
encoding pipeline.

Both Sciter and electron (maybe) are a way to generate a framebuffer that can be mixed into the existing wolf stream to overlay the UI.

[austonpramodh]

Gotcha makes sense. But I dont understand why UI needs to be rendered(or generate raw framebuffer) instead of having traditional Rest APIs and client web pages. Unless the idea is to make streaming work on the webpage, like may be Xbox cloud gaming.

I am assuming @ABeltramo has some idea going on with this.

[ABeltramo]

But I dont understand why UI needs to be rendered(or generate raw framebuffer) instead of having traditional Rest APIs and client web pages.

It's actually fairly simple, not every Moonlight client has browser support! Think of TVs, handhelds (Switch, 3DS, PSP), etc.
Having our custom UI rendering in Moonlight will allow us to instantly support every client. Also, the flow will not be very smooth right? You have to switch back and forth from the browser to Moonlight, or even worse: pickup your phone to access the browser because you can't do that from your TV!
On top of that you get joypad/remotes support for free via Moonlight because we can't assume that a mouse and keyboard will always be available to navigate the web UI.

This "UI" is still a tad up in the air, we were discussing this just earlier today with @shocklateboy92 and @salty2011 in a call; I think we are going to approach this in a series of steps:

Step 1: expose an API for Wolf

I'm working on it here: #101
The idea is that we'll expose all the events that are currently powering Wolf over a Linux socket. The type of events and what would be possible it's not yet flashed out but ideally most of the Wolf components will be available over it, ex:

• Moonlight pair requests
• Audio/Video encoding and pipelines
• Virtual devices: joypads, mouse, keyboard, pen, etc.

All this stuff would be initially just to listen for those events when Moonlight clients trigger them, but, potentially, we can also allow applications to trigger them without any Moonlight session. Ex: create a virtual joystick and send inputs from a different application; start a video pipeline that you can view from VLC...

Step 2: create an "admin" web UI

Working on it here: #105 (but possibly moving it to a separate dedicated project page)

Very early day and needs to be fleshed out, but ideally this would allow you to manage and pair Moonlight clients (without going through the logs), change the settings (without manually editing the config.toml), see the logs...
This will communicate with Wolf via the API created in Step 1 and expose a web UI.

Step 3: create a "user/running" UI

This is what this issue was originally talking about and what I'd really like to build.
In my head the flow is as follows:

• You connect via Moonlight to Wolf
• From there you pick up a single app that it's available
• Wolf starts the stream, instead of running the Docker apps straight away it instead shows a UI (you'll be able to navigate the UI both with mouse and keyboard and joypads all via Moonlight)
  • It opens up as a netflix-like list of users:
    • If you pick another user that is running something you can join the session and play together (or even just watch)
    • Otherwise, you pick another user and it just show a list of apps; same as it is right now in Moonlight and it'll eventually start the app that you pick there
    • In both cases, when a selection has been made Wolf will switch the video/audio sources in the pipeline from the UI to the actual Docker container running.

It's an intermediary step compared to what we have at the moment so that we can properly support users.

Now, the dream solution is that this UI renders directly in Wolf; this way we don't have to run a Desktop, an additional Docker container and all that stuff that can fail, but we can provide feedback directly to the users.

But, that turns out to be more complicated than expected. So I'm leaning towards a more simple approach: let's just run a fullscreen browser or an Electron app that shows this UI and communicates with Wolf via the APIs. We can build all that and still re-use everything if we ever want to actually embed it in Wolf without the additional container + browser.

---

I hope this helps clarify it a bit, feel free to share feedback and ideas; they are very welcome!

[austonpramodh]

Yup, it does clarify, Now I understand. Thank you. Super excited for this.

[ABeltramo]

Quick status update: in #101 I've added a fully functional REST API, you can see the docs for it.

The plan now is to start having fun with Electron (or similar) and build the actual user UI that will then trigger stuff via the Wolf socket..

Stay tuned! 😉

[SamuelDudley]

The API looks amazing! Out of interest I spent an afternoon looking at open source lighter weight options (something similar to Sciter) and found egui (rust) https://github.com/emilk/egui/tree/master

Most of the integrations seem to support rendering to a window, which defeats the point of it being lightweight for the purposes of wolf. The skia I integrations https://github.com/lucasmerlin/egui_skia has an example that can render to an image while also being headless, which seems closer to what you are after?
Then there is the whole issue of it being written in rust rather than cpp. From an API perspective you have solved that issue :) but I'm guessing there would also need to be some gstreamer integration that would be a touch point between rust and cpp?

[ABeltramo]

Thanks for looking into it, I think I'm really going to keep it simple to get the ball rolling.

We have a Wayland compositor already, I'll just install Electron in the Wolf container and fire it up when the session starts. Then, via the APIs, the Electron UI can control additional apps, and I'll just swap the "main focused" window from our comp to the newly started app.

I'm going to try to keep it simple this time, we can always overcomplicate things later (as usual).

[SamuelDudley]

Thanks for looking into it, I think I'm really going to keep it simple to get the ball rolling.

We have a Wayland compositor already, I'll just install Electron in the Wolf container and fire it up when the session starts. Then, via the APIs, the Electron UI can control additional apps, and I'll just swap the "main focused" window from our comp to the newly started app.

I'm going to try to keep it simple this time, we can always overcomplicate things later (as usual).

Yep, makes sense!