wayland: add support for tabel input

[jp7677]

This is a draft PR about adding support for tablet/stylus input for Wayland. The motivation behind this is that this PR lets me control mpv on a Gnome Wayland session with a tablet/stylus as I'm usually using a tablet as a full mouse replacement. Gnome Desktop unfortunately doesn't offer a mouse fallback for tablet input when a client doesn't support the tablet protocol, so a stylus can't interact with the mpv window there at all.

This is still early draft, but functional (tested on Gnome and labwc/wlroots) except for drag&drop like operations. It still needs polishing, some refactoring to avoid code duplication (cursor visibility) and splitting commits and better commit messages, but I wanted to create this PR in this state for general feedback. The main question is: is there interest in tablet/stylus support and does this PR has a chance of getting merged? The tablet protocol is not that simple and a lot of boilerplate code is needed for just a few basic handlers. I hope not, but I would understand if this is too much for an edge case.

Lua script demonstrating the client properties:

tablet-input.lua

local utils = require "mp.utils"

mp.observe_property("pause", "bool", function(name, value)
	pos = mp.get_property("tablet-pos")
	-- mp.msg.log("info", pos)
end)

function toggle_fullscreen()
	mp.set_property("fullscreen", mp.get_property("fullscreen") == "no" and "yes" or "no")
end

tool_tip_state = ""
tool_stylus_btn1_state = ""
mp.observe_property("tablet-pos", "native", function(name, value)
	tool_tip = value["tool-tip"]
	tool_stylus_btn1 = value["tool-stylus-btn1"] 
	
	-- use --input-tablet-emulate-mouse=no to prevent
        -- all default mouse bindings

	if (tool_tip ~= tool_tip_state and tool_tip == "down") then
		mp.command("keypress MBTN_RIGHT")
        end
	
	if (tool_stylus_btn1 ~= tool_stylus_btn1_state and tool_stylus_btn1 == "pressed") then
		toggle_fullscreen()
        end
        
        if (value["pad-btns"]["1"] == "pressed") then
		mp.command("seek -30")
        end

        if (value["pad-btns"]["2"] == "pressed") then
		mp.command("keypress MBTN_RIGHT")
        end

        if (value["pad-btns"]["3"] == "pressed") then
		mp.command("seek 45")
        end
		
        if (value["pad-btns"]["4"] == "pressed") then
		toggle_fullscreen()
        end
        
        tool_tip_state = tool_tip
        tool_stylus_btn1_state = tool_stylus_btn1
end)

Thanks for creating and maintaining mpv and looking forward to some general early feedback!

[llyyr]

Gnome Desktop unfortunately doesn't offer a mouse fallback for tablet input when a client doesn't support the tablet protocol, so a stylus can't interact with the mpv window there at all.

Is this something GNOME won't implement or something that simply has never been reported? sway falls back to pointer events if a client doesn't support touch or tablet protocols, it would be much simpler to add this fallback in Mutter than go around fixing every single client

[jp7677]

Is this something GNOME won't implement or something that simply has never been reported? sway falls back to pointer events if a client doesn't support touch or tablet protocols, it would be much simpler to add this fallback in Mutter than go around fixing every single client

Looking at https://gitlab.gnome.org/GNOME/mutter/-/issues/2226 from three years ago, I think it is unfortunately the former, at least at that time.

PS: I agree by the way that mouse emulation in the compositor would be better, Labwc did the same as Sway for tablet input and that feels much more robust to me (that is, the Labwc implementation, I never looked that detailed at Sway and can't comment on how well it works there).

[kasper93]

sway falls back to pointer events if a client doesn't support touch or tablet protocols,

Note that it is completely broken in mpv, at least for osc.lua. Sway reset pointer position to "real" mouse pos when all fingers leave the touch screen. By the time osc.lua processes the touch, the pointer is in another place. Making all buttons non-
clickable in practice.

And there is no good way to "fix" or workaround that. Because we have no idea if the move was real or it was emulated switch to mouse.

So proper native touch support might be the only way forward.

[github-actions]

Download the artifacts for this pull request:

Windows

• mpv-aarch64-pc-windows-msvc
• mpv-i686-w64-mingw32
• mpv-x86_64-pc-windows-msvc
• mpv-x86_64-w64-mingw32

macOS

• mpv-macos-13-intel
• mpv-macos-14-arm
• mpv-macos-15-arm

[llyyr]

So proper native touch support might be the only way forward.

We already have native touch support, this is about tablet support. And sway translating tablet events to pointer events works perfectly in mpv. I can't say for touch events as I don't have a touch device, and would need to rip out touch support from mpv to even test it.

[kasper93]

Doesn't work on my end.

[llyyr]

Doesn't work on my end.

What doesn't work? Touch or tablet? If touch doesn't work that's a mpv bug, or a bug with sway/wlroots sending us touch events (not one with translating touch to pointer events).

edit: I checked and it's a sway bug with touch input, works on river.

[mahkoh]

sway falls back to pointer events if a client doesn't support touch or tablet protocols

That cannot work. Let's say an application has no tablet support and uses libdecor to draw window decorations, which (let's say) also doesn't support tablets. The user can now drag the window by its decorations in sway because sway sends emulated pointer events. But then the application adds support for tablet events and now the user can no longer interact with the window decorations because sway no longer sends emulated events and libdecor still doesn't support tablet events.

[jp7677]

I think this is pretty much ready from my side, moving the window by dragging the window with the stylus works now and with that I think the stylus support is solid. That said, I think I got the wiring correctly, but a keen eye is very welcome, especially for shutting things down. Let me know what you think and if it's worth it.

@mahkoh I contributed to the tablet support for labwc, which also uses mouse emulation for clients that do not support the tablet protocol. If you have a case with a reproduction scenario where it doesn't or shouldn't work, could you open an issue there since I would be curious to play with it.

[mahkoh]

If you have a case with a reproduction scenario where it doesn't or shouldn't work

What I described above should be easy enough to reproduce in any application that uses both libdecor and supports tablet input. You might have to enable CSD in your compositor for libdecor to do anything.

[jp7677]

CI for the latest pipeline on master fails with the same errors like the latest pipeline for this PR, so I don’t think this is on me.

[llyyr]

CI for the latest pipeline on master fails with the same errors like the latest pipeline for this PR, so I don’t think this is on me.

Flaky windows CI is new mpv contributor hazing ritual don't worry

[kasper93]

CI for the latest pipeline on master fails with the same errors like the latest pipeline for this PR, so I don’t think this is on me.

Flaky windows CI is new mpv contributor hazing ritual don't worry

I don't understand, why is this Windows related?

EDIT:

Build with GCC 15 is fixed in #16293

[Dudemanguy]

Late here but I finally took a look.

[jp7677]

@Dudemanguy thanks a lot for your review! Is there anything left I need to address? GitHub says that there are still changes requested, but I can’t see any open discussions anymore. I got two of your earlier points only by mail, hence the double check, but may be GitHub is just slightly confused today…

[Dudemanguy]

That's just how github works. I have to give it another review to change it.

[Dudemanguy]

Actually it turns out that I do have a tablet in my closet that can test this. Seems to work well for me (previously mentioned issues I cannot reproduce) and the code looks good now. This is a pretty big wayland protocol so nice work on going through all of it.

[jp7677]

Very cool that this PR has landed. Thanks a lot to all of you for the guidance and patience!