[Feature]: dynamic keymapping based on in-game information

[FlyMeToTheMoonAndLetMePlayAmongTheStars]

Is your feature request related to a problem?

Right now keymapping is static, if you set a keymap circle at a specific position it will stay that way regardless if the game interface changes on its own. Problem is, many games have UI or control elements that are often dynamic and it's impossible to account for all of them with fixed position keymap circles without adding too many keys that make the controls overly complicated.

Describe the solution you'd like

I wonder if it's possible for playcover (playtools to be exact) to somehow "understand" what is being shown in-game and dynamically change the keymap buttons based on this information.

How this might work is when the user adds a keymap while in-game (the circle for a single touch point input), it could "learn" the arrangement of in-game pixels/colors contained within the keymap button area. This kind of keymap can be inactive when the pixels do not match, and activated when there is a match.

As an example, in Genshin left mouse button (LMB) and right mouse button (RMB) are statically bound to the virtual on-screen button for controlling normal/charged attacks, with this dynamic system, the user could in theory map LMB and RMB to other contextual menus that pop up from time to time, such as using when choosing to use condensed or original resin for domain drops, ley lines, and many other confirmation/cancel pop-ups.

This kind of dynamic system would work best if there was also on-screen input hints like the idea on-screen keymapping hint UI, and is an alternative direction to multi-dimensional keymapping.

Of course all of this is under the assumption that it is indeed possible to capture the in-game screen output and make use of some of that information in a way that runs efficiently and does not degrade game performance.

Anything else?

I'll make some concepts if this idea is proven to be feasible by people who understand programming.

Issue Language

• Yes my issue is written in English

[XuYicong]

I would not say this is impossible but I'm definitely not going to do this if there is any other open issue in this repo lol.
It is appreciated if anyone else wants to challenge this.