Relative image match / click (like ui.vision's XClickRelative)

[johtso]

So, ui.vision has a really handy feature where you can take an image of a portion of the ui, draw a green box around some consistent anchor point, and a pink box around the area that you want to interact with.

An example is a label for a form field, and then clicking on the input box that contains changing / dynamic data.

Is this something that can currently be done with nut? I didn't see anything in the docs.

I don't know if ui.vision does anything more clever than matching on the area in the green box and then calculating an offset using the relative position of the pink box.

https://ui.vision/rpa/docs/xclick

[s1hofmann]

Hi @johtso 👋

I'll have to check out the link you provided in detail before I can answer this thoroughly.
I'll take some time to do so, please be patient :)

[s1hofmann]

@johtso Maybe I’m missing something, but what’s the purpose of the green box, exactly?

If I understand this correctly it first searches for the green box, then searches for the pink box and moves the cursor there.
But it has to search for the pink one anyways, so what’s the point of having the green one?

[johtso]

@s1hofmann I think the idea is it never searches for the pink area, it just clicks relative to the green area based on distance. This means the pink area can be targeted even if the contents changes so dramatically it couldn't be reliably matched visually.

Basically it's an easy way to specify a visual region and an offset. After taking the screenshot it would pop up a little image editor type thing that allows you to apply the markup.

I'm imagining maybe you'd have a vscode extension that activates a region selection tool, and then saves it to a sensible place in your project and adds the reference to your code.