Incorrect click location

[JGKle]

Whenever I ask it to click on a particular point, it misses - it clicks below and to the right. Below is a screenshot: I visited excalidraw, drew the black dot, changed the color to red, told it to click the black dot. Then I changed it to blue and told it to try it again. As you can see, red and blue are shifted a little to the right and down significantly.

[Jeomon]

Windows-mcp is operating based on the accessibility tree data
what you shown here is a limitation of the MCP server
because the dots has no information about it in the accessibility tree
and whatever elements lies inside a canvas can't be detected by the accessibility tree

[JGKle]

Windows-mcp is operating based on the accessibility tree data

Hmm...so that's what it tried at first, but it said "the clicks didn’t register. I’m not seeing any interactive elements from the accessibility snapshot, so the automation can’t “see” the hamburger/menu as a target. That usually means the app UI isn’t exposing controls to
the automation layer." I'm trying to use it to automate testing of a .net application.

But even so, shouldn't it be able to take a screenshot, find the location of a pixel on the screenshot, and then click that same coordinate? Why would it be "shifted"? Like if it can locate the x,y on the image, shouldn't it be able to just simulate a click at that same x,y?

[Jeomon]

[Jeomon]

Hmm...so that's what it tried at first, but it said "the clicks didn’t register. I’m not seeing any interactive elements from the accessibility snapshot, so the automation can’t “see” the hamburger/menu as a target. That usually means the app UI isn’t exposing controls to the automation layer." I'm trying to use it to automate testing of a .net application.

But even so, shouldn't it be able to take a screenshot, find the location of a pixel on the screenshot, and then click that same coordinate? Why would it be "shifted"? Like if it can locate the x,y on the image, shouldn't it be able to just simulate a click at that same x,y?

When you develop the .NET application, make the application developed accessibility friendly, then you can test it.

Which model you have tried

[JGKle]

Which model you have tried

Codex on gpt-5.2-codex, high reasoning

When you develop the .NET application, make the application developed accessibility friendly, then you can test it.

Got it. The trick was to assign AutomationProperties.Name in axaml like: <Button AutomationProperties.Name="Open" ...... />, or in code: AutomationProperties.SetName(x, "Open"). Then it can find them.

Still, given that it can take screenshots & understand what it sees, why wouldn't it be able to say "I see a dot at pixel x,y, so now I'll click at x,y"? Strange that it ends up shifted. If there's some scaling going on (i.e. screenshot dimensions isn't an exact representation of input/screen dimensions), maybe all it needs is context on how to map them/translate between them. It's clicking reasonably close to the dot, so clearly it understands what to do. And at a given resolution, it always misses by the exact same amount, so there just seems to be some sort of shifting/scaling between the screenshot coordinate system & the input coordinate system.

[Jeomon]

The accessibility tree coordinates are based on logical screen, while the screenshot we have to move to physical coordinates

[JGKle]

Right - so presumably if the LLM were given info on how to map logical to physical coordinates, it could easily calculate how to translate between them & thus where to click

[Jeomon]

yes but that i have already done already but still let me check

[Clusters]

any news regarding this?
I can still confirm the misalignment between screenshot detection and click position.
For instance if the model (rightfully) tries to click a desktop shortcut, if the app won't start through the app tool.

I'm not sure if I understood how the model decides a position to click depending on the screenshot. Afaik it has no understanding of absolute x and y positions of elements in the image, right? Or is this just my model? (using Qwen3.5-122B)

[shaojun]

i had similar issues, i'm using qwen-code with model qwen3.6-plus, my screen size is: 3079,1927.

I've asked the agent to do a coordinate alignment for 5 points in my screen, 4 corners and 1 center of the screen, i can see the mouse arraow are all correctly positioned.

But once i asked it to operate on application, then it never made it, the operating position(which is wrong) always a bit distance in the left up from grand-truth position.