0.21.0: --depth on the iOS Simulator returns COMMAND_FAILED "regular iOS snapshot presentation requires a valid viewport"

[benjaminkomen]

On 0.21.0, is <selector> visible --depth <n> returns COMMAND_FAILED with the message
regular iOS snapshot presentation requires a valid viewport on every screen we tried. The same
command on 0.20.10 answered normally on some of those screens.

Found while confirming the fix for #2273, which is confirmed and working. Filing separately because
this is a different path.

Version

agent-device 0.21.0 on an ubuntu-latest GitHub Actions runner. The daemon is a remote EAS
Simulator session and also reports 0.21.0. The target is an iOS simulator. The app is a React Native
app built with Expo.

What we ran

Ten is <selector> visible calls per variant per screen, on six screens. The variants are unscoped,
--scope <identifier>, and --depth 20.

The depth argument is a plain integer and our invocation is unchanged from the 0.20.10 measurement.
The screen numbering matches #2273 and #2274.

--depth on 0.21.0 against 0.20.10

screen	0.20.10 depth	0.21.0 depth	0.21.0 error
screen 1	0 of 10, correct	0 of 10	regular iOS snapshot presentation requires a valid viewport
screen 2	10 of 10	0 of 10	regular iOS snapshot presentation requires a valid viewport
screen 3	10 of 10	0 of 10	regular iOS snapshot presentation requires a valid viewport
screen 5	0 of 10	0 of 10	regular iOS snapshot presentation requires a valid viewport
screen 6	0 of 10	0 of 10	regular iOS snapshot presentation requires a valid viewport
screen 7	0 of 10	0 of 10	regular iOS snapshot presentation requires a valid viewport

All 60 calls returned COMMAND_FAILED with that one message. The message does not appear anywhere
in our 0.20.10 logs.

Two rows are the clearest evidence. Screens 2 and 3 answered 10 of 10 on 0.20.10 and answer 0 of 10
now.

Screen 1 is worth naming too. On 0.20.10 its depth-capped calls returned COMMAND_FAILED with
Selector did not match, which is the correct answer, because its landmark sits deeper than a cap of 20. That correct negative is now replaced by the viewport error, so no screen gets a real answer from
--depth any more.

The unscoped path is fixed

For completeness, since it is the same command without the flag. The three screens that answered 0
of 30 on 0.20.10 now answer 10 of 10 unscoped, and XCTEST_RECORDED_FAILURE is gone from the whole
run. Median latency fell from 1.2 to 5.2 seconds down to 0.5 to 1.1 seconds. That part is #2273 and
it works.

An observation about --scope, which we are not calling a defect

We do not have enough to claim a regression here, and the error code is a correct-negative code
rather than an infrastructure one, so this may be intended.

screen	0.20.10 scoped	0.21.0 scoped
screen 1	10 of 10	10 of 10
screen 2	10 of 10	0 of 10
screen 3	10 of 10	0 of 10
screen 5	0 of 10	0 of 10
screen 6	0 of 10	10 of 10
screen 7	0 of 10	0 of 10

Every 0.21.0 failure above is COMMAND_FAILED with Selector did not match. We scope to the same
identifier we then query for, so a scoped query for an element that the unscoped query finds should
match. If scope resolution changed deliberately, this is us needing to change how we call it, and a
note to that effect would be enough.

One correction to what we told you in #2273

We said there that a truncated capture reports a node count which is a lower bound, and reasoned that
the affected screens were therefore probably the largest trees.

On 0.21.0 screen 5 reports the same 178 nodes it reported when it was truncated, and it is no longer
truncated. So that count was complete all along and our inference was wrong. Recording it here
because we put the claim in front of you.