feat: add agent-friendly simulator CLI controls

Adds project default simulator selection, agent-oriented CLI aliases, unified simulator actions, native-AX-first query paths, interactive AX pruning, benchmark coverage, and reliability fixes for transient AX snapshots.

Author: DjDeveloperr

AGENTS.md

@@ -79,7 +79,7 @@ Private simulator behavior is implemented locally in:
 
 The current repo uses the private boot path, private display bridge, and private accessibility translation bridge directly. The browser streams frames from that bridge, injects touch and keyboard events through the same native session layer, inspects accessibility through `AccessibilityPlatformTranslation`, and renders device chrome from `cli/XCWChromeRenderer.*`.
 CoreSimulator service contexts resolve the active developer directory from `DEVELOPER_DIR`, then `xcode-select -p`, then `/Applications/Xcode.app/Contents/Developer`. The display bridge prefers direct CoreSimulator screen IOSurface callbacks and activates the SimulatorKit offscreen renderable view only if direct callbacks are unavailable.
-Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot.
+Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. Shallow snapshots with `maxDepth <= 2` use the tokenized frontmost application translation directly when it is available, and only run the expensive recovery sweep if frontmost lookup fails, so agent-oriented describe loops avoid launchctl and hit-test recovery overhead. Interactive-only snapshots also prune non-actionable native AX leaves during Objective-C serialization before the Rust-side compacting pass; keep this native pruning conservative so selector taps still retain actionable rows plus their ancestors. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot.
 Physical chrome button support uses DeviceKit `chrome.json` input geometry for browser hit targets. Volume, action, mute, Apple Watch digital crown, Watch side button, and Watch left-side button dispatch through `IndigoHIDMessageForHIDArbitrary` with consumer/telephony/vendor HID usage pairs from the device chrome metadata; home, lock, and app-switcher remain on the existing SimulatorKit button paths. Apple Watch Digital Crown rotation dispatches through `IndigoHIDMessageForDigitalCrownEvent` when SimulatorKit exposes it, with `IndigoHIDMessageForScrollEvent(..., target=0x34)` as the fallback. tvOS simulators do not support direct screen touch; browser/API tap maps to Enter, swipe maps to arrow keys, and the native bridge rejects tvOS touch packets before they reach guest `SimulatorHID`. watchOS/tvOS skip dynamic pointer/mouse service warm-up because those guest runtimes abort on unsupported virtual services. Apple TV and Apple Watch simulators are fixed-orientation devices, so client and server rotation paths must not expose or dispatch device rotation for those families.
 Two-point multi-touch dispatch prefers the current SimulatorKit/Indigo packet constructor and falls back to SimDeck's manual Indigo packet adapter. On Xcode 26 SimulatorKit, the constructor expects pixel-space points and stable two-finger movement requires sending `LeftMouseDown` for both `began` and `moved`, then `LeftMouseUp` for `ended`/`cancelled`; using `LeftMouseDragged` for multi-touch moves only advances one contact in UIKit. Do not coalesce multi-touch move packets in the WebSocket or WebRTC control paths, because gesture recognizers need the intermediate two-contact samples.
 WebKit inspection uses the simulator `webinspectord` Unix socket named `com.apple.webinspectord_sim.socket` and WebKit's binary-plist Remote Inspector selectors. It lists only WebKit content that the runtime exposes as inspectable. For app-owned `WKWebView` on iOS 16.4 and newer, the app must set `isInspectable = true`.
@@ -137,32 +137,43 @@ Useful direct commands:
 
 ```sh
 ./build/simdeck list
+./build/simdeck use <udid>
 ./build/simdeck boot <udid>
-./build/simdeck shutdown <udid>
-./build/simdeck erase <udid>
-./build/simdeck install <udid> /path/to/App.app
-./build/simdeck uninstall <udid> com.example.App
-./build/simdeck open-url <udid> https://example.com
-./build/simdeck launch <udid> com.apple.Preferences
-./build/simdeck pasteboard set <udid> "hello"
-./build/simdeck pasteboard get <udid>
-./build/simdeck screenshot <udid> --output screen.png
-./build/simdeck screenshot <udid> --with-bezel --output screen-bezel.png
-./build/simdeck record <udid> --seconds 5 --output screen-recording.mp4
-./build/simdeck describe <udid>
-./build/simdeck tap <udid> 120 240
-./build/simdeck tap <udid> --label "Continue" --wait-timeout-ms 5000
-./build/simdeck swipe <udid> 200 700 200 200
-./build/simdeck gesture <udid> scroll-down
-./build/simdeck pinch <udid> --start-distance 160 --end-distance 80
-./build/simdeck rotate-gesture <udid> --radius 100 --degrees 90
-./build/simdeck key-sequence <udid> --keycodes h,e,l,l,o
-./build/simdeck key-combo <udid> --modifiers cmd --key a
-./build/simdeck type <udid> "hello"
-./build/simdeck button <udid> lock --duration-ms 1000
-./build/simdeck home <udid>
+./build/simdeck shutdown
+./build/simdeck erase
+./build/simdeck install /path/to/App.app
+./build/simdeck uninstall com.example.App
+./build/simdeck open-url https://example.com
+./build/simdeck launch com.apple.Preferences
+./build/simdeck pasteboard set "hello"
+./build/simdeck pasteboard get
+./build/simdeck screenshot --output screen.png
+./build/simdeck screenshot --with-bezel --output screen-bezel.png
+./build/simdeck record --seconds 5 --output screen-recording.mp4
+./build/simdeck describe --format agent --max-depth 4 -i
+./build/simdeck wait-for --label "Welcome" --timeout-ms 5000
+./build/simdeck tap 120 240
+./build/simdeck tap --label "Continue" --wait-timeout-ms 5000
+./build/simdeck tap "Continue"
+./build/simdeck tap --id com.apple.settings.screenTime --expect-id BackButton
+./build/simdeck back
+./build/simdeck swipe 200 700 200 200
+./build/simdeck gesture scroll-down
+./build/simdeck pinch --start-distance 160 --end-distance 80
+./build/simdeck rotate-gesture --radius 100 --degrees 90
+./build/simdeck key-sequence --keycodes h,e,l,l,o
+./build/simdeck key-combo --modifiers cmd --key a
+./build/simdeck type "hello"
+./build/simdeck button lock --duration-ms 1000
+./build/simdeck home
 ```
 
+Most simulator commands accept `[<udid>]`; when it is omitted, SimDeck uses
+`--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the saved project default, or the
+only booted simulator, in that order. For agent navigation, prefer
+`describe -i`, `wait-for`, `tap --id/--label`, `tap "Text"`, `back`, and
+`batch` over coordinate-only loops.
+
 ## Expectations For Future Changes
 
 - If you add an API route, add the matching client affordance or document why it stays CLI-only.

README.md

@@ -94,131 +94,89 @@ CLI commands automatically use the same warm daemon:
 
 ```sh
 simdeck list
-simdeck tap <udid> 0.5 0.5 --normalized
-simdeck describe <udid> --format agent --max-depth 2
-```
-
-## Daemon
-
-Manage the project daemon explicitly when needed:
-
-```sh
-simdeck daemon start
-simdeck daemon restart
-simdeck daemon status
-simdeck daemon stop
-simdeck daemon killall
-```
-
-`simdeck daemon` manages the normal per-project warm process. `daemon killall`
-stops SimDeck daemons across all workspaces.
-
-Use software H.264 when the hardware encoder is unavailable, busy, or starved
-by screen recording:
-
-```sh
-simdeck daemon start --video-codec software
-```
-
-Restart the CoreSimulator service layer when `simctl` reports a stale service
-version or the live display gets stuck before the first frame:
-
-```sh
-simdeck core-simulator restart
-```
-
-You can also start or stop the CoreSimulator service layer explicitly:
-
-```sh
-simdeck core-simulator start
-simdeck core-simulator shutdown
+simdeck use <udid>
+simdeck tap 0.5 0.5 --normalized
+simdeck tap "Continue"
+simdeck describe --format agent --max-depth 2 --interactive
+simdeck press @e3
+simdeck snapshot --format agent --max-depth 2 -i
+simdeck --device <other-udid> describe --format agent --max-depth 2
 ```
 
 ## CLI
 
 ```sh
 simdeck list
+simdeck use <udid>
 simdeck boot <udid>
-simdeck shutdown <udid>
-simdeck erase <udid>
-simdeck install <udid> /path/to/App.app
-simdeck install <udid> /path/to/App.ipa
+simdeck shutdown
+simdeck erase
+simdeck install /path/to/App.app
+simdeck install /path/to/App.ipa
 simdeck install android:<avd-name> /path/to/app.apk
-simdeck uninstall <udid> com.example.App
-simdeck open-url <udid> https://example.com
-simdeck launch <udid> com.apple.Preferences
-simdeck toggle-appearance <udid>
-simdeck pasteboard set <udid> "hello"
-simdeck pasteboard get <udid>
-simdeck screenshot <udid> --output screen.png
-simdeck screenshot <udid> --with-bezel --output screen-bezel.png
-simdeck record <udid> --seconds 5 --output screen-recording.mp4
-simdeck stream <udid> --frames 120 > stream.h264
-simdeck describe <udid>
-simdeck describe <udid> --format agent --max-depth 4
-simdeck describe <udid> --point 120,240
-simdeck wait-for <udid> --label "Welcome" --timeout-ms 5000
-simdeck assert <udid> --id login.button --source auto --max-depth 8
-simdeck tap <udid> 120 240
-simdeck tap <udid> --label "Continue" --wait-timeout-ms 5000
-simdeck swipe <udid> 200 700 200 200
-simdeck gesture <udid> scroll-down
-simdeck pinch <udid> --start-distance 160 --end-distance 80
-simdeck rotate-gesture <udid> --radius 100 --degrees 90
-simdeck touch <udid> 0.5 0.5 --phase began --normalized
-simdeck touch <udid> 120 240 --down --up --delay-ms 800
-simdeck key <udid> enter
-simdeck key-sequence <udid> --keycodes h,e,l,l,o
-simdeck key-combo <udid> --modifiers cmd --key a
-simdeck type <udid> "hello"
-simdeck type <udid> --file message.txt
-simdeck button <udid> lock --duration-ms 1000
-simdeck button <udid> volume-up
-simdeck button <udid> action --duration-ms 1000
-simdeck button <udid> digital-crown
-simdeck crown <udid> --delta 50
-simdeck button <udid> left-side-button
-simdeck batch <udid> --step "tap --label Continue" --step "type 'hello'" --step "wait-for --label hello"
-simdeck dismiss-keyboard <udid>
-simdeck button <udid> software-keyboard
-simdeck home <udid>
-simdeck app-switcher <udid>
-simdeck rotate-left <udid>
-simdeck rotate-right <udid>
-simdeck chrome-profile <udid>
-simdeck logs <udid> --seconds 30 --limit 200
-simdeck processes <udid>
-simdeck stats <udid> --watch
-simdeck sample <udid> --seconds 3
+simdeck uninstall com.example.App
+simdeck open-url https://example.com
+simdeck launch com.apple.Preferences
+simdeck toggle-appearance
+simdeck pasteboard set "hello"
+simdeck pasteboard get
+simdeck screenshot --output screen.png
+simdeck screenshot --with-bezel --output screen-bezel.png
+simdeck record --seconds 5 --output screen-recording.mp4
+simdeck stream --frames 120 > stream.h264
+simdeck describe
+simdeck describe --format agent --max-depth 4
+simdeck describe --format agent --max-depth 4 --interactive
+simdeck snapshot --format agent --max-depth 4 -i
+simdeck describe --point 120,240
+simdeck wait-for --label "Welcome" --timeout-ms 5000
+simdeck wait --label "Welcome" --timeout-ms 5000
+simdeck assert --id login.button --source auto --max-depth 8
+simdeck tap 120 240
+simdeck tap --label "Continue" --wait-timeout-ms 5000
+simdeck tap --id com.apple.settings.screenTime --expect-id BackButton
+simdeck tap "Continue"
+simdeck press @e3
+simdeck back
+simdeck swipe 200 700 200 200
+simdeck gesture scroll-down
+simdeck pinch --start-distance 160 --end-distance 80
+simdeck rotate-gesture --radius 100 --degrees 90
+simdeck touch 0.5 0.5 --phase began --normalized
+simdeck touch 120 240 --down --up --delay-ms 800
+simdeck key enter
+simdeck key-sequence --keycodes h,e,l,l,o
+simdeck key-combo --modifiers cmd --key a
+simdeck type "hello"
+simdeck type --file message.txt
+simdeck button lock --duration-ms 1000
+simdeck button volume-up
+simdeck button action --duration-ms 1000
+simdeck button digital-crown
+simdeck crown --delta 50
+simdeck button left-side-button
+simdeck batch --step "tap --label Continue --expect-label Done" --step "type 'hello'" --step "back"
+simdeck dismiss-keyboard
+simdeck button software-keyboard
+simdeck home
+simdeck app-switcher
+simdeck rotate-left
+simdeck rotate-right
+simdeck chrome-profile
+simdeck logs --seconds 30 --limit 200
+simdeck processes
+simdeck stats --watch
+simdeck sample --seconds 3
 ```
 
 `simdeck list` defaults to compact JSON for agent-friendly device selection.
 Use `simdeck list --format json` for the full inventory with paths and display
 metadata.
 
-`boot` uses SimDeck's private CoreSimulator boot path so it can start devices
-without launching Simulator.app. If that private path is unavailable, the
-command returns the CoreSimulator error instead of falling back to
-`xcrun simctl boot`.
-
-Android emulators appear in `simdeck list` with IDs like
-`android:SimDeck_Pixel_8_API_36`. For Android IDs, lifecycle, install, launch,
-URL, screenshot, logs, UIAutomator `describe`, tap, swipe, text, key, home, app
-switcher, rotation, pasteboard, and browser live view route through the Android
-SDK tools (`emulator` and `adb`) plus the emulator gRPC screenshot stream for
-live video. `simdeck stream` remains iOS-only because it writes the iOS H.264
-transport stream.
-
-`stream` writes an Annex B H.264 elementary stream to stdout for diagnostics or
-external tools such as `ffplay`.
-
-`describe` uses the project daemon to prefer React Native, NativeScript,
-Flutter, or UIKit in-app inspectors, then falls back to the built-in private
-CoreSimulator accessibility bridge. Use `--format agent` or
-`--format compact-json` for
-lower-token hierarchy dumps. Coordinate commands accept screen coordinates from
-the accessibility tree by default; pass `--normalized` to send `0.0..1.0`
-coordinates directly.
+`simdeck use <udid>` stores a default simulator for the current project
+directory. Most device commands accept `[<udid>]`; when it is omitted, SimDeck
+uses `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the saved project default,
+or the only booted simulator, in that order.
 
 ## JS/TS Tests
 
@@ -240,7 +198,11 @@ try {
 `connect()` starts the project daemon when needed, reuses it when it is already
 healthy, and only stops daemons it started itself. Pass `udid` to `connect()`
 to make it the default for session methods; each method still accepts an
-explicit UDID as the first argument when needed.
+explicit UDID as the first argument when needed. Query helpers such as
+`tree()`, `query()`, `waitFor()`, `assert()`, and selector `tapElement()`
+default to `source: "native-ax"` for fast agent control; pass
+`source: "auto"` when a test intentionally wants richer framework inspector
+trees first.
 
 ## NativeScript Inspector
 
@@ -297,13 +259,6 @@ void main() {
 }
 ```
 
-## VS Code
-
-Install the `nativescript.simdeck-vscode` extension from the VS Code Marketplace, then
-run `SimDeck: Open Simulator View` from the Command Palette. The extension
-opens the simulator inside a VS Code panel and auto-starts the local daemon
-when it is not already reachable.
-
 ## Contributing
 
 Contributors should read [CONTRIBUTING.md](CONTRIBUTING.md) for local build

cli/XCWAccessibilityBridge.h

@@ -7,6 +7,7 @@ NS_ASSUME_NONNULL_BEGIN
 + (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udid
                                                          atPoint:(nullable NSValue *)pointValue
                                                         maxDepth:(NSUInteger)maxDepth
+                                                 interactiveOnly:(BOOL)interactiveOnly
                                                            error:(NSError * _Nullable * _Nullable)error;
 
 @end