Deliver account screenshots to Nessie within the job result budget

[rafiki270]

Summary

A Nessie agent can now receive a screenshot from an account-connected Kelpie even when the viewport PNG is larger than one job result.

Before this change, Nessie's canonical tab.screenshot returned a viewport PNG. A photographic or large page goes over the connect ledger's 768 KiB output limit once base64-encoded. The ledger then refused the completion (kelpie_connect_execute returned null). The image never arrived:

• Desktop dropped the connection, and the attempt stayed reserved until a restart turned it into UNKNOWN_EFFECTS.

• Apple and Android reported a generic "Browser action failed".

• native/core-connect ledger: an oversized result now ends its attempt as a bounded failed event. The error starts with RESULT_TOO_LARGE and the output is discarded, so the attempt is no longer stranded. Local completions larger than one wire frame (1 MiB) are accepted so they can reach that limit. Wire frames are still capped at 1 MiB.

• ConnectScreenshotBudget (Apple, shared by macOS and iOS through apps/macos/Kelpie/Connect; Android connect/ConnectScreenshotBudget.kt):

  • Applies to account tab.screenshot dispatch (ConnectBrowserControlJob.dispatch, ConnectBrowserAccountControl).
  • When the base64 image exceeds 640 KiB, the screenshot is re-taken from the same pinned tab as JPEG at quality 80, then 60, then 40.
  • Each attempt goes back through the job's authorization, document and active-tab checks.
  • The native result reports its real format, so the gateway emits image/jpeg.
  • If no attempt fits, the request fails with SCREENSHOT_TOO_LARGE.

• Unchanged: local HTTP/MCP tab.screenshot behaviour and the gateway's canonical PNG-only request validation.

• Legacy snapshot + includeScreenshot: this path passed quality: 0.5 to handlers that take 0–100. Apple encoded at 0.5% and Android ignored the value. It now requests quality 80, reduced if needed under the same limit.

• Docs: docs/nessie-connect.md and docs/functionality.md.

• Versions: macOS 0.1.33 (27), iOS 0.1.18 (18), Android 0.1.17 (17), Windows 0.1.14, Linux 0.1.13.

• Windows/Linux (native/engine-chromium-desktop, native/desktop-connect):

  • Account tab.screenshot uses the same budget. The shared kelpie/screenshot_fit.h helper runs the JPEG ladder, enabled per server by Config::screenshot_base64_limit, which only the Windows and Linux account callbacks set.
  • Each retake re-runs step authorization inside the job's ScopedBrowserControlLease.
  • Chromium capture (BrowserScreenshot, DesktopDevToolsSession::ScreenshotParams, screenshot_handler.cpp) now supports CDP JPEG with an integer quality from 1 to 100. As a result, local /v1/screenshot and the kelpie_screenshot extension on Windows/Linux also accept format:"jpeg"/quality, matching docs/api/core.md.
  • Legacy includeScreenshot uses the same ladder instead of failing above 640 KiB.
  • Linux account jobs exist only in the CEF build, so the non-CEF linux_app_api.cpp PNG endpoint is unchanged.

• native/core-mcp gateway: a native SCREENSHOT_TOO_LARGE from a tab.screenshot step is returned as that canonical code (mayHaveCompleted:false) instead of COMMAND_FAILED, so every platform reports the same refusal.

The Nessie side that stores the image and shows it to the model is UnlikeOtherAI/Nessie#1197.

Testing

• kelpie_connect_test (host CMake/Ninja, Mac): new case where a result over 1 MiB terminates as failed/RESULT_TOO_LARGE, with terminalState persisted.
• macOS KelpieTests: full suite 203/203 passed. This includes new ConnectScreenshotBudgetTests and a real-WebKit test where a random-noise viewport comes back as image/jpeg within 640 KiB while a normal page stays image/png.
• iOS simulator (iPhone 17 Pro Max): ConnectScreenshotBudgetTests and ConnectAuthorizationTests, 14/14 passed.
• make lint-swift: 0 violations.
• Android: full ./gradlew build on Minis (ktlint, lint, unit tests including the new ConnectScreenshotBudgetTest) passed.
• Host CMake on the Mac: test_browser_gateway (including the new SCREENSHOT_TOO_LARGE passthrough, while other methods still map to COMMAND_FAILED), test_desktop_screenshot_fit, test_desktop_handlers, test_desktop_mcp_server and kelpie_connect_test all passed.
• Ubuntu umac (fresh clone, CEF 120 Release): full kelpie-linux build and 35/35 CTest cases passed on the desktop changes, before the gateway passthrough simplification.
• Windows Minis (fresh clone, scripts/build-windows.ps1, sandboxed CEF): full build and 53/54 CTest cases passed. The failing kelpie_windows_browser_view_test needs an interactive desktop session, and its win32_browser_view.cpp is not touched here.
• Not run: real CDP JPEG capture against a live page on Windows/Linux (covered only by the params test); a live screenshot through a deployed Nessie to an agent.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.