perf(recording): let ScreenCaptureKit scale Studio captures above 4096px

[AatmanAJ]

On macOS, Studio mode only bounds the ScreenCaptureKit capture size when the camera is active in Compatibility quality. With a capture source larger than 4096 px (a 5K Studio Display, a 6K Pro Display XDR, or an area larger than that), the problem starts with the frames ScreenCaptureKit delivers:

1. ScreenCaptureKit delivers full-size frames.
2. calculate_gpu_compatible_size asks the encoder for 4096 px output.
3. The encoder scales each frame with swscale (bicubic) on the CPU.
4. Because the input and output sizes differ, build_videotoolbox_hw_input refuses, so these recordings never get zero-copy VideoToolbox input.

This PR bounds the Studio capture to H264_MAX_DIMENSION on macOS, as Instant mode already does through max_capture_size. ScreenCaptureKit then does the scaling on the GPU. The encoded size doesn't change: a new test checks that the bounded capture size matches calculate_gpu_compatible_size for 5K, 6K, ultrawide and portrait sources. Windows and Linux are untouched.

Results

I measured this with a synthetic benchmark, a variant of examples/vt-hwframe-encode-benchmark.rs. It encodes 300 frames of moving IOSurface-backed NV12 through SegmentedVideoEncoder with the Studio Ultrafast preset, paced at the target fps, on an M5 Max running macOS 27.

Case	Encoder path	Queue time p50	CPU per frame
5120×2880 → 4096×2304 @ 30, before	swscale + software-frame VideoToolbox	17.3 ms	17.2 ms
4096×2304 @ 30, after	zero-copy VideoToolbox	0.96 ms	0.87 ms
5120×2880 → 4096×2304 @ 60, before	swscale + libx264	18.7 ms	51.4 ms
4096×2304 @ 60, after	libx264	4.7 ms	42.5 ms

At 60 fps the old path spends 18.7 ms per frame on the encoder thread, over the 16.7 ms frame budget, so it can't keep up. At 4096×2304 and 60 fps, both before and after are above estimate_hw_encoder_max_fps, so libx264 is used either way; that part isn't changed. The benchmark doesn't count ScreenCaptureKit's own GPU scaling, which runs before frames reach Cap.

Tests

• studio_capture_is_bounded_to_the_h264_limit_on_macos. Checked by mutation: it fails when the bound is removed.
• h264_bounded_capture_matches_the_encoder_downscale.
• cargo test -p cap-recording --lib: 564 passed.
• cargo clippy -p cap-recording --all-targets is clean.

Hand-tested

I tested this on a debug GPUI build with this change, on macOS 27.0.1 with an M5 Max, using the built-in 3024×1964 display (below the bound). A 12.7 s Studio area recording (a 2694×1798 capture, 570 frames) recorded and finalized normally, and the editor played it back at 59.9 fps with no dropped frames.

Not yet tested

I don't have a display larger than 4096 px, so the path this PR changes, a real recording on a 5K or 6K display, hasn't been run.

Confidence Score: 5/5

The PR appears safe to merge; no actionable failures were found.

Summary

Studio screen captures on macOS now use H264_MAX_DIMENSION as their capture-size limit. This lets ScreenCaptureKit scale large sources before encoding.

• Studio captures on macOS stay within the H.264 size limit.

No actionable issues were found. Review checks were based on code inspection; no builds, tests, or real large-display recordings were run.

_{Reviews (1) · Last reviewed commit: "perf(recording): let ScreenCaptureKit sc..." · Reviewed by Greptile}