feat: bring the desktop editor to web by richiemcilroy · Pull Request #2312 · CapSoftware/Cap

richiemcilroy · 2026-09-18T10:14:58Z

Web recordings previously flattened the webcam over the screen, so the desktop editor could not reposition or remove it. This PR records and stores the screen and camera as separate synchronized clips in both the web recorder and Chrome extension, then opens the existing Solid desktop editor in a same-origin Next.js view. A media worker runs the native preview and exporter. Existing single-track recordings still open.

Web captions are Cap Pro-only. They use the existing AssemblyAI/shareable-link transcription setup, reuse a compatible completed share transcript, and transcribe through the same AssemblyAI options when a separate pass is needed. The UI, API, caption job, save, and worker boundaries enforce the Pro gate. The desktop app and its downloaded local caption models are unchanged.

What this includes

Shared editor panels and styling for timeline/clip edits, camera effects and removal, backgrounds and crop, cursor/zoom controls when source metadata exists, music and audio, captions, image/video overlays, scenes and 3D, project bundles, and MP4/GIF/MOV export.
Web Studio Sound defaults now persist in authenticated account preferences. The web recorder and Chrome extension snapshot the setting when each instant recording is created; the native worker applies it only to a new project, while saved projects keep their audio settings.
Distinct screen, camera, microphone, system-audio, and available tab-input sources with start offsets; bounded recovery for interrupted uploads; authenticated saves, revisions, worker sockets, imports, and downloads.
Lazy-loaded editor panels, validated worker-owned project roots and media paths, isolated subprocess credentials, and one active editor per worker process with bounded waiting when the pool is full.

Verification

A fresh production Solid build and headed Chromium Free/WebKit Pro replays passed separate camera controls, crop, image import, recording-bundle download, caption plan transitions, playback, and export preview. Both had no page errors, failed responses, or invalid frame closes. Pro caption generation used a fixture-backed web API response; this does not exercise live AssemblyAI.
A second headed replay selected Reupload in the same Solid view, rendered a native MP4, streamed it through the browser multipart client, and checked every uploaded byte, part size, ETag, and completion metadata against the worker export. Chromium and WebKit each passed the Free and Pro account flows with no page errors or failed responses. The storage endpoint was mocked, so this does not prove an authenticated hosted upload.
Headed cold-load replays in Chromium Free and WebKit Pro each delayed one skeleton and one editor asset. Loading controls stayed disabled until connection, then the full editor, caption gate/generation, and Pro playback recovered without page errors, failed responses, or invalid socket closes. This simulates slow asset delivery rather than a hosted load-time SLA.
Focused web-caption Vitest passed 34 cases across the AssemblyAI workflow, source timing/cache identity, client polling, and Free/Pro access, including reuse of a compatible share transcript and API denial for Free users.
A synthetic 7.93-second spoken AAC clip completed against the real AssemblyAI API using the same options helper as shareable-link transcription. It produced 27 English words and caption VTT through the existing edit-transcript formatter. This verifies the upstream API/helper, not the authenticated editor job or customer media.
Ten extension Vitest cases pass input-batch validation, Chrome tab-to-video pointer coordinates, pause/resume timing, and v4 collector IDs when randomUUID() is unavailable. A Chromium content-script replay compiled from the exact PR source started and stopped input capture on real HTTP and HTTPS tabs, sending four trusted pointer events per tab. The production extension build and typecheck against exact PR package types pass. This covers current-tab metadata, not OS-global input capture for arbitrary apps.
The built MV3 extension passed four focused Chromium E2E scenarios: separate camera sidecar upload, separate clip downloads after camera completion failure, spool recovery after offscreen closure, and a camera multipart part of at least 5 MiB under a simulated slow upload. An exact-source offscreen Chromium replay separately encoded screen and camera VP9 clips at about 30 FPS; pause/resume kept their frame timelines aligned without compositing. These use a mock backend and do not prove hosted authenticated capture. The exact-head Linux CI job rebuilt the extension and passed those four scenarios in 1.8 minutes; the full local MV3 suite passed 10, with four headed-only tests skipped.
A trusted-event Chromium replay bundled the exact extension content source and omitted pointer metadata for 16 sensitive, editable, and recorder-overlay targets, including formless numeric inputs, custom numeric widgets, SVG controls, open and closed shadow-root controls, and payment forms, while keeping ordinary button and shipping-form clicks. The published 3d4761ba86a86e8b71dbd453051a30442c86feb2 filter already excluded the formless numeric input; this expanded replay fails against that head because of the custom-control cases. Eight focused input/coordinate tests, extension typecheck, the production extension build, Biome, and workflow actionlint pass locally. The current CI runs the privacy replay before four packaged extension scenarios.
On the immediately preceding head b2bdc2f6cda0af516314017495133bf9f90c5bbd, the paired-source native route passed preview, thumbnail decoding, shutdown, and live Free→Pro→Free caption access on one worker session and command socket with 237 assertions. The Pro update enabled caption config and memory writes without reconnecting; the Free update denied both and hid caption loads. Next type generation and the full repository tsc -b passed in an isolated exact-head snapshot. That head’s CI Typecheck, amd64 and arm64 workers, paired web/extension capture, and Vercel checks also passed. The web bridge suite passed two tests, including the font-picker command, and the worker command-socket suite passed ten. Service Rust format, check, denied-warning Clippy, and root tests passed. Rust CodeQL reports zero path-injection results; the five earlier PR-head path alerts are fixed.
On earlier head ee5c71fd213d29c5f0370b644a185c1cb637ebb9, a Pro→Free plan update closes the open transcript, clears caption selection, and hides the header transcript action while preserving saved paid captions for a future upgrade. The isolated two-file change passed scoped Biome, Next type generation plus full tsc -b, ten Solid bridge tests, and five caption-access tests. Exact-head GitHub Typecheck, amd64 and arm64 editor-worker jobs, and Vercel passed. The current Preview loaded its sessionless editor shell with disabled controls and no console warnings or errors; hosted editing still needs a configured worker.
This follow-up also passed a real native legacy replay that reopened a saved project without changing its audio settings, a Store reload/partial-update/failed-save test, a malformed-preference test, the full monorepo typecheck after Next type generation, Rust denied-warning Clippy, and a fresh production Solid build. The standalone media-server typecheck still reports unrelated pre-existing test typing errors. All 41 applicable CI checks passed on the exact head 82c387bd7c16ec6747d0e90fb2c022fffe58b82d; four checks were skipped. The passing set includes macOS, Linux, and Windows desktop app builds, both editor-worker architectures, web and extension capture, platform sync tests, and the final Vercel Preview.
In a local 1920×1080 screen plus separate 1280×720 camera fixture at simulated 5 Mb/s, Chromium painted 600 frames in 20 seconds (30 FPS), with no frame gaps or browser long tasks; native preparation took 515 ms. This does not establish sustained performance on a hosted worker.
The previous-head Vercel deployment dpl_ANSHi9ZxYuEV6f5FpF43vUondr65 is READY. An in-app browser loaded /editor-solid/index.html from that deployment with no console warnings or errors. The then-current Preview deployment dpl_9RW7heKsyUfroKwHBBb1gzdWoq6W is also READY: an in-app browser loaded its editor skeleton with controls disabled before a session and no console warnings or errors, and the shell served HTTP 200. These checks verify asset delivery; they do not exercise a signed-in edit or hosted worker.
On the previous PR head 1cf35fa325456d38c136cfe4513f871943f1e920, amd64 and arm64 editor-worker CI passed. The amd64 job completed six native Reupload sessions: Free/Pro in Chromium and WebKit, plus two further WebKit Free replays. Each matched native MP4 multipart bytes with no failed responses or page errors. Recording Reliability passed both web-recorder engines, recording import and recovery, the 16-target Chrome privacy replay, and four packaged extension scenarios. One initial Chrome attempt lost its MV3 start message port before tab capture; the exact-head failed-job rerun and a separate local targeted run passed. All 41 applicable checks passed on that head, including GPUI tests and final Tauri app builds on Linux, Windows, and macOS; four workflow checks were skipped.
On head bc92dbda0800fcc00ce3b56c125ce51866a9869f, a fresh Solid production build and native worker replay in Chromium and WebKit proved that an open Pro transcript closes on downgrade, SRT/VTT and the header caption action disappear, the Free gate denies generation and styling while preserving saved paid content, and re-upgrade restores the action without reload. Free and Pro runs in both engines completed native MP4 Reupload with no page errors or failed responses. Scoped Biome and a targeted replay-file TypeScript check passed. A media-server-wide TypeScript run reports four existing errors in unrelated test files.
At the exact bc92dbda0800fcc00ce3b56c125ce51866a9869f head, the 15-minute synthetic 49-screen/49-camera clip benchmark on a local Apple M4 Max prepared in 3.16 seconds. Ten timeline seeks had a 19.5 ms median and 29.8 ms p95; native editor RSS moved from 279.0 to 288.7 MB. The repeated 640×360 sources test long timelines and clip count, not representative 1080p footage or a hosted worker.
With looped 1920×1080 screen and separate 1280×720 webcam sources, Chromium painted 60.3 fps at CDP-emulated 20 Mbps and 61.1 fps at 10 Mbps; unthrottled Playwright WebKit painted 60.2 fps. All three 20-second H.264 replays completed a live seek with no browser long tasks or packet sequence gaps, and native playback RSS stayed near 740–743 MB during the sample. Paused browser pixels matched the native SHA-256 hash. WebKit moving frames measured 27.8–27.9 dB RGB PSNR and 2.3 mean absolute RGB error on a saturated motion fixture because H.264 playback is lossy. These local engine and fixture results do not prove sustained capacity or exact moving-frame pixels on the eventual host.
On head c7c56800bfd3633a2fe724931335aa67d117115f, the paired browser recorder passed eight Chromium/WebKit scenarios. An injected microphone-sidecar resume error returned screen, camera, and microphone to paused; retry then uploaded all three clips with offsets. The WebKit run also completed after a stalled IndexedDB camera backup switched to bounded memory, and upload confirmation was no longer held by optional backup deletion. Recorder-core passed 78 tests, full workspace TypeScript and extension typecheck passed, and the production extension build plus normal separate-camera/microphone Chromium E2E passed. A test-only fault in an isolated extension build made resume fail after screen and camera restarted; both clips still uploaded.
On head 34af945b39a0845321e9bc8acf4e1da313c2ee5d, the web recorder and Chrome extension mark each confirmed clip upload outside IndexedDB before optional backup deletion. An uploaded backup is omitted from orphan recovery even if IndexedDB deletion stalls; successful deletion clears the marker. Ten real Chromium/WebKit paired browser scenarios passed, including screen/camera/microphone resume retry, a failed camera completion, and injected backup deletion. Chromium then recovered no uploaded orphans and reclaimed their markers; WebKit kept its uploaded marker while its IndexedDB cleanup was unavailable. Recorder-core passed 79 tests, the isolated PR-tree workspace typecheck and extension production build passed, and normal camera/microphone extension E2E passed. A test-only extension backup-deletion fault still reached completed with both source uploads confirmed and a durable uploaded marker. The WebKit recovery database read itself timed out in an exploratory same-page probe, so this is recovery-filter and upload-completion proof, not a guarantee that Safari storage is always responsive after recording.
Local 15-minute paired-source Chromium and WebKit performance replays painted about 60 frames per second through H.264 at the requested 60 fps. Chromium also passed 10/20 Mbps throttled playback, seek, paused native-pixel hash equality, no-long-task, and 20-second memory checks. The eight-CPU Linux software-rendered camera-removal replay reached 52.0 fps; four CPUs reached 31.1 fps with removal. These fixture results are documented with repeat commands in apps/media-server/editor-worker.md; sustained performance and capacity on the eventual hosted worker remain unmeasured.
On head 8470b89c9441e7351afe1b8e5841654e93ab79c9, the extension options page excludes confirmed-upload spool markers before presenting interrupted recordings and also groups tab-input sidecars with their parent recording for recovery and removal. The exact PR-tree extension typecheck, production build, and scoped Biome check passed. Three packaged Chromium MV3 scenarios passed: the new uploaded-marker options UI case plus two genuine crash-recovery regressions. These use extension-local fixture media and a mock backend, so they do not prove hosted upload recovery.
On head 0bb337c85bccb72e9e660f2f9ffcf93130224edb, the recording-reliability workflow now selects the uploaded-backup options UI scenario alongside its existing paired-capture and crash-recovery cases. The exact five-case packaged Chromium filter passed locally in 1.7 minutes, and scoped actionlint accepted the workflow. All named exact-head GitHub checks completed: 40 successful, two neutral, two skipped, and none failed; the separate Vercel deployment status is successful. The first macOS sync attempt exceeded tolerance in one unchanged synthetic 1,000-fps overload case (250 ms), while its failed-job retry passed all 41 cases and measured 40 ms in that case. Hosted worker and customer-media validation remain open.
On head 32e7ad6, the web Solid caption transport reuses the source/track hash during camera and style edits and invalidates it for word edits, track edits, and added captions. Twelve browser-condition bridge tests, scoped typecheck, Biome, actionlint, and the Solid production build passed. A local 2.67 MB caption fixture in Chromium and WebKit averaged 0.008 ms and 0 ms for cached camera updates, versus 94.3 ms and 62.2 ms for fresh hashes. CI now replays the fixture in both engines. These measurements are local fixture results, not a hosted worker performance guarantee.
On exact head 6fa805246a668a3e2a7a1eff2e55cdca4b4d0ad8, amd64 and arm64 native worker checks passed. The amd64 job ran six browser Reupload sessions plus Chromium Free and WebKit Pro cursor-only MOV exports. Both MOV downloads matched the worker SHA-256 and byte size, decoded as alpha ProRes, showed cursor motion in six and four sampled frames, and finished without page errors or failed responses; each one-use ticket returned 404 after use. Linux WebKit previously navigated to a QuickTime plug-in page despite the attachment header, so the direct MOV download now uses application/octet-stream and nosniff; the internal worker /file endpoint remains video/quicktime. Exact-source local browser replays, focused Studio Sound tests/build, scoped Biome, and the full exact-head Typecheck also passed.
On exact head 92e742e77ae687907383bda2405bf4cef57fd1f5, the amd64 editor-worker job ran Chromium Free and WebKit Pro with OffscreenCanvas, WebCodecs, createImageBitmap, and WebGPU disabled before the Solid view loaded. Both replays painted real preview pixels ([255, 0, 254, 255] and [0, 0, 255, 255]), passed crop ratios/themes and export preview, and had no page errors, failed responses, or invalid socket closes. Pro playback advanced in WebKit. Their single crop-frame loads were 267 ms and 541 ms in Linux software-rendered browser fallback; these samples are compatibility checks, not sustained frame-rate results. The exact-head amd64/arm64 worker checks, Typecheck, actionlint, scoped Biome, and local source replays passed.

Release gates and browser limits

The inspected Vercel project and shared settings list neither CAP_WEB_EDITOR_WORKER_URL nor CAP_WEB_EDITOR_WORKER_POOL in any environment. With no deployment-specific override, direct Studio preparation returns Service Unavailable because its worker pool is empty; the ordinary /edit route stays on the existing web editor until a worker URL or pool is configured. The READY preview serves the Solid shell but cannot open a live hosted editor. A signed-in paired recording → edit → export run, same-state native/web pixel comparison, and sustained performance on the eventual worker host remain open. Browser capture APIs cannot provide OS-global pointer positions or key events for arbitrary apps; the extension captures trusted current-tab input only. Web export downloads the file because browser clipboard APIs cannot guarantee desktop-style MP4 file paste/Open File behavior. Wallpaper selection uses an image file chooser. These limits prevent a literal every-detail parity claim today.

The accessible Railway Media Server production project currently has no editor-worker service, and its media-server service shows no editor-worker origin variable keys. The project-scoped token cannot enumerate other Railway projects, so this check is limited to that project.

Each worker needs a public origin and stable CAP_WEB_EDITOR_WORKER_ID. Next.js needs CAP_WEB_EDITOR_WORKER_URL or CAP_WEB_EDITOR_WORKER_POOL; the worker also uses existing MEDIA_SERVER_WEBHOOK_SECRET and CAP_WEB_EDITOR_PUBLIC_ORIGIN. Worker packaging is in apps/media-server/Dockerfile.editor and apps/media-server/railway.editor.toml, with setup documented in apps/media-server/editor-worker.md. No database migration is required.

Confidence Score: 4/5

The latest changes appear behaviorally safe, but the PR is not ready to merge while two explicit repository-convention findings remain unresolved.

Findings

Component Filename Violates Convention ▶
Narrative Comments Violate Policy ▶

Fix with agent prompt

### Issue 1
apps/desktop/src/routes/editor/CaptionsTab.tsx:undefined-89
This changed component remains in the PascalCase-named `CaptionsTab.tsx` file, but the repository guide requires filenames to use kebab-case. Rename the file and update its imports. The same violation appears in `ConfigSidebar.tsx`, `ExportPage.tsx`, and other modified desktop editor component files. This repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
apps/chrome-extension/src/content/overlay.tsx:852-853
These lines narrate the service worker’s recovery flow instead of documenting a non-obvious invariant, workaround, or investigation context. The repository defaults to no code comments for ordinary control flow and requires clearer naming or types instead. The same pattern appears in `apps/chrome-extension/src/background/service-worker.ts` and `apps/chrome-extension/src/offscreen/recorder.ts`. This repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The changes since the previous review add Chromium and WebKit coverage for the editor’s reduced-capability canvas-rendering path.

Adds a canvas-fallback replay mode that disables OffscreenCanvas, image-bitmap, WebCodecs, and WebGPU APIs.
Verifies that the editor still produces a non-empty preview pixel and completes its existing editing and export-preview flow.
Runs the new replay in both browser engines in the editor-worker CI job.

_{Reviews (79) · Last reviewed commit: "improve: test legacy canvas preview in e..."}

superagent-security

Superagent found 2 security concern(s).

richiemcilroy · 2026-09-18T12:14:34Z