Skip to content

Add video frame format render option#1142

Draft
xuelongmu wants to merge 1 commit into
heygen-com:mainfrom
xuelongmu:video-frame-format-png-extraction
Draft

Add video frame format render option#1142
xuelongmu wants to merge 1 commit into
heygen-com:mainfrom
xuelongmu:video-frame-format-png-extraction

Conversation

@xuelongmu
Copy link
Copy Markdown

Summary

Adds a first-class render option for source-video frame extraction:

hyperframes render --video-frame-format <auto|jpg|png>

and the matching programmatic config field:

videoFrameFormat?: "auto" | "jpg" | "png";

The default remains auto, preserving the existing behavior: alpha or alpha-capable video sources extract as PNG, and opaque sources extract as JPG.

Rationale

In an apple-devices-pip render, the timeline/browser preview looked close to the original iPhone recording, but the rendered MP4 shifted saturated UI reds. Changing final H.264 color tags/range did not fix it, and pixel checks showed the bad color was already present in HyperFrames' captured browser PNG frames.

The root cause was earlier in the pipeline: non-alpha source videos were extracted as JPEG frames before browser capture. That JPEG step visibly changed saturated UI colors before final encoding ever ran. In the reported case, the original iPhone frame sampled RGB(221,56,46) on the red "Elevated Risk" indicator, while HyperFrames' browser PNG frame sampled RGB(186,51,58). A direct ffmpeg source-video composite preserved the same pixel closely at RGB(220,56,50), showing the final H.264 encoder can preserve the color when the source video does not pass through JPEG extraction.

This PR adds an explicit, opt-in PNG extraction path for UI recordings, screen captures, and other color-sensitive source videos. It avoids an RGB lift or color-correction workaround and leaves final encoder defaults unchanged.

What Changed

  • Adds --video-frame-format auto|jpg|png to hyperframes render.
  • Adds RenderConfig.videoFrameFormat.
  • Threads the option through local render, Docker render, producer server render, and distributed planning.
  • Updates resolveFrameFormat so:
    • explicit png extracts opaque videos as PNG
    • explicit jpg extracts opaque videos as JPG
    • auto/undefined preserves existing behavior
    • alpha and alpha-capable codecs still force PNG
  • Ensures extraction cache entries use the effective frame format, so JPG and PNG frame caches cannot collide.
  • Documents the option in CLI and rendering docs.

Validation

  • bun run --filter @hyperframes/engine test -- src/services/videoFrameExtractor.test.ts
  • bun run --filter @hyperframes/cli test -- src/commands/render.test.ts src/utils/dockerRunArgs.test.ts
  • bun run --filter @hyperframes/aws-lambda test -- src/sdk/validateConfig.test.ts
  • bun run --filter @hyperframes/engine typecheck
  • bun run --filter @hyperframes/producer typecheck
  • bun run --filter @hyperframes/cli typecheck
  • bun run --filter @hyperframes/aws-lambda typecheck
  • bunx oxfmt --check
  • bunx oxlint

The new regression test synthesizes a tiny saturated-red UI fixture on a pale pink background and verifies PNG extraction keeps sampled red pixels within a max channel delta of 5 from the decoded source, while also proving JPG and PNG extraction caches remain separate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant