Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

Git

OBS Studio Version (Other)

recent master (30.2.2-257-g06642fdee)

OBS Studio Log URL

https://obsproject.com/tools/analyzer?log_url=https://obsproject.com/logs/zNvvtlxOY9TNrv1L

OBS Studio Crash Log URL

No response

Expected Behavior

OBS should recover/clean up to a working state after any encoder times out when using the custom (hybrid MP4) muxer.

Current Behavior

After an encoder times out using Hybrid MP4...

Any subsequent recording silently fails to actually record any frames, and hangs. Regardless of if the second recording uses Hybrid MP4, regular MP4, or Fragmented MP4. (Other containers not tested by me.)

(Although the second recording does appear to start in the UI, stats do not increment encoded frames, trying to stop the recording doesn't work/hangs, the second recording ends up being 0 bytes, and exiting OBS results in a straggler thread with a recording icon in Task Manager, which has to be closed via Task Manager or else OBS will say another instance is running on next start.)

Steps to Reproduce

1. Select Hybrid MP4 as the container.
2. Select encode settings that are too heavy and will time out. I used SVT-AV1 Preset 6 at 1080p120Hz to reliably reproduce this issue. (I used a game capture source with active game content to make the scene more complex so the encoder would work harder.)
3. Start a recording, wait for the "Recording error" popup about an encoder timing out to appear (which automatically ends the first recording).
4. Start a second recording, without closing OBS first.

Anything else we should know?

This is a recent build of OBS. I grabbed the build from official OBS CI on GitHub. The 30.2.3 patch version happened outside of master branch, so this build's version string still says 30.2.2, but this build is from recent master and has all the fixes from 30.2.3 and then some. Just the version string isn't bumped to 30.2.3.

EDIT to add: FWIW, on release 30.2.3 these steps to reproduce simply produce a crash during the first recording. So at least hanging is already an improvement over crashing.

EDIT 2: Happens on latest commit 4dd3cf6 as well.