WebRTC: Off-sync audio/video in DVR files #3993

S0mbre · 2024-03-18T03:23:30Z

Describe the bug
In SRS Stack 5, enabling DVR with any settings (including default) for the default vhost results in off-sync audio/video files when trying to record WebRTC streams. The problem is especially conspicuous for small video fragments (10 sec or less).

Version
SRS Stack 5 (latest) via Docker: ossrs/srs-stack:5

To Reproduce
Steps to reproduce the behavior:

Use the following bare-bones server config:

server_id           srs-server;
listen              1935;
max_connections     1000;
daemon              on;
disable_daemon_for_docker off;
utc_time            on;
ff_log_dir          /data/ffmpeg_logs;

rtc_server {
    enabled         on;
    listen          8000;          
    candidate       $CANDIDATE;
    protocol        udp;            
    reuseport       5;
    use_auto_detect_network_ip off;
    api_as_candidates off;    
}

vhost __defaultVhost__ {

    # General settings
    tcp_nodelay         on;
    min_latency         on;

    # WebRTC
    rtc {
        enabled         on;
        nack            on;
        twcc            on;
        rtmp_to_rtc     off;
        rtc_to_rtmp     on;
    }

    # Digital Video Recording
    dvr {
        enabled         on;
        dvr_apply       all;
        dvr_plan        session;
        dvr_path        /data/record/[app]/[stream].[timestamp].mp4;    # tried also ...flv, but no effect
        dvr_wait_keyframe       on;
        time_jitter             full;  # full / zero
    }
}

Create a WebRTC stream by publishing to https://MY-HOST.com/rtc/v1/whip/?app=live&stream=100&secret=SRS_PUB_SECRET
Go to your DVR path (/data/record/live) and download the resulting video(s), e.g. 100.TIMESTAMP.mp4
Observe off-sync audio/video.
Replace dvr_path in config for .....flv and do steps 2-5 above. Same issue remains.

Expected behavior
Expected fully-sync DVR files (mp4 or flv), with no data loss.

Additional context
For small videos, 10 sec or less, even video data is often lossy: the resulting DVR files get frozen or scrambled, with audio completely off-sync.

The text was updated successfully, but these errors were encountered:

winlinvip · 2024-03-18T09:21:54Z

Does this issues exists, when play the rtmp stream? You can play by ffplay or VLC, the url should be rtmp://MY-HOST.com/live/100 as such.

And, could you help to enable HLS, and check whether the HLS files is sync or not?

S0mbre · 2024-03-18T13:30:27Z

The HLS was in sync, but very, very slow. Actually, I couldn't manage to get HLS to play with satisfactory speed, so had to disable it. As to RTMP, I will check, but the live FLV streams are OK, video and audio is in sync. I also tried to switch to RTC via TCP, but it didn't help. There seems to be something wrong with transcoding from RTMP.

S0mbre · 2024-03-18T21:00:02Z

Update: I've checked RTMP - no sync issues found. So my previous guess appears right - something is wrong with DVR.

winlinvip · 2024-03-19T00:19:46Z

After converting RTC to RTMP, DVR muxes RTMP to FLV or MP4 file.

If RTMP is synced and FLV/MP4 is not synced, the most possible root cause may be:

Your video stream content is special and causes this problem. Please try publishing by Chrome in your PC, and check whether it works well.
The DVR logical is incorrect. Please try to record the RTMP to FLV by FFmpeg, and check whether the file is ok.

BTW: What's your WebRTC client? Did you try Chrome, Firefox, or Safari? Did you try another different PC, like Mac or Windows? This can help us to reproduce the problem.

S0mbre · 2024-03-19T02:56:10Z

I use Chrome as client, but the problem persists even when using mobile clients (including Android Chrome and Apple Safari). As t recording to FLV, what code should I add to srs.conf to do that? I suppose, it should be somewhere in transcode?

winlinvip · 2024-03-19T02:57:24Z

Not transcode of SRS, no need to change the config for SRS, you only need to record the RTMP by FFmpeg, for example:

ffmpeg -i rtmp://your-stream -c copy -f flv -y output.flv

Then you can check the output file.

S0mbre · 2024-03-19T03:04:55Z

Here's an example of off-sync DVR that I recorded just now.

livestream.1710817176050.1.mp4

S0mbre · 2024-03-19T03:05:42Z

As you can see, the audio is ahead of video.

S0mbre · 2024-03-19T03:13:30Z

Not transcode of SRS, no need to change the config for SRS, you only need to record the RTMP by FFmpeg, for example:
ffmpeg -i rtmp://your-stream -c copy -f flv -y output.flv
Then you can check the output file.

This method produces well-sync videos, I've checked both FLV and MP4 encoding. No issues found except for poor quality (maybe just my browser outputs a small picture size).

winlinvip · 2024-03-19T11:44:31Z

Thank you, I believe this issue is in the DVR module, not about streaming or even HLS.

S0mbre · 2024-03-20T21:50:26Z

OK, so the bug is accepted? I've managed to make DVR using manual ffmpeg in the stream on_publish callback. I suggest making the same change in the SRS code. The ffmpeg command I use:

ffmpeg -i rtmp://<MY-STREAM> -async 1 -c:v copy -c:a copy -y OUTFILE.mp4

The -async 1 is required to cope with out-of-sync audio.

winlinvip · 2024-03-21T00:51:29Z

@S0mbre Indeed, the bug is accepted, and I appreciate your assistance. :)

Regrettably, I cannot commit to a specific deadline for resolving this bug, as it depends on the developers' time availability and interest. In summary, due to my lack of time and interest in this issue, I will not be rectifying it.

Everyone is welcome to address this bug, and if you wish to do so, please submit a pull request. The reproduction steps and background information on this issue are crucial, so thank you for clarifying this bug.

green-cats · 2024-08-14T04:09:31Z

@winlinvip same issue

c2707637 · 2024-11-07T18:24:32Z

Same issue here.

suzp1984 · 2024-11-12T09:00:13Z

Cause

The cause is the incorrect first stts entry value.
stts entry in the audio and video track control the audio-video sync.

check the doc:

// 8.6.1.2 Decoding Time to Sample Box (stts), for Audio/Video.
// ISO_IEC_14496-12-base-format-2012.pdf, page 48

srs/trunk/src/kernel/srs_kernel_mp4.hpp

Lines 1573 to 1583 in 7951bf3

    
           // 8.6.1.2 Decoding Time to Sample Box (stts), for Audio/Video. 
        
           // ISO_IEC_14496-12-base-format-2012.pdf, page 48 
        
           // This box contains a compact version of a table that allows indexing from decoding time to sample number. 
        
           // Other tables give sample sizes and pointers, from the sample number. Each entry in the table gives the 
        
           // number of consecutive samples with the same time delta, and the delta of those samples. By adding the 
        
           // deltas a complete time-to-sample map may be built. 
        
           class SrsMp4DecodingTime2SampleBox : public SrsMp4FullBox 
        
           { 
        
           public: 
        
               // An integer that gives the number of entries in the following table. 
        
               std::vector<SrsMp4SttsEntry> entries;

#4230 fixed this issue, try it.

winlinvip added the EnglishNative This issue is conveyed exclusively in English. label Mar 18, 2024

winlinvip self-assigned this Mar 18, 2024

winlinvip added the WebRTC WebRTC, RTC2RTMP or RTMP2RTC. label Mar 18, 2024

winlinvip changed the title ~~Off-sync audio/video in DVR files~~ WebRTC: Off-sync audio/video in DVR files Mar 19, 2024

winlinvip added good first issue Easy to fix issues, good for newcomers help wanted Extra attention is needed labels Mar 21, 2024

suzp1984 added a commit to suzp1984/srs that referenced this issue Nov 12, 2024

issue ossrs#3993: fix av off sync for mp4 dvr.

8a9ab11

suzp1984 linked a pull request Nov 12, 2024 that will close this issue

fix av off sync for mp4 dvr. #4230

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebRTC: Off-sync audio/video in DVR files #3993

WebRTC: Off-sync audio/video in DVR files #3993

S0mbre commented Mar 18, 2024

winlinvip commented Mar 18, 2024 •

edited

Loading

S0mbre commented Mar 18, 2024 •

edited

Loading

S0mbre commented Mar 18, 2024

winlinvip commented Mar 19, 2024 •

edited

Loading

S0mbre commented Mar 19, 2024

winlinvip commented Mar 19, 2024 •

edited

Loading

S0mbre commented Mar 19, 2024

S0mbre commented Mar 19, 2024

S0mbre commented Mar 19, 2024 •

edited

Loading

winlinvip commented Mar 19, 2024

S0mbre commented Mar 20, 2024

winlinvip commented Mar 21, 2024

green-cats commented Aug 14, 2024

c2707637 commented Nov 7, 2024

suzp1984 commented Nov 12, 2024

WebRTC: Off-sync audio/video in DVR files #3993

WebRTC: Off-sync audio/video in DVR files #3993

Comments

S0mbre commented Mar 18, 2024

winlinvip commented Mar 18, 2024 • edited Loading

S0mbre commented Mar 18, 2024 • edited Loading

S0mbre commented Mar 18, 2024

winlinvip commented Mar 19, 2024 • edited Loading

S0mbre commented Mar 19, 2024

winlinvip commented Mar 19, 2024 • edited Loading

S0mbre commented Mar 19, 2024

S0mbre commented Mar 19, 2024

S0mbre commented Mar 19, 2024 • edited Loading

winlinvip commented Mar 19, 2024

S0mbre commented Mar 20, 2024

winlinvip commented Mar 21, 2024

green-cats commented Aug 14, 2024

c2707637 commented Nov 7, 2024

suzp1984 commented Nov 12, 2024

Cause

winlinvip commented Mar 18, 2024 •

edited

Loading

S0mbre commented Mar 18, 2024 •

edited

Loading

winlinvip commented Mar 19, 2024 •

edited

Loading

winlinvip commented Mar 19, 2024 •

edited

Loading

S0mbre commented Mar 19, 2024 •

edited

Loading