Unable to re-encode WebRTC input stream

[Aidan63]

Hello,

I've ran into an issue where if I'm inputting a WebRTC stream (via WHIP or the custom signalling protocol) I'm unable to transcode that input stream, e.g. with the following encodes element.

<Encodes>
    <Video>
        <Name>test</Name>
        <Codec>h264</Codec>
        <Bitrate>20000000</Bitrate>
        <Width>1280</Width>
    </Video>
</Encodes>

Any player which attempts to watch the stream will fail and a "stream has created but not started" warning along with a "Cannot find stream" error will show in the server log

[2024-03-03 20:50:05.346] I [SPRtcSig-t3333:5] HTTP.Server | http_server.cpp:149  | Client(<ClientSocket: 0x7f6397001410, #23, Connected, TCP, Nonblocking, 192.168.1.241:49379>) is connected on *:3333
[2024-03-03 20:50:05.347] I [SPRtcSig-t3333:5] Signalling | rtc_signalling_server.cpp:316  | New client is connected: <ClientSocket: 0x7f6397001410, #23, Connected, TCP, Nonblocking, 192.168.1.241:49379>
[2024-03-03 20:50:05.350] W [SPRtcSig-t3333:5] WebRTC Publisher | webrtc_publisher.cpp:340  | (#default#app/stream) stream has created but not started.
[2024-03-03 20:50:05.350] E [SPRtcSig-t3333:5] Signalling | rtc_signalling_server.cpp:407  | Cannot find stream [#default#app/stream]
[2024-03-03 20:50:05.351] I [SPRtcSig-t3333:5] Signalling | rtc_signalling_server.cpp:450  | Client is disconnected: HttpConnection(0x7f6399a00410) : WebSocket <ClientSocket: 0x7f6397001410, #23, Connected, TCP, Nonblocking, 192.168.1.241:49379> TLS(Disabled) (#default#app / stream, ufrag: local: (N/A), remote: (N/A))
[2024-03-03 20:50:05.351] I [SPRtcSig-t3333:5] HTTP.Server | http_server.cpp:216  | Client(<ClientSocket: 0x7f6397001410, #23, Closed, TCP, Nonblocking, 192.168.1.241:49379>) has been disconnected by *:3333

Interestingly if I switch from a WebRTC input to a RTMP input, everything works fine! Players can connect to the transcoded stream. If I remove that transcode and just do a video bypass instead then the WebRTC stream can be watched fine as well. In all cases the player is trying to connect to "ws://192.168.1.144:3333/app/stream" and the video stream is being input to "app/stream" via either the rtmp or webrtc input.

Looking at the server log when the webrtc input stream is setup shows the following

[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] Provider | stream.cpp:51   | Unknown/stream(100) has been started stream
[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] MediaRouter | mediarouter_application.cpp:340  | [#default#app/stream(100)] Trying to create a stream
[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] Monitor | application_metrics.cpp:58   | Create StreamMetrics(stream/b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/i) for monitoring
[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] MediaRouter | mediarouter_application.cpp:440  | [#default#app/stream(100)] Stream has been created
[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] Transcoder | transcoder_stream.cpp:63   | [#default#app/stream(100)] Transcoder stream has been started
[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] ICE | ice_port.cpp:368  | Added session: 0 (ufrag: gur3At:8+ey)
[2024-03-03 20:47:41.570] I [SPRtcSig-t3333:5] WHIP | whip_server.cpp:420  | WHIP Response: <HttpResponse> Status(201) Reason(Created) Header(10) Data(1300)

[Header]
Link: <turn:10.88.0.48:3478?transport=tcp>; rel="ice-server"; username="ome"; credential="airen"; credential-type="password"
Link: <turn:86.157.56.206:3478?transport=tcp>; rel="ice-server"; username="ome"; credential="airen"; credential-type="password"
ETag: CjnEArt8
Location: /app/stream/2FTVygLR?direction=whip
Access-Control-Allow-Headers: *
Access-Control-Allow-Methods: GET
Access-Control-Allow-Credentials: true
Vary: Origin
Access-Control-Allow-Origin: *
Content-Type: application/sdp
Server: OvenMediaEngine

[2024-03-03 20:47:41.571] I [SPRtcSig-t3333:5] HTTP.Server | http_server.cpp:223  | Client(<ClientSocket: 0x7f639d240a10, #23, Disconnected, TCP, Nonblocking, 192.168.1.241:49362>) has disconnected from *:3333
[2024-03-03 20:47:41.572] I [SPICE-u10004:9] ICE | ice_port.cpp:843  | Session 0 uses candidate: <SocketAddressPair: 0x7f6397dfd0a0, local: 10.88.0.48:10004, remote: 192.168.1.241:59240>
[2024-03-03 20:47:42.602] W [SPICE-u10004:9] Provider | stream.cpp:301  | Wait for RTCP SR for 5 seconds before starting the stream.
[2024-03-03 20:47:42.616] I [SPICE-u10004:9] Provider | stream.cpp:287  | Since this stream has received an RTCP SR, it counts the PTS with the SR.
[2024-03-03 20:47:44.612] I [InboundWorker:19] MediaRouter | mediarouter_application.cpp:477  | [#default#app/stream(100)] Stream has been prepared 
[Stream Info]
id(100), msid(0), output(stream), SourceType(WebRTC), RepresentationType(Source), Created Time (Sun Mar  3 20:47:41 2024) UUID(b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/i)

        Audio Track #-972228116: Public Name(Audio_3322739180) Variant Name(Audio) Bitrate(1.22Kb) Codec(8,OPUS,none) BSF(OPUS_RTP_RFC_7587) Samplerate(48.0K) Format(none, 0) Channel(stereo, 2) timebase(1/48000)
        Video Track #-972228115: Public Name(Video_3322739181) Variant Name(Video) Bitrate(0b) Codec(1,H264,none:0) BSF(H264_RTP_RFC_6184) Resolution(1920x1080) Framerate(0.00) KeyInterval(0/frame) SkipFrames(-1) BFrames(0) timebase(1/90000)
        Data  Track #1: Public Name(Data_1) Variant Name(Data) Codec(0,Unknown,none) BSF(Unknown) timebase(1/1000)
[2024-03-03 20:47:44.612] I [InboundWorker:19] Transcoder | transcoder_stream.cpp:128  | [#default#app/stream(100)] Using local output profiles by webhook
[2024-03-03 20:47:44.612] I [InboundWorker:19] Transcoder | transcoder_stream.cpp:410  | [#default#app/stream(100)] Output stream has been created. [#default#app/stream(3957146102)]
[2024-03-03 20:47:44.615] I [InboundWorker:19] Transcoder | transcoder_decoder.cpp:242  | The decoder has been created successfully. track(#-972228115) codec(H264), module(default:0)
[2024-03-03 20:47:44.615] I [InboundWorker:19] MediaRouter | mediarouter_application.cpp:340  | [#default#app/stream(3957146102)] Trying to create a stream
[2024-03-03 20:47:44.615] I [InboundWorker:19] Monitor | application_metrics.cpp:58   | Create StreamMetrics(stream/b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/o) for monitoring
[2024-03-03 20:47:44.615] I [InboundWorker:19] MediaRouter | mediarouter_application.cpp:440  | [#default#app/stream(3957146102)] Stream has been created
[2024-03-03 20:47:44.615] I [InboundWorker:19] Transcoder | transcoder_stream.cpp:150  | [#default#app/stream(100)] Transcoder stream has been prepared
[2024-03-03 20:47:44.625] I [Dech264:23] Transcoder | decoder_avc.cpp:210  | [#default#app/stream(100)] input track information: [video] h264 (Constrained Baseline 4.0), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 195 kbps, timebase: 1/60, 
[2024-03-03 20:47:44.625] I [Dech264:23] Transcoder | transcoder_stream_internal.cpp:613  | Framerate of the output stream is not set. set the estimated framerate from decoded frame duration. 0.50fps
[2024-03-03 20:47:44.626] W [Dech264:23] FFmpeg | third_parties.cpp:115  | [AVCodecContext: 0x7f638c506400] [OpenH264] this = 0x0x7f638c406060, Warning:bEnableFrameSkip = 0,bitrate can't be controlled for RC_QUALITY_MODE,RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.
[2024-03-03 20:47:44.637] I [Dech264:23] Transcoder | transcoder_encoder.cpp:278  | The encoder has been created successfully. track(#0), codec(H264), module(default:0)
[2024-03-03 20:47:44.638] I [Dech264:23] Transcoder | filter_rescaler.cpp:296  | Rescaler parameters. track(#3322739181 -> #0), module(default:0 -> default:0). desc(src:video_size=1920x1080:pix_fmt=yuv420p:time_base=1/90000:pixel_aspect=1/1:frame_rate=30/1 -> output:settb=1/90000,scale=1280x720:flags=bilinear,format=yuv420p)

and the rtmp input shows the following

[2024-03-03 20:52:28.419] I [SPRTMP-t1935:16] RTMPProvider | rtmp_provider.cpp:189  | A RTMP client has connected from <ClientSocket: 0x7f6391a01010, #23, Connected, TCP, Nonblocking, 192.168.1.241:49386>
[2024-03-03 20:52:29.552] I [SPRTMP-t1935:16] MediaRouter | mediarouter_application.cpp:340  | [#default#app/stream(102)] Trying to create a stream
[2024-03-03 20:52:29.553] I [SPRTMP-t1935:16] Monitor | application_metrics.cpp:58   | Create StreamMetrics(stream/b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/i) for monitoring
[2024-03-03 20:52:29.553] I [SPRTMP-t1935:16] MediaRouter | mediarouter_application.cpp:440  | [#default#app/stream(102)] Stream has been created
[2024-03-03 20:52:29.553] I [SPRTMP-t1935:16] Transcoder | transcoder_stream.cpp:63   | [#default#app/stream(102)] Transcoder stream has been started
[2024-03-03 20:52:29.558] I [InboundWorker:19] MediaRouter | mediarouter_application.cpp:477  | [#default#app/stream(102)] Stream has been prepared 
[Stream Info]
id(102), msid(0), output(stream), SourceType(Rtmp), RepresentationType(Source), Created Time (Sun Mar  3 20:52:28 2024) UUID(b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/i)

        Video Track #0: Public Name(Video_0) Variant Name(Video) Bitrate(2.50Mb) Codec(1,H264,none:0) BSF(AVCC) Resolution(1920x1080) Framerate(30.00) KeyInterval(0/frame) SkipFrames(-1) BFrames(0) timebase(1/1000)
        Audio Track #1: Public Name(Audio_1) Variant Name(Audio) Bitrate(160.00Kb) Codec(6,AAC,none) BSF(AAC_RAW) Samplerate(48.0K) Format(s16, 16) Channel(stereo, 2) timebase(1/1000)
        Data  Track #2: Public Name(Data_2) Variant Name(Data) Codec(0,Unknown,none) BSF(ID3v2) timebase(1/1000)
[2024-03-03 20:52:29.558] I [InboundWorker:19] Transcoder | transcoder_stream.cpp:128  | [#default#app/stream(102)] Using local output profiles by webhook
[2024-03-03 20:52:29.558] I [InboundWorker:19] Transcoder | transcoder_stream.cpp:410  | [#default#app/stream(102)] Output stream has been created. [#default#app/stream(335931652)]
[2024-03-03 20:52:29.559] I [InboundWorker:19] Transcoder | transcoder_decoder.cpp:242  | The decoder has been created successfully. track(#0) codec(H264), module(default:0)
[2024-03-03 20:52:29.559] I [InboundWorker:19] MediaRouter | mediarouter_application.cpp:340  | [#default#app/stream(335931652)] Trying to create a stream
[2024-03-03 20:52:29.559] I [InboundWorker:19] Monitor | application_metrics.cpp:58   | Create StreamMetrics(stream/b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/o) for monitoring
[2024-03-03 20:52:29.559] I [InboundWorker:19] MediaRouter | mediarouter_application.cpp:440  | [#default#app/stream(335931652)] Stream has been created
[2024-03-03 20:52:29.559] I [InboundWorker:19] Transcoder | transcoder_stream.cpp:150  | [#default#app/stream(102)] Transcoder stream has been prepared
[2024-03-03 20:52:29.564] I [Dech264:51] Transcoder | decoder_avc.cpp:210  | [#default#app/stream(102)] input track information: [video] h264 (Constrained Baseline 4.0), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 195 kbps, timebase: 1/60, 
[2024-03-03 20:52:29.565] W [Dech264:51] FFmpeg | third_parties.cpp:115  | [AVCodecContext: 0x7f638c506400] [OpenH264] this = 0x0x7f638c406240, Warning:bEnableFrameSkip = 0,bitrate can't be controlled for RC_QUALITY_MODE,RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.
[2024-03-03 20:52:29.572] I [Dech264:51] Transcoder | transcoder_encoder.cpp:278  | The encoder has been created successfully. track(#0), codec(H264), module(default:0)
[2024-03-03 20:52:29.572] I [Dech264:51] Transcoder | filter_rescaler.cpp:296  | Rescaler parameters. track(#0 -> #0), module(default:0 -> default:0). desc(src:video_size=1920x1080:pix_fmt=yuv420p:time_base=1/1000:pixel_aspect=1/1:frame_rate=30/1 -> output:settb=1/90000,scale=1280x720:flags=bilinear,format=yuv420p)
[2024-03-03 20:52:30.670] I [OutboundWorker:20] MediaRouter | mediarouter_application.cpp:477  | [#default#app/stream(335931652)] Stream has been prepared 
[Stream Info]
id(335931652), msid(0), output(stream), SourceType(Transcoder), RepresentationType(Source), Created Time (Sun Mar  3 20:52:29 2024) UUID(b124cdc7-9ca8-464b-b30f-95c370499de8/default/#default#app/stream/o)
        >> Origin Stream Info
        id(102), output(stream), SourceType(Rtmp), Created Time (Sun Mar  3 20:52:28 2024)

        Video Track #0: Public Name(Video_0) Variant Name(test) Bitrate(20.00Mb) Codec(1,H264,default:0) BSF(H264_ANNEXB) Resolution(1280x720) Framerate(30.00) KeyInterval(30/frame) SkipFrames(-1) BFrames(0) timebase(1/90000)
        Data  Track #1: Public Name(Data_2) Variant Name(Data) Codec(0,Unknown,Passthrough) BSF(ID3v2) timebase(1/1000)
[2024-03-03 20:52:30.672] I [OutboundWorker:20] WebRTC Publisher | rtc_stream.cpp:200  | WebRTC Stream has been created : stream/335931652
Rtx(false) Ulpfec(false) JitterBuffer(false) PlayoutDelay(false min:0 max: 0)
[2024-03-03 20:52:30.672] I [OutboundWorker:20] Publisher | stream.cpp:212  | WebRTC Publisher Application application has started [stream(335931652)] stream (MSID : 0)

[2024-03-03 20:52:53.914] I [SPRtcSig-t3333:5] HTTP.Server | http_server.cpp:149  | Client(<ClientSocket: 0x7f6397001410, #24, Connected, TCP, Nonblocking, 192.168.1.241:49387>) is connected on *:3333
[2024-03-03 20:52:53.915] I [SPRtcSig-t3333:5] Signalling | rtc_signalling_server.cpp:316  | New client is connected: <ClientSocket: 0x7f6397001410, #24, Connected, TCP, Nonblocking, 192.168.1.241:49387>
[2024-03-03 20:52:53.925] I [SPRtcSig-t3333:5] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/stream on the WebRTC publisher. WebRTC(1)/Stream total(1)/App total(1)
[2024-03-03 20:52:53.925] I [SPRtcSig-t3333:5] ICE | ice_port.cpp:368  | Added session: 2 (ufrag: MG9DF0:vUSm)
[2024-03-03 20:52:53.985] I [SPICE-t3478:14] ICE | ice_port.cpp:606  | Turn client has connected : <ClientSocket: 0x7f639d34e010, #25, Connected, TCP, Nonblocking, 192.168.1.241:49390>
[2024-03-03 20:52:53.987] W [SPICE-t3478:14] ICE | ice_port.cpp:690  | Could not find agent(<SocketAddressPair: 0x7f63937fdc90, local: 10.88.0.48:3478, remote: 192.168.1.241:49390>) information. Dropping... [Packet type : DTLS GateType : SEND_INDICATION]
[2024-03-03 20:52:53.987] I [SPICE-t3478:14] ICE | ice_port.cpp:843  | Session 2 uses candidate: <SocketAddressPair: 0x7f63937fdc90, local: 10.88.0.48:3478, remote: 192.168.1.241:49390>
[2024-03-03 20:52:54.029] I [SPICE-t3478:14] ICE | ice_port.cpp:606  | Turn client has connected : <ClientSocket: 0x7f6392a01210, #26, Connected, TCP, Nonblocking, 91.196.221.18:49388>

From what I understand this shows that the transcoded stream is created with RTMP but is not with WebRTC input. In both cases we see the 1920x1080 input, but only the RTMP input actually creates the 1280x720 transcode stream. Any idea why this might be? If it matters this is using an image created from master, not the last official release.

Server.xml

Thanks.

[getroot]

Try adding Framerate to your encoding settings. And it would be a good idea to also specify Height.

<Video>
									<Name>conditional_video_encoding</Name>
									<Codec>h264</Codec>
									<Bitrate>2024000</Bitrate>
									<Width>1280</Width>
									<Height>720</Height>
									<Framerate>30</Framerate>

[Aidan63]

Thanks, adding the height and framerate fixed it.
I was ommitting height as the docs seemed to say that it would be auto calculated based on width to maintain aspect ratio. Is the framerate option the equivilent of ffmpegs -r where it will duplicate and drop frames to reach that target? Some of the video streams I'm planning to input are variable framerate (can vary from 2 - 30fps in a file).

[getroot]

OvenMediaEngine's encoder does not support variable frame rates for several reasons. If you do not specify Framerate in the encoder options, the framerate of the input stream at that moment is used as the output framerate. And if it's 0.5, you're going to have problems. RTMP will have a constant framerate (30 or 60), so your problem is not occurring.

[Aidan63]

Thanks for the clarification, I've realised the framerate issue will not be a problem anyway. My provisioner is built on an earlier proof of concept demo where I separated out the rate at which video frames are sent to the server and the framerate of the source, so even if its 2fps source it will send at 30fps just repeating the last frame to avoid issues we were encountering with other software and very low framerate video.