[BUG] - UE 5.7 PixelStreaming H264 NVENC commit memory grows due to frequent nvEncReconfigureEncoder calls from WebRTC rate control

[yuxueliyuxl]

Engine:
UE 5.7

Platform:
Windows 10/11, D3D12, NVIDIA GPU, H264 NVENC

Plugin:
PixelStreaming / PixelStreaming2

Issue:
When using H264 Pixel Streaming, process commit size continuously grows.
The growth is faster when remote clients have packet loss or unstable network.
VP9 does not show the same commit growth.

Reproduction:

1. Create/open a UE 5.7 project with PixelStreaming2 enabled.
2. Run with H264:
   -PixelStreamingEncoderCodec=H264
   -PixelStreamingWebRTCMaxFps=60
   -PixelStreamingEncoderMaxBitrate=30000000
   -PixelStreamingWebRTCMaxBitrate=30000000
3. Connect from a remote browser/client.
4. Observe Windows Task Manager Commit Size or VMMap committed heap.
5. Commit size keeps growing over time.
6. The growth accelerates under packet loss / poor network.

Expected:
Commit size should stabilize after streaming starts.

Actual:
Commit size keeps increasing. In our case it could grow many GB within minutes.

Investigation:
UMDH/VMMap show the growth is heap commit under nvEncodeAPI64.dll.
Top stack includes:
nvEncodeAPI64.dll
FEncoderNVENC::ApplyConfig
FVideoEncoderNVENCD3D12::ApplyConfig
FVideoEncoderNVENCD3D12::SendFrame
TEpicRtcVideoEncoder::Encode
EpicRtc::Video::EncoderWebRtc::Encode
webrtc::VideoStreamEncoder::EncodeVideoFrame

We instrumented FEncoderNVENC::ApplyConfig and found WebRTC/EpicRTC frequently changes bitrate/FPS.
Each rate-only change calls nvEncReconfigureEncoder.

Workaround verified:
If FEncoderNVENC::ApplyConfig skips nvEncReconfigureEncoder when only bitrate/maxBitrate/framerate/derived VBV fields changed, commit size stops growing.
Resolution changes are still allowed to reconfigure.

Fields ignored in workaround:
averageBitRate
maxBitRate
frameRateNum
frameRateDen
vbvBufferSize
vbvInitialDelay

[Belchy06]

Hey @yuxueliyuxl,

This is a known issue with the NVidia drivers. There is currently no known version of drivers that do not contain this leak.

5.8 introduced a work around for this issue which is to either use the -AVCodecs.NvEnc.D3D12UsesCUDA=true launch argument, or to add D3D12UsesCUDA=true under a AVCodecs.NvEnc section in the config ini. However, this workaround uses a D3D12->Cuda->NvEnc pipeline which may introduce artefacts if you use placed (instead of committed) D3D12 resources.

Try the workaround and let me know how it goes. Other than that, we are awaiting a fix from NVidia.

[yuxueliyuxl]

I’ve already tested -AVCodecs.NvEnc.D3D12UsesCUDA=true and D3D12UsesCUDA=true, but neither worked. The solution I provided is effective.