OOM with --diarize on long audio (4 hours)

[agorman]

I'm using GPU and during the diarization process system memory stays steady at around 4GB but at the end of the process system memory blows up and the process is killed. The process is using around 18GB before it's killed.

Any thoughts? I can provide the input file if that would help.

Using whisper 3.2.0 and speaker-diarization-3.1.