Microphone delay incorrect (does not equal sum of Playback latency + Input latency)

[adroste]

Actual behaviour

The Microphone delay cannot be calculated by summing Playback latency + Input latency.
For some reason there is an additional lag? Maybe it is introduced by an internal buffer or the pitch detection algorithm.

Expected behaviour

Microphone Delay = Playback Latency + Input Latency.

Steps to reproduce

1. macOS
2. Pull + merge improve logging of portaudio + sdl audio latency #575 for improved logging
3. (optional:) Pull + merge Fix macOS crackling/stuttering related to SDL buffer, use Portaudio for playback instead of SDL #576 to use Portaudio for Playback
4. compile
5. check the log and write down the latencies for input and output
6. Set microphone delay to playback latency + input latency
7. Notice that the microphone delay is too small and needs to be corrected up

Details

Provide some additional information:

• USDX version: 2020.4.0 and self-compiled master d49e916
• Operating System + version: macOS Big Sur 11.2.3

As the log shows, the summed delay should be 4.55ms + 10ms = 14.55 ms.
However the Microphone delay must be set to a value larger than 60ms for a rough match.
Where is the additional delay coming from?

Log:

UltraStar Deluxe - www.usdx.eu V 2020.4.1 dev Build Error Log
Date: 14-4-21 Time: 19:38:41
-------------------
STATUS: Load Language [Initialization]
STATUS: Load Default Language English [Initialization]
STATUS: Loading Skin List [Initialization]
STATUS: Loading Theme List [Initialization]
STATUS: Searching for Theme : /Users/xyz/Library/Application Support/UltraStarDeluxe1.3/themes/*.ini [Theme.LoadList]
STATUS: Found Theme: Modern.ini [Theme.LoadList]
STATUS: Found Theme: Deluxe.ini [Theme.LoadList]
STATUS: Website-Manager [Initialization]
STATUS: DataBase System [Initialization]
STATUS: Initializing database: "/Users/xyz/Library/Application Support/UltraStarDeluxe1.3/Ultrastar.db" [TDataBaseSystem.Init]
STATUS: Load Ini [Initialization]
STATUS: GamePath : /Users/xyz/Library/Application Support/UltraStarDeluxe1.3/ []
STATUS: Using config : /Users/xyz/Library/Application Support/UltraStarDeluxe1.3/config.ini [Ini]
[...]
INFO:   Audio-Output API-Type: Core Audio [TAudioPlayback_Portaudio.OpenDevice]
INFO:   Audio-Output Device: Komplete Audio 6 [TAudioPlayback_Portaudio.OpenDevice]
STATUS: Audio-Output Latency: 4.5578231292517ms [TAudioPlayback_Portaudio.OpenDevice]
STATUS: InputDevice "Komplete Audio 6"@6x44100Hz (0.1sec) [Portaudio.EnumDevices]
STATUS: InputDevice "MacBook Pro Microphone"@1x48000Hz (0.1sec) [Portaudio.EnumDevices]
STATUS: InputDevice "Komplete Audio Stereo"@2x44100Hz (0.1sec) [Portaudio.EnumDevices]
STATUS: InputDevice "Komplete Audio CH1 Mono"@2x48000Hz (0.1sec) [Portaudio.EnumDevices]
STATUS: #Input-Devices: 4 [Portaudio]
INFO:   Using decoder FFmpeg_Decoder for "/Users/alexander/Library/Application Support/UltraStarDeluxe1.3/sounds/Common start.mp3" [TAudioPlaybackBase.OpenDecodeStream]
[...]
STATUS: SDL_Init [UGraphic.Initialize3D]
STATUS: Use config resolution [SDL_SetVideoMode]
STATUS: Creating window [SDL_SetVideoMode]
STATUS: Set Video Mode...   Windowed [SDL_SetVideoMode]
[...]
STATUS: Main Loop [Initialization]
INFO:   VideoStreamIndex : -1 [TVideoPlayback_ffmpeg.Open]
INFO:   Using mpeg4 codec [TVideoPlayback_ffmpeg.Open]
INFO:   Framerate: 12fps [TVideoPlayback_ffmpeg.Open]
INFO:   VideoStreamIndex : -1 [TVideoPlayback_ffmpeg.Open]
INFO:   Using mpeg4 codec [TVideoPlayback_ffmpeg.Open]
INFO:   Framerate: 12fps [TVideoPlayback_ffmpeg.Open]
INFO:   VideoStreamIndex : -1 [TVideoPlayback_ffmpeg.Open]
INFO:   Using mpeg4 codec [TVideoPlayback_ffmpeg.Open]
INFO:   Framerate: 12fps [TVideoPlayback_ffmpeg.Open]
INFO:   VideoStreamIndex : -1 [TVideoPlayback_ffmpeg.Open]
INFO:   Using mpeg4 codec [TVideoPlayback_ffmpeg.Open]
INFO:   Framerate: 24fps [TVideoPlayback_ffmpeg.Open]
STATUS: Begin [OnShow]
INFO:   VideoStreamIndex : -1 [TVideoPlayback_ffmpeg.Open]
INFO:   Using mpeg4 codec [TVideoPlayback_ffmpeg.Open]
INFO:   Framerate: 24fps [TVideoPlayback_ffmpeg.Open]
STATUS: Open Komplete Audio 6 [Portaudio]
STATUS: Latency of Komplete Audio 6: 10ms (requested: 10ms) [Portaudio]
STATUS: End [OnShow]
INFO:   Using decoder FFmpeg_Decoder for "/Users/xyz/song.mp3" [TAudioPlaybackBase.OpenDecodeStream]
STATUS: Closing DB file [Finalization]
INFO:   TDataBaseSystem.Free [TDataBaseSystem.Destroy]
STATUS: Finalize Media [Finalization]
STATUS: Uninitialize 3D [Finalization]
STATUS: Kill Fonts [UnloadFontTextures]
STATUS: Finalize SDL [Finalization]
STATUS: Finalize Log [Finalization]

[s09bQ5]

Note that these numbers are just the delay caused by the ring buffers shared between the application and the hardware. The hardware might have an additional internal delay, but it is usually much less than 60ms (IIRC my HDA on-board sound adds around 8ms).

Yes, we do resample audio and we don't know how much latency is added by libswresample. Having a small microphone delay is nice but not a must. I don't think there are many people that enable pass-through of the microphone input and the visual feedback (i.e. the coloring of the sung bars) is probably still ok at 60ms.

But patches to optimize this are welcome.