Streaming support?

[weijia-yu]

Does it support streaming mode?

[sidroopdaska]

Not yet @weijia-yu. I'll let you know once we release this feature.

In the meantime, if useful, our model can run faster than real-time on Ampere, Ada LoveLace & Hopper GPUs

[medemi68]

I think it should be possible to implement this because there is causal_sample available. I'm not a Python / AI developer but this should be possible. If I manage to get this working (with the help of AI of course) I'll submit a PR.

[medemi68]

Okay so after doing some analysis it seems like the first stage of the model is causal. Which means its suitable for streaming, but the problem seems to come in with the second stage of the model. So I was thinking that maybe we can chunk the audio (I think the minimum is 400ms) and run the second stage on each one of that? I'm still tinkering around with this but hopefully someone from the team can explain.

[vatsalaggarwal]

Best to use #164 which replaces the second stage model with an convolutional decoder and also significantly improves signal quality. It works on top of the first causal LLM and can also be made streamable by waiting for receptive fields worth of input.

[medemi68]

Best to use #164 which replaces the second stage model with an convolutional decoder and also significantly improves signal quality. It works on top of the first causal LLM and can also be made streamable by waiting for receptive fields worth of input.

I’ll give that a shot! I haven’t had the bandwidth over the last week to really work on this (my team has sort of been focusing on Parler TTS as it’s proven to be a bit lower latency, albeit with my personal opinion that the quality is worse though) but that sounds very promising. Looking forward to working on this and I’ll keep you updated.

[medemi68]

It seems you already have a streaming version though? I noticed a branch called sidroopdaska/streaming ? Haven't looked to far at the code yet but it looks like you've done all of the leg work already? Perhaps you could shed some light on it?