Any chance of keeping the model loaded

[StuartIanNaylor]

Rather than streaming could the model load and just watch a folder and move audio or delete when processed.
So you don't get any load time from processing file inputs than the current ./main ?

[tomheno]

Hi there, looking to achieve the same thing, maybe simply by loading the model into memory and accessing it on the next ./main run or so...
Our goal is to run on audio chunks of 3/4s, and to avoid the ~500ms of loading, once it's already been loaded.
Any suggestion on a way to achieve that would be very appreciated, thanks in advance.

[RndyP]

You call whisper_init() only once, and then start acquiring and processing your chunks. What is your platform/environment? Are you writing and app, or trying to shell out to main in a loop???

[tomheno]

Hi @RndyP, thanks for your answer !
Currently using the python implementation, but shifting to cpp implementation to experiment on performances, the final goal is to have either a local port receiving simple POST requests including a single file path (of a .wav file), returning the transcript for it as a response.
The implementation you are describing is what I had in mind, to precise it a bit : the goal is for the process to run continuously and to receive/transcript files on the go, not to run on a predefined batch of files.

Thanks again :)

[RndyP]

Sounds like you need a hybrid of the stream and "WAV" file examples. If you start with the stream example you will need to replace the audio card interface with a thread that scans a folder for new WAV files, and then keeps calling whisper_full() with each chunk read in from the file (and then erasing the file?). You will probably want to tweak the default whisper_parms depending on what you finally decide.

I would guess the nuances of either of these methods may depend on how the "gapping" is done. Are your files separated in real time audio by seconds? Minutes? Continuous?

[ggerganov]

@tomheno @RndyP
You don't need to start from stream example.

Just modify main - instead of looping over the input files:

whisper.cpp/examples/main/main.cpp

Lines 504 to 505 in 4ef3398

	for (int f = 0; f < (int) params.fname_inp.size(); ++f) {
	const auto fname_inp = params.fname_inp[f];

Scan the folder for any new files and load them instead. The rest of the loop is the same.
Also, remember which files have been processed so you don't re-process them again.

[StuartIanNaylor]

That would be really cool as maybe use inotify to scan a in-folder and move to an out-folder with inference text.

If any does do something please shout.

[silvacarl2]

i have looked at installing it on a RAM disk but have not finished testing that yet. also, trying to see what sort of a wrapper could be used to create an API out of it so it is always in memory.