Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream not properly closed (NoBackendError | OSError: Too many open files) #134

Open
ne555 opened this issue May 23, 2023 · 4 comments
Open

Comments

@ne555
Copy link

ne555 commented May 23, 2023

import audioread

filename = 'some_audio_file.ogg'
try:
    for K in range(2048):
        with audioread.audio_open(filename) as audio:
            print(K, audio.duration)
except audioread.exceptions.NoBackendError:
    with open(filename, 'rb') as file: # OSError: [Errno 24] Too many open files
        pass

the audio is openned in a loop using with, but it seems that is not properly closed
in my system it will print about 500 lines, then raises the NoBackendError and the OSError when trying to open a new file

using lsof shows lines of the form (my limit is 1024)

COMMAND   PID    USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
python  38124 ne555 1023u     unix 0x00000000d813f451      0t0  2420780 type=STREAM (CONNECTED)

I've observed this issue with audioread.gstdec.GstAudioFile and audioread.ffdec.FFmpegAudioFile backends, couldn't test with audioread.rawread.RawAudioFile

@sampsyo
Copy link
Member

sampsyo commented May 24, 2023

Thanks for the complete script for testing this! I wasn't able to reproduce this with some quick testing (macOS, FFmpeg backend, using lsof to check if things went out of control). I don't have access to something using Gstreamer at the moment, but I would very much believe that that backend could have some kind of a leak.

For FFmpeg in particular, is there any chance you can check whether the loopy script also leaves a bunch of ffmpeg processes running? Like, I can imagine ps | grep ffmpeg showing 1024 processes if we're not correctly cleaning things up there.

@ne555
Copy link
Author

ne555 commented May 30, 2023

good morning, I didn't realise that before trying to open the file with FFmpegAudioFile it was using GstAudioFile
when limited the backends to only FFmpegAudioFile it worked fine.

so the issue seems to be only with the GstAudioFile backend (either if it opens or not the file correctly)

@sampsyo
Copy link
Member

sampsyo commented May 30, 2023

Ah, that makes sense! I unfortunately don't have a great way to test this out here… I can imagine that we may need to do some additional explicit resource cleanup here:

def close(self, force=False):
"""Close the file and clean up associated resources.
Calling `close()` a second time has no effect.
"""
if self.running or force:
self.running = False
self.finished = True
# Unregister for signals, which we registered for above with
# `add_signal_watch`. (Without this, GStreamer leaks file
# descriptors.)
self.pipeline.get_bus().remove_signal_watch()
# Stop reading the file.
self.dec.set_property("uri", None)
# Block spurious signals.
self.sink.get_static_pad("sink").disconnect(self.caps_handler)
# Make space in the output queue to let the decoder thread
# finish. (Otherwise, the thread blocks on its enqueue and
# the interpreter hangs.)
try:
self.queue.get_nowait()
except queue.Empty:
pass
# Halt the pipeline (closing file).
self.pipeline.set_state(Gst.State.NULL)

But it will require some real GStreamer expertise or trial and error to figure out exactly what to clean up.

@nikita-petrashen
Copy link

nikita-petrashen commented Jul 22, 2024

I have run into the same problem while using librosa.load.
Passing a file descriptor instead of the path helped:

with open(path, "rb") as fp:
    librosa.load(fp)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants