OSError: headers already sent

[sshellmann]

I'm getting this error on every socket disconnect and it is flooding my logs. I'm hesitant to open a bug request because I won't be surprised if this is somehow a problem of my code's usage. I found a similar bug reported and fixed here:

miguelgrinberg/Flask-SocketIO#377

But the fix for that issue was to upgrade python-engineio and I am currently on the latest version. Relevant versions:

Flask-SocketIO==5.3.3
gevent==22.10.2
greenlet==2.0.2
python-engineio==4.8.0
python-socketio==5.10.0

I have traced through the code with a pdb on the request causing the error and I can't see anything that stands out as wrong. It is hitting this part of the code in engineio/server.py:

if sid in self.sockets and self.sockets[sid].closed: del self.sockets[sid]

Then it proceeds on with a normal response:

{'status': '200 OK', 'headers': [('Content-Type', 'text/plain')], 'response': b'OK'}

and finally errors:

start_response(r['status'], r['headers'] + cors_headers)

With:

Traceback (most recent call last):
  File "/home/halo-runner/halo/.venv/lib/python3.10/site-packages/flask/app.py", line 2551, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/halo-runner/halo/.venv/lib/python3.10/site-packages/flask_socketio/__init__.py", line 43, in __call__
    return super(_SocketIOMiddleware, self).__call__(environ,
  File "/home/halo-runner/halo/.venv/lib/python3.10/site-packages/engineio/middleware.py", line 63, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "/home/halo-runner/halo/.venv/lib/python3.10/site-packages/socketio/server.py", line 428, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "/home/halo-runner/halo/.venv/lib/python3.10/site-packages/engineio/server.py", line 323, in handle_request
    start_response(r['status'], r['headers'] + cors_headers)
OSError: headers already sent

The errors happen only on a disconnect and it happens on every disconnect. It is easy to reproduce. It isn't causing any problems with our users but it causes many errors in our logs so I'd like to fix it if possible.

[miguelgrinberg]

This may be a regression of that old bug, I don't think this is something that you are doing wrong. What I need to know to help me troubleshoot is the versions of the following packages (whichever you have installed):

simple-websocket
gevent-websocket

[sshellmann]

simple-websocket==1.0.0

[miguelgrinberg]

I'm unable to reproduce this error with latest versions of all packages. What version of simple-websocket do you have? Try the latest if you aren't on it yet.

[sshellmann]

Unfortunately, I think I'm on the latest:

$ pip freeze | grep -i socket
Flask-SocketIO==5.3.3
python-socketio==5.10.0
simple-websocket==1.0.0

To ensure it wasn't some other lingering effect, I spun up a docker and fresh installed everything and I still get the error. Is there anything I can add or print within the installed engineio package that would give you a hint?

[sshellmann]

From digging in a bit more. I can see that the server is paused in server.py at:

packets = socket.handle_get_request(
environ, start_response)

And also in socket.py at:

def _upgrade_websocket(self, environ, start_response):
...
ws(environ, start_response)

As soon as I send the disconnect, it continues from there, hits:

return self.server._ok()

And when it is trying to send the OK, it hits the error.

[sshellmann]

I found out that this error does not occur on engineio version 4.7.1. So I will treat that as a fix for now.