Bridge crashes on removing channels from endpoints

[allisonkarlitskaya]

This is suddenly showing up a whole lot in cockpit-files, but only on arch:

Jun 20 12:09:05 archlinux cockpit-ws[2919]: Traceback (most recent call last):
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self._context.run(self._callback, *self._args)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/transports.py", line 110, in _read_ready
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self._protocol.data_received(data)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/protocol.py", line 193, in data_received
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     result = self.consume_one_frame(self.buffer)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/protocol.py", line 143, in consume_one_frame
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self.frame_received(data[start:end])
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/protocol.py", line 97, in frame_received
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self.control_received(data)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/protocol.py", line 107, in control_received
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self.channel_control_received(channel, command, message)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/peer.py", line 212, in channel_control_received
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self.send_channel_control(channel, command, message)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:   File "/usr/lib/python3.12/site-packages/cockpit/router.py", line 101, in send_channel_control
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     self.router.endpoints[self].remove(channel)
Jun 20 12:09:05 archlinux cockpit-ws[2919]:     ~~~~~~~~~~~~~~~~~~~~~^^^^^^
Jun 20 12:09:05 archlinux cockpit-ws[2919]: KeyError: <cockpit.superuser.SuperuserPeer object at 0x7129b8676930>

It seems to be caused by having added this to a test:

     def testSorting(self) -> None:
         b = self.browser
         m = self.machine

         self.enter_files()

+        # set a bogus sort value in localStorage to make sure we handle it gracefully
+        b.eval_js("""window.localStorage.setItem("files:sort", 'bzzt')""")
+        b.reload()
+        self.enter_files()

ie: I guess superuser mode is being torn down and reinitialized?

We've seen this around in flakes for a long time, but only very occasionally, but with this test change it's happening really often. This definitely needs to be looked into sooner rather than later — even if we can't figure it out, we ought to introduce an exception for the message or so.