Can't connect to remote multiplexer

[harshasrisri]

What Operating System(s) are you seeing this problem on?

Linux X11, Linux Wayland, macOS

Which Wayland compositor or X11 Window manager(s) are you using?

Linux is headless remote. Not using GUI.

WezTerm version

wezterm 20241129-152148-4906789a

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

Nightly:
Server: Ubuntu 22.04, installed WezTerm nightly via apt after updating the sources
Client: WezTerm nightly installed via Homebrew

Release:
Server and Client running the latest release build from Feb 2024.

Connection to the server via wezterm connect is failing after the server froze and went through a restart. Regular ssh and all other programs are unaffected after the restart. I have been running the latest release build of Wezterm from Feb 2024 until now. For the sake of this bug report, I did install the latest nightly version specified above. There was no change in config before and after. Another point of interest is that I connect to the remote server via a proxy service called coder. Connecting via the proxy was just fine until now.

To Reproduce

No response

Configuration

No change in config before and after seeing the issue. I can't point to a relevant section. But here it is in full if needed: https://github.com/harshasrisri/dotRC/tree/main/dot/config/wezterm.

Expected Behavior

Work as before, establish a remote ssh connection using the wezterm multiplexer on the remote host.

Logs

❯ wezterm connect coder --workspace test
[2024/12/02 22:22:18.744282, 1] local_parse_file:  Cannot find file /etc/ssh/ssh_config.d/* to load
[2024/12/02 22:22:18.745039, 1] socket_callback_connected:  Socket connection callback: 1 (0)
[2024/12/02 22:22:22.117306, 1] ssh_known_hosts_read_entries:  Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
[2024/12/02 22:22:22.173171, 1] ssh_known_hosts_read_entries:  Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
22:22:24.723  ERROR  wezterm_client::client > ssh stderr: 06:22:24.634  WARN   wezterm_client::client > While connecting to Socket("/home/harshasrisri/.local/share/wezterm/sock"): connecting to /home/harshasrisri/.local/share/wezterm/sock.  Will try spawning the server.
06:22:24.634  WARN   wezterm_client::client > Running: "/usr/bin/wezterm-mux-server" "--daemonize"

22:22:27.075  ERROR  wezterm_client::client > ssh stderr: 06:22:27.087  ERROR  wezterm                > (after spawning server) failed to connect to Socket("/home/harshasrisri/.local/share/wezterm/sock"): connecting to /home/harshasrisri/.local/share/wezterm/sock: Connection refused (os error 111); terminating

22:22:27.180  ERROR  wezterm_client::client > wezterm cli --prefer-mux proxy: Exited with code 1
22:22:27.180  ERROR  wezterm_client::client > Error while decoding response pdu: decoding a PDU: decode_raw_async failed to read PDU length: EOF while reading leb128 encoded value
22:22:27.181  ERROR  wezterm_gui            > Please install the same version of wezterm on both the client and server! The server reported error 'Error while decoding response pdu: decoding a PDU: decode_raw_async failed to read PDU length: EOF while reading leb128 encoded value' while being asked for its version.  This likely means that the server is older than the client, but it could also happen if the remote host outputs to stdout prior to running commands. Check your shell startup!; terminating

Anything else?

If I connect using plain ssh -v, this is the log:

localhost ❯ ssh -v remote_server
OpenSSH_9.7p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/harshasrisri/.ssh/config
debug1: /Users/harshasrisri/.ssh/config line 22: Applying options for remote_server
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Executing proxy command: exec /opt/homebrew/bin/coder --global-config "/Users/harshasrisri/Library/Application Support/coderv2" ssh --stdio remote_server
debug1: identity file /Users/harshasrisri/.ssh/id_rsa type 0
debug1: identity file /Users/harshasrisri/.ssh/id_rsa-cert type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ecdsa type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ed25519 type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/harshasrisri/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/harshasrisri/.ssh/id_xmss type -1
debug1: identity file /Users/harshasrisri/.ssh/id_xmss-cert type -1
debug1: identity file /Users/harshasrisri/.ssh/id_dsa type -1
debug1: identity file /Users/harshasrisri/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug1: Authenticating to remote_server:22 as 'harshasrisri'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:jurgMEO3Xd7SjJMLIAmoIUaIRIP3bmXLUvugQBaY8Fw
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
Warning: Permanently added 'remote_server' (RSA) to the list of known hosts.
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: kex_ext_info_check_ver: [email protected]=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
Authenticated to remote_server (via proxy) using "none".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Entering interactive session.
debug1: pledge: proc
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: pledge: fork
warning: completion was already initialized before completion module. Will call compinit again. See https://github.com/zimfw/zimfw/wiki/Troubleshooting#completion-is-not-working
Brewing
Disk usage normal at 47%

remote_server ❯                   # shell on the remote server