High CPU usage (300%+) caused by keepAliveLoop goroutine leak and nil pointer panic in handleWebsocket

Description

When running chisel server in a Docker container, CPU usage spikes to 300%+ after client sessions disconnect. The process consumes excessive CPU even with no active tunnels, and eventually becomes unresponsive.

Environment

Chisel server: v1.11.8
Chisel client: v1.11.7 (version mismatch observed)
Running in Docker container
Reverse tunnel enabled (-R)
Reverse proxies: R:2333=>22

Observed Behavior

Top output:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3630019 root      20   0 1270408  16980   7048 S 344.0   0.1      8:15 bin

Goroutine stack traces show:

Multiple keepAliveLoop goroutines stuck in [runnable] state, spinning on ssh.(*mux).SendRequest
Nil pointer dereference panic in handleWebsocket at server_handler.go:94

Logs show:

Clients rapidly reconnecting with exponential backoff (sessions [Feature Request] add X-Forwarded-For header #4 through Add proxy support #14 in ~7 minutes)
Version mismatch warnings on every connection

Root Cause Analysis

Bug 1: `keepAliveLoop` goroutine leak (High)

share/tunnel/tunnel.go:178-193 — The keepAliveLoop function has no context.Context parameter and can only exit when sshConn.SendRequest returns an error:

func (t *Tunnel) keepAliveLoop(sshConn ssh.Conn) {
    for {
        time.Sleep(t.Config.KeepAlive)  // no context awareness
        _, b, err := sshConn.SendRequest("ping", true, nil)
        if err != nil {
            break
        }
        if len(b) > 0 && !bytes.Equal(b, []byte("pong")) {
            t.Debugf("strange ping response")
            break
        }
    }
    //close ssh connection on abnormal ping
    sshConn.Close()
}

When an SSH connection dies:

The goroutine may be sleeping in time.Sleep and won't detect the failure until the next SendRequest call (delay up to KeepAlive interval)
SendRequest on a closed mux returns immediately without blocking — the goroutine enters a rapid cycle of: sleep → wake → SendRequest(immediate error) → break → Close → exit
Multiple such goroutines from disconnected sessions remain in [runnable] state simultaneously, causing the Go scheduler to consume excessive CPU managing transitions

With rapid client reconnect cycles (observed in logs), new keepAliveLoop goroutines are spawned on each BindSSH call before the previous ones have exited, causing goroutine accumulation.

Bug 2: Nil pointer dereference panic (Critical)

server/server_handler.go:94 — When the SSH connection closes before the client sends a "config" request, the reqs channel is closed. Reading from a closed channel returns the zero value (nil for *ssh.Request):

select {
case r = <-reqs:   // r = nil when channel is closed
case <-time.After(10 * time.Second):
    ...
}
if r.Type != "config" {  // nil pointer dereference → panic!

This panic kills the handleWebsocket handler, leaving associated goroutines (including keepAliveLoop) orphaned with no parent context to clean them up, further contributing to the goroutine leak.

Suggested Fixes

Priority	Fix	Location
Critical	Add nil check: `if r == nil { return }` after receiving from `reqs` channel	`server_handler.go:84-88`
High	Pass `context.Context` to `keepAliveLoop`, replace `time.Sleep` with context-aware `select` + `time.After`	`tunnel.go:178-193`
High	Ensure session disconnect explicitly cancels all spawned goroutines	`server_handler.go`, `tunnel.go`

Reproduction

Run chisel server with reverse tunnel: chisel server -R --reverse
Connect a client with version mismatch (1.11.7 client → 1.11.8 server)
Disconnect the client (network interruption or manual stop)
Observe CPU usage climb as keepAliveLoop goroutines accumulate
Reconnect repeatedly to accelerate the issue

i've observed this behavior outside of docker as well; specifically, i'm using the version distributed by nixos/nixpkgs:

$ chisel --version
1.11.8

chisel is being invoked with the following parameters:

chisel server --host "[::1]" --port 8077 --authfile /run/agenix/chisel-authfile --reverse

logs from chisel server are showing this for me:

Jul 15 19:45:00 piano chisel[306139]: 2026/07/15 19:45:00 server: session#1: Client version (0.0.0-src) differs from server version (1.11.8)
Jul 15 19:45:00 piano chisel[306139]: 2026/07/15 19:45:00 server: session#1: tun: proxy#R:127.0.0.1:47990=>host.docker.internal:47990: Listening
Jul 16 11:32:37 piano chisel[306139]: 2026/07/16 11:32:37 server: session#2: Client version (0.0.0-src) differs from server version (1.11.8)
Jul 16 11:32:37 piano chisel[306139]: 2026/07/16 11:32:37 server: session#2: tun: proxy#R:127.0.0.1:47990=>host.docker.internal:47990: Listening

looking back at cpu usage for my server, i noticed it started spiking at right about the same time the second set of connections occured here

fix(server): don't panic when client drops before config request

1.12: reliability & security pass (#609)

v1.12.0-rc1 fixes both bugs reported here: the keepalive loop now exits when the connection dies (and pings time out via CHISEL_PING_TIMEOUT, so no more spinning goroutines), and the nil-pointer panic in handleWebsocket is guarded — a client disconnecting before its config request no longer panics the handler (with a regression test).

Please try out https://github.com/jpillora/chisel/releases/tag/v1.12.0-rc1 and report any issues in #610

The high CPU isn't a goroutine leak, and it isn't in chisel's code. It's a bug in
golang.org/x/crypto/ssh v0.52.0, which is the exact version v1.11.8 pins.

v0.52.0 added a drain step at the top of (*mux).SendRequest, before the message goes out:

drain:
	for {
		select {
		case <-m.globalResponses:
		default:
			break drain
		}
	}

The only way out is default:, and that needs m.globalResponses to be empty and open. When the
SSH transport dies, mux.loop() runs close(m.globalResponses). Receiving from a closed channel
always succeeds right away, so default: never gets picked and the loop runs forever.

So keepAliveLoop sleeps its 25s, calls SendRequest("ping", true, nil), and never comes back out.
The drain loop sits before sendMessage, so the write that would have failed on the dead socket
never happens, which means if err != nil { break } never fires and sshConn.Close() never runs.

v0.51.0 doesn't have the drain loop at all, and v0.53.0 fixed it by checking the receive
(case _, ok := <-m.globalResponses: with if !ok { break drain }). v0.52.0 is the only broken
one, and it's what v1.11.8 depends on.

On a live 1.11.8 server here (x86-64 Linux, reverse tunnel, two dead sessions), perf puts 88% of
8048 samples in one stack:

runtime.chanrecv ← runtime.selectnbrecv
  golang.org/x/crypto/ssh.(*mux).SendRequest
    golang.org/x/crypto/ssh.(*ServerConn).SendRequest
      github.com/jpillora/chisel/share/tunnel.(*Tunnel).keepAliveLoop
        github.com/jpillora/chisel/share/tunnel.(*Tunnel).BindSSH.gowrap1

and the compiled loop only tests the first return value of selectnbrecv:

795a73:  mov 0x50(%rdx),%rbx      ; rbx = m.globalResponses
795a77:  xor %eax,%eax
795a79:  call runtime.selectnbrecv
795a80:  test %al,%al             ; `selected` only, `recvOK` in %bl is never tested
795a82:  jne  795a6b              ; loop while selected

CPU sits at 79929s over 42106s elapsed, so a flat 1.9 cores, one per dead session, and it never
moves. The spinning threads make zero syscalls (/proc/<tid>/syscall reads running), which rules
out scheduler overhead and confirms nothing is being written. Only six fds are open: stdio, the log,
the listen socket, epoll, eventfd. No connection sockets left, every session long gone, CPU still
never drops. runtime.empty also shows up as a hot leaf at 22%, which happens because chanrecv
calls empty(c) twice on the closed-channel path but only once when the channel is open and empty.

Two things in the report above are off. SendRequest on a closed mux doesn't return immediately
with an error, it doesn't return at all, so the goroutines aren't cycling through sleep, wake,
error, exit. Each one is stuck inside a single SendRequest call for the rest of the process's
life. And the CPU isn't the scheduler moving runnable goroutines around, it's one real infinite loop
per dead session, each sitting on a core at 100%. That changes the fix, because making time.Sleep
context-aware won't help a goroutine that never gets back to the sleep.

v1.12.0-rc3 does fix this, but it's the bump to x/crypto v0.54.0 doing the work, not the ping
timeout. On v0.52.0 the timeout wrapper wouldn't have been enough:

errc := make(chan error, 1)
go func() {
	_, b, err := sshConn.SendRequest("ping", true, nil)
	errc <- err
}()
select {
case err := <-errc:
	return err
case <-time.After(timeout):
	return errors.New("ping timeout")
}

If SendRequest never returns, the timeout lets keepAliveLoop exit but the goroutine it just
spawned keeps spinning. The core stays pegged, and now nothing in the stack even points at the
keepalive loop. That only matters if any of this gets backported to 1.11.x without the dependency
bump, but worth knowing.

Last thing: v1.11.8 is still the latest stable and it's the version distros package (nixpkgs, per the comment
above). Anyone on it loses a core for good on every dropped session until they restart. Master and
the v1.12.0 pre-releases already have x/crypto v0.54.0. Could you release a v1.11.9 with just that
one-line bump? It'd fix the stable line without waiting for v1.12.0 to settle.

RDODCP-914: Chisel Sync with Upstream and dependency Updates (#60)

!18047 merge bump-chisel-tunnel-1.12.0 into main

@ojura

Closed in e588d21, released in v1.12.0.

This updates x/crypto from v0.52.0 to v0.54.0, fixing the SSH SendRequest infinite loop, and guards the nil config request in handleWebsocket. It also adds keepalive timeouts and exits the keepalive loop when the connection closes. Thanks @ojura for pinpointing the dependency bug; the dependency update is what fixes the CPU spin.

Navigation Menu

Repository navigation

High CPU usage (300%+) caused by `keepAliveLoop` goroutine leak and nil pointer panic in `handleWebsocket` #608

Description

Environment

Observed Behavior

Root Cause Analysis

Bug 1: `keepAliveLoop` goroutine leak (High)

Bug 2: Nil pointer dereference panic (Critical)

Suggested Fixes

Reproduction

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

High CPU usage (300%+) caused by keepAliveLoop goroutine leak and nil pointer panic in handleWebsocket #608

Description

Description

Environment

Observed Behavior

Root Cause Analysis

Bug 1: keepAliveLoop goroutine leak (High)

Bug 2: Nil pointer dereference panic (Critical)

Suggested Fixes

Reproduction

Activity

KiyoNetcat commented on Jul 17, 2026

jpillora commented on Jul 17, 2026

ojura commented on Aug 29, 2026

jpillora commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

High CPU usage (300%+) caused by `keepAliveLoop` goroutine leak and nil pointer panic in `handleWebsocket` #608

Bug 1: `keepAliveLoop` goroutine leak (High)