Skip to content

Close pooled QMP and guest-agent connections when a VM is deleted or stopped - #566

Draft
yummybomb wants to merge 4 commits into
mainfrom
hypeship/release-qemu-client-on-delete
Draft

yummybomb wants to merge 4 commits into
mainfrom
hypeship/release-qemu-client-on-delete

Conversation

@yummybomb

@yummybomb yummybomb commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Deleting a VM leaves two pooled connections behind in hypeman. Neither is closed until hypeman restarts.

QMP client. Hypeman keeps one QMP client per QEMU socket in a process-wide pool (lib/hypervisor/qemu/pool.go). Pool entries are only dropped by hv.Shutdown(), a QMP error, or a later start that reuses the same socket path. Delete never calls any of those. It shuts the guest down over vsock or kills QEMU directly, and the socket path is unique per instance. So every deleted QEMU VM leaves its QMP client in the pool, which keeps the unix socket fd open. Stop has the same gap when the guest shuts down by itself.

Guest-agent connection. Delete closes the pooled guest-agent gRPC connection, then sends the graceful shutdown RPC, which opens and pools it again. Nothing closes that second connection. A gRPC client whose peer is gone keeps three goroutines. Stop and standby already close it after the VM exits. This one is not specific to QEMU.

On a production GPU host, the count of dead unix connections (ESTAB, no local path, peer inode 0) tracks the count of deletes since the last hypeman restart: 880 connections against 882 deletes over about 29 hours. Hypeman goroutines on QEMU hosts climb by about 2.5 per deleted VM, and they only reset when hypeman restarts.

The goroutines come from the guest-agent connection, not from QMP. QEMU closes its QMP socket cleanly both when it exits and when it is killed, so go-qemu's reader goroutines exit by themselves and only the fd stays open.

Changes

  • hypervisor.RegisterClientReleaser / hypervisor.ReleaseClient: a per-type hook that closes and forgets a cached client for a socket whose VMM has exited. Hypervisor types without a client cache ignore it.
  • QEMU (standard and microvm) registers qemu.Remove as its releaser, which closes the client and its fd.
  • deleteInstanceWithOptions releases the client once the VMM is confirmed gone, whether through graceful guest shutdown or kill, and closes the guest-agent connection again at the same point. A failed kill still returns before this.
  • stopInstance releases the client next to the existing stale-socket cleanup. Standby and the stop fallback already evict through hv.Shutdown(), so they don't change.
  • A QEMU start that fails after connecting to QMP now drops its pooled client when the process is cleaned up. Before, a failed create left it pooled.

Testing

Leaked per deleted VM, measured with real QEMU 7.2 under KVM (4 VMs per mode, goroutines classified from a goroutine profile):

delete with graceful shutdown delete with SkipGracefulShutdown
main 1 socket fd, 3 goroutines 1 socket fd, 0 goroutines
this PR 0 0

All leaked goroutines were grpcsync.(*CallbackSerializer).run, and none were in go-qemu. With only the QMP release and not the guest-agent close, the fds were gone and the 3 goroutines per graceful delete remained. The scratch test that took these numbers is not part of this PR.

  • TestReleaseClientClosesPooledConnection (qemu): a pooled client connected to a fake QMP server is removed from the pool, and the server sees the connection close. Covers both QEMU types. It fails if the releaser isn't registered.
  • TestDeleteAndStopReleaseHypervisorClient (instances): delete and stop of a running instance each call the registered releaser with the instance socket. Each subtest fails if its call site is removed.
  • TestDeleteClosesGuestConn (instances, Linux): deleting a running instance closes the guest-agent connection that the shutdown RPC pooled. It fails without the new close.
  • TestCleanupDropsPooledClient (qemu): cleaning up a started process removes its pooled client. It fails without the reset.
  • go vet ./lib/hypervisor/... ./lib/instances/ is clean, and go test -race passes for the new tests and the existing lifecycle, stop and delete unit tests.
  • I haven't validated this on a GPU host. After deploy, the check is that dead unix connections on QEMU hosts stay near zero and that the goroutine count stops climbing with deletes.

t.TempDir() on macOS produces a path longer than the unix socket limit, so
binding the fake QMP socket failed there.
Delete closes the pooled guest-agent connection before the graceful shutdown
RPC, and the RPC opens it again. Nothing closed that second connection, so
each delete of a running instance left a gRPC client and its goroutines in
the pool.
A start that fails after connecting to QMP kills the process but left its
client in the pool.
@yummybomb yummybomb changed the title Release QEMU QMP client when a VM is deleted or stopped Close pooled QMP and guest-agent connections when a VM is deleted or stopped Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant