Repository navigation
Conversation
t.TempDir() on macOS produces a path longer than the unix socket limit, so binding the fake QMP socket failed there.
Delete closes the pooled guest-agent connection before the graceful shutdown RPC, and the RPC opens it again. Nothing closed that second connection, so each delete of a running instance left a gRPC client and its goroutines in the pool.
A start that fails after connecting to QMP kills the process but left its client in the pool.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deleting a VM leaves two pooled connections behind in hypeman. Neither is closed until hypeman restarts.
QMP client. Hypeman keeps one QMP client per QEMU socket in a process-wide pool (
lib/hypervisor/qemu/pool.go). Pool entries are only dropped byhv.Shutdown(), a QMP error, or a later start that reuses the same socket path. Delete never calls any of those. It shuts the guest down over vsock or kills QEMU directly, and the socket path is unique per instance. So every deleted QEMU VM leaves its QMP client in the pool, which keeps the unix socket fd open. Stop has the same gap when the guest shuts down by itself.Guest-agent connection. Delete closes the pooled guest-agent gRPC connection, then sends the graceful shutdown RPC, which opens and pools it again. Nothing closes that second connection. A gRPC client whose peer is gone keeps three goroutines. Stop and standby already close it after the VM exits. This one is not specific to QEMU.
On a production GPU host, the count of dead unix connections (
ESTAB, no local path, peer inode 0) tracks the count of deletes since the last hypeman restart: 880 connections against 882 deletes over about 29 hours. Hypeman goroutines on QEMU hosts climb by about 2.5 per deleted VM, and they only reset when hypeman restarts.The goroutines come from the guest-agent connection, not from QMP. QEMU closes its QMP socket cleanly both when it exits and when it is killed, so go-qemu's reader goroutines exit by themselves and only the fd stays open.
Changes
hypervisor.RegisterClientReleaser/hypervisor.ReleaseClient: a per-type hook that closes and forgets a cached client for a socket whose VMM has exited. Hypervisor types without a client cache ignore it.qemu.Removeas its releaser, which closes the client and its fd.deleteInstanceWithOptionsreleases the client once the VMM is confirmed gone, whether through graceful guest shutdown or kill, and closes the guest-agent connection again at the same point. A failed kill still returns before this.stopInstancereleases the client next to the existing stale-socket cleanup. Standby and the stop fallback already evict throughhv.Shutdown(), so they don't change.Testing
Leaked per deleted VM, measured with real QEMU 7.2 under KVM (4 VMs per mode, goroutines classified from a goroutine profile):
SkipGracefulShutdownmainAll leaked goroutines were
grpcsync.(*CallbackSerializer).run, and none were in go-qemu. With only the QMP release and not the guest-agent close, the fds were gone and the 3 goroutines per graceful delete remained. The scratch test that took these numbers is not part of this PR.TestReleaseClientClosesPooledConnection(qemu): a pooled client connected to a fake QMP server is removed from the pool, and the server sees the connection close. Covers both QEMU types. It fails if the releaser isn't registered.TestDeleteAndStopReleaseHypervisorClient(instances): delete and stop of a running instance each call the registered releaser with the instance socket. Each subtest fails if its call site is removed.TestDeleteClosesGuestConn(instances, Linux): deleting a running instance closes the guest-agent connection that the shutdown RPC pooled. It fails without the new close.TestCleanupDropsPooledClient(qemu): cleaning up a started process removes its pooled client. It fails without the reset.go vet ./lib/hypervisor/... ./lib/instances/is clean, andgo test -racepasses for the new tests and the existing lifecycle, stop and delete unit tests.