Reclaim the snapshot a record stops naming - #2171
Closed
MayCXC wants to merge 4 commits into
Closed
Conversation
Pulling or loading a reference that already exists replaces its record and leaves the replaced generation's unpacked snapshot owned by no image. Nothing reclaims those directories outside the delete and prune commands, so a machine that rebuilds or refreshes the same tags accumulates dead snapshots indefinitely; a week of image rebuilds on one machine held 28G this way, invisible to every per-run cleanup. Sweep the snapshot store after every operation that lands records, using the same keep set the prune path uses. The sweep is housekeeping: its failure warns in the log and never fails the pull or load that carried it.
The service is an actor, and every await inside a store operation admits the next call, so a garbage collection can overlap an in-flight landing: its keep set is computed without the arriving image, and the cleanup prunes blobs the landing is about to reference and sweeps the snapshot a container is about to attach. A lock held across each whole mutating operation admits one at a time, the way PodsService holds its lock across lifecycle operations; containerd guards the same window with leases. https://github.com/containerd/containerd/blob/main/docs/garbage-collection.md
A snapshot is owned by the record that names it, so a record that stops naming it leaves it owned by nobody. Registration over an existing reference was swept and removal was not, so every deleted image left its unpacked snapshot behind, and nothing outside the prune verbs ever collected one. What made that expensive is that removal is what a caller does when it is being tidy: an integration run that deletes each image its builds register leaves a snapshot per build, and a suite of them fills a disk that the same suite without the tidying would not have.
The sweep reads every image to learn which snapshot it claims, then removes the snapshots no image claims. A failure anywhere in either loop left the whole pass. Reading an image whose content the store no longer holds fails, so one record from an interrupted pull stopped every snapshot from ever being reclaimed again for as long as the record existed; and one snapshot that would not remove abandoned the rest of the set and took the bytes already freed with it, so the caller was told nothing had been reclaimed when some of it had. An integration run left twenty one gigabytes of the unpacked filesystems of images nothing names, and the control plane failed on the disk they took. Read each image and remove each snapshot on its own terms, logging the error against the item that raised it. An image the store cannot read is one it cannot unpack or run either, so the snapshot that image would have claimed is not worth keeping the store full for.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type of Change
Motivation and Context
Closes #2164.
A snapshot is owned by the record that names it, so a record that stops naming one leaves it owned by nobody. Three paths let that happen and one of them disables the collector.
Registration that replaces a reference. Pulling or loading a reference that already exists replaces its record and leaves the replaced generation's unpacked snapshot owned by no image. Nothing collects those outside the delete and prune commands, so a machine that rebuilds or refreshes the same tags accumulates dead snapshots indefinitely; a week of image rebuilds on one machine held 28G this way, invisible to every per-run cleanup. The sweep now runs after every operation that lands records, using the same keep set the prune path uses, and its failure warns rather than failing the pull or load that carried it.
Removal. Registration over an existing reference was swept and removal was not, so every deleted image left its unpacked snapshot behind. That penalises the tidy caller: an integration run that deletes each image its builds register leaves a snapshot per build, and a suite of them fills a disk that the same suite without the tidying would not have.
One bad item stopping the pass. The sweep reads every image to learn which snapshot it claims, then removes the snapshots no image claims, and a failure anywhere in either loop abandoned the whole pass. One record from an interrupted pull, whose content the store no longer holds, stopped every snapshot from ever being reclaimed again for as long as that record existed; and one snapshot that would not remove abandoned the rest of the set and discarded the bytes already freed, so the caller was told nothing had been reclaimed when some of it had. Each image is read and each snapshot removed on its own terms now, with the error logged against the item that raised it. An image the store cannot read is one it cannot unpack or run either, so the snapshot that image would have claimed is not worth keeping the store full for.
Serialization. The service is an actor, and every
awaitinside a store operation admits the next call, so a collection can overlap an in-flight landing: its keep set is computed without the arriving image, and the cleanup prunes blobs the landing is about to reference and sweeps the snapshot a container is about to attach. A lock held across each whole mutating operation admits one at a time, the wayPodsServiceholds its lock across lifecycle operations. containerd guards the same window with leases: https://github.com/containerd/containerd/blob/main/docs/garbage-collection.mdAn integration run here left 21G of unpacked filesystems for images nothing named, and the control plane then failed on the disk they took.
Testing
Integration suite: 397 passed. Unit suite: 772 passed.
make fmt,make checkclean.