Skip to content

Reclaim the snapshot a record stops naming - #2171

Closed
MayCXC wants to merge 4 commits into
apple:mainfrom
MayCXC:snapshot-gc-on-replace
Closed

Reclaim the snapshot a record stops naming#2171
MayCXC wants to merge 4 commits into
apple:mainfrom
MayCXC:snapshot-gc-on-replace

Conversation

@MayCXC

@MayCXC MayCXC commented Aug 27, 2026

Copy link
Copy Markdown

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Motivation and Context

Closes #2164.

A snapshot is owned by the record that names it, so a record that stops naming one leaves it owned by nobody. Three paths let that happen and one of them disables the collector.

Registration that replaces a reference. Pulling or loading a reference that already exists replaces its record and leaves the replaced generation's unpacked snapshot owned by no image. Nothing collects those outside the delete and prune commands, so a machine that rebuilds or refreshes the same tags accumulates dead snapshots indefinitely; a week of image rebuilds on one machine held 28G this way, invisible to every per-run cleanup. The sweep now runs after every operation that lands records, using the same keep set the prune path uses, and its failure warns rather than failing the pull or load that carried it.

Removal. Registration over an existing reference was swept and removal was not, so every deleted image left its unpacked snapshot behind. That penalises the tidy caller: an integration run that deletes each image its builds register leaves a snapshot per build, and a suite of them fills a disk that the same suite without the tidying would not have.

One bad item stopping the pass. The sweep reads every image to learn which snapshot it claims, then removes the snapshots no image claims, and a failure anywhere in either loop abandoned the whole pass. One record from an interrupted pull, whose content the store no longer holds, stopped every snapshot from ever being reclaimed again for as long as that record existed; and one snapshot that would not remove abandoned the rest of the set and discarded the bytes already freed, so the caller was told nothing had been reclaimed when some of it had. Each image is read and each snapshot removed on its own terms now, with the error logged against the item that raised it. An image the store cannot read is one it cannot unpack or run either, so the snapshot that image would have claimed is not worth keeping the store full for.

Serialization. The service is an actor, and every await inside a store operation admits the next call, so a collection can overlap an in-flight landing: its keep set is computed without the arriving image, and the cleanup prunes blobs the landing is about to reference and sweeps the snapshot a container is about to attach. A lock held across each whole mutating operation admits one at a time, the way PodsService holds its lock across lifecycle operations. containerd guards the same window with leases: https://github.com/containerd/containerd/blob/main/docs/garbage-collection.md

An integration run here left 21G of unpacked filesystems for images nothing named, and the control plane then failed on the disk they took.

Testing

  • Tested locally
  • Added/updated tests
  • Added/updated docs

Integration suite: 397 passed. Unit suite: 772 passed. make fmt, make check clean.

MayCXC added 4 commits August 27, 2026 20:16
Pulling or loading a reference that already exists replaces its record
and leaves the replaced generation's unpacked snapshot owned by no
image. Nothing reclaims those directories outside the delete and prune
commands, so a machine that rebuilds or refreshes the same tags
accumulates dead snapshots indefinitely; a week of image rebuilds on one
machine held 28G this way, invisible to every per-run cleanup.

Sweep the snapshot store after every operation that lands records, using
the same keep set the prune path uses. The sweep is housekeeping: its
failure warns in the log and never fails the pull or load that carried
it.
The service is an actor, and every await inside a store operation
admits the next call, so a garbage collection can overlap an in-flight
landing: its keep set is computed without the arriving image, and the
cleanup prunes blobs the landing is about to reference and sweeps the
snapshot a container is about to attach. A lock held across each whole
mutating operation admits one at a time, the way PodsService holds its
lock across lifecycle operations; containerd guards the same window
with leases.
https://github.com/containerd/containerd/blob/main/docs/garbage-collection.md
A snapshot is owned by the record that names it, so a record that stops
naming it leaves it owned by nobody. Registration over an existing
reference was swept and removal was not, so every deleted image left its
unpacked snapshot behind, and nothing outside the prune verbs ever
collected one.

What made that expensive is that removal is what a caller does when it
is being tidy: an integration run that deletes each image its builds
register leaves a snapshot per build, and a suite of them fills a disk
that the same suite without the tidying would not have.
The sweep reads every image to learn which snapshot it claims, then
removes the snapshots no image claims. A failure anywhere in either
loop left the whole pass. Reading an image whose content the store no
longer holds fails, so one record from an interrupted pull stopped
every snapshot from ever being reclaimed again for as long as the
record existed; and one snapshot that would not remove abandoned the
rest of the set and took the bytes already freed with it, so the
caller was told nothing had been reclaimed when some of it had. An
integration run left twenty one gigabytes of the unpacked filesystems
of images nothing names, and the control plane failed on the disk
they took.

Read each image and remove each snapshot on its own terms, logging
the error against the item that raised it. An image the store cannot
read is one it cannot unpack or run either, so the snapshot that
image would have claimed is not worth keeping the store full for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Unpacked snapshots leak on image replace and delete, and one unreadable record disables the sweep for good

2 participants