-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add nix-snapshotter support to the embedded containerd #9319
base: master
Are you sure you want to change the base?
Conversation
This is interesting; can you describe in more detail how it might be useful, and when it can be used? See the eStargz documentation for an example of what sort of info we might be looking for before adding another snapshotter. |
Nix is a package manager / build system that has a complete understanding of build & runtime inputs for every package. Nix packages are stored in a global hashed path like: Runtime dependencies down to
If you inspect its ELF data, you can indeed see its linked against that specific glibc: $ readelf -d /nix/store/s66mzxpvicwk07gjbjfw9izjfa797vsw-hello-2.12.1/bin/hello | grep runpath
0x000000000000001d (RUNPATH) Library runpath: [/nix/store/3n58xw4373jp0ljirf06d8077j15pc4j-glibc-2.37-8/lib] This means that a root filesystem containing that closure is sufficient to run Kubernetes is a great orchestration engine for minimal Nix images, but the current mechanism of building Nix-based image need to compress Nix packages into Docker layer tarballs. There are some heuristics to split Nix packages across layers to improve deduplication but because overlayfs has a 128 layer limit there's bound to be layers containing multiple Nix packages. Nix-snapshotter is a containerd snapshotter that natively understands Nix packages. It also provides a client-side library to build special OCI images with tiny layers that contain the Nix package closure in the For teams that deploy to both bare metal & Kubernetes, this is a huge boon to maintain a single stack and the overhead of containerization is just a few kilobytes of metadata (in the OCI manifest). Most teams using Nix already upload all their artifacts to a binary cache or S3, but also have to re-upload essentially the same data to a Docker registry backend. Lastly, since When using See the README and architecture docs for more details. There is also a HackerNews discussion. |
@brandond We have it working end-to-end now in a qemu VM with k3s built with our patches. It did require also forking For example in k3s master, we're currently pinned on Since this can happen asynchronously, wanted to bring up early that we need these two cherry picks into
|
Are those patches in containerd v1.7.13? If so we can look at updating to that for the March releases. |
e708b47
to
337292a
Compare
It's been merged for a while now but looks like it's only been cherry-picked to |
Signed-off-by: Edgar Lee <[email protected]>
337292a
to
0a7b227
Compare
Ah. I'm not sure when exactly we'll go to 2.0. |
I've started an issue in containerd to see if we can get it cherry-picked to In the mean time, we can maintain our patches for NixOS. |
@brandond I've managed to cherry-pick those changes into the next containerd |
Proposed Changes
Types of Changes
nix-snapshotter
as a go dependencynix
as a valid snapshotter option for the agentnix
snapshotter, also use it as an image serviceVerification
Dockerfile
to copy the k3s binary to thenixos/nix
docker image, to enable quick verification of k3s + nix-snapshotterTesting
User-Facing Change
Further Comments
Currently still work-in-progress, but wanted to start the conversation whether k3s has the appetite for including
nix
support. As noted in #9309, in rootless mode the only way is to use the embedded containerd, so rootless k3s + nix-snapshotter is only possible via a contribution.See this for an exciting feature of nix-snapshotter as an image service that enables fully declarative pods (down to the contents of the container image). An image like
nix:0/nix/store/f8b1hia3hcqwa5d46anzy3cszi3s6ybk-nix-image-redis.tar
is resolved by nix-snapshotter, which is a merkle hash of the Nix packages that make up the image tarball.