Plans for ZFS support? #3980
Replies: 24 comments 8 replies
-
|
We specifically removed the ZFS snapshotter. The reason being that we don't intend to include the ZFS user space as I believe that is not portable across kernel versions. So we can include the ZFS snapshotter and you would be required to first install the ZFS tools, which is common place already. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick feedback! The ZFS userspace tools are generally present when ZFS is used because otherwise ZFS cannot be used. So that’s really a non-issue as far as I can see. Except perhaps when running k3s in a Docker container on top of ZFS, but I’m not sure whether that would work anyway. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a timeline for including the ZFS snapshotter? |
Beta Was this translation helpful? Give feedback.
-
|
ZFS is sadly quite painful with Docker in Docker and similar scenarios. It might be best to avoid the problem by creating a volume in your ZFS pool, formatting that volume to ext4, and having docker use "overlay2" on top of that, instead of "zfs". The I just finished setting this up and it nicely solves my problems with k3s and also kind. I use these for testing and development, and there the volume should be just fine. |
Beta Was this translation helpful? Give feedback.
-
|
I was able to point k3s at the host's containerd (which had the zfs snapshotter and was configured to use it by default) and successfully run pods. $ k3s server --container-runtime-endpoint /run/containerd/containerd.sock
$ sudo k3s kubectl -n kube-system get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
local-path-provisioner-58fb86bdfd-tx9lt 1/1 Running 0 4m52s 10.88.0.137 rincon <none> <none>
coredns-57d8bbb86-7t6v6 1/1 Running 0 4m52s 10.88.0.136 rincon <none> <none>
$ zfs list -r vault/storage/containerd
NAME USED AVAIL REFER MOUNTPOINT
vault/storage/containerd 37.2M 466G 26K /var/lib/containerd/io.containerd.snapshotter.v1.zfs
vault/storage/containerd/1 450K 466G 450K legacy
vault/storage/containerd/10 25K 466G 20.0M legacy
vault/storage/containerd/3 15K 466G 451K legacy
vault/storage/containerd/4 15K 466G 451K legacy
vault/storage/containerd/5 238K 466G 238K legacy
vault/storage/containerd/6 3.67M 466G 3.67M legacy
vault/storage/containerd/7 19.8M 466G 20.0M legacy
vault/storage/containerd/8 13.0M 466G 16.6M legacy
vault/storage/containerd/9 35K 466G 16.6M legacy |
Beta Was this translation helpful? Give feedback.
-
|
@terinjokes Hi, could you please share more details about containerd with zfs support. How can I install containerd with zfs support? |
Beta Was this translation helpful? Give feedback.
-
|
Another possible workaround, again it involves creating ext4 filesystem, but this time using Docker itself through Docker loopback volume plugin, not by asking zfs to do it. Described in rancher/k3d docs https://github.com/rancher/k3d/blob/dc4c29361fd433d3592b62d3d8d23256a5ed5728/docs/examples.md#running-on-filesystems-k3s-doesnt-like-btrfs-tmpfs- |
Beta Was this translation helpful? Give feedback.
-
|
Well you can apparently leverage the host's docker for k3s, which can spawn containers on a ZFS-based graph. This however completely defeats the purpose of packaging CRI in k3s. |
Beta Was this translation helpful? Give feedback.
-
This is not necessary the case now for Proxmox-based distribution with version >= 5, and also now in Ubuntu 20.04 (albeit deemed "experimental"), both of which cherry-picks and integrates OpenZFS Linux module codebase into their kernel recipe. OpenZFS ABI is very stable but is denied entering kernel mainline due to some stupid license politics problem. And even if it is unstable, you can always use DKMS to hopefully keep it up with the new kernel. Despite this, I think k3s team still won't consider adding ZFS support anyway, but what about re-adding AUFS support? AUFS works generally fine atop of ZFS despite there are some reported edges cases related to file mounts not in sync, and AUFS is more general than ZFS and easier to merge in k3s I think. EDIT: aaaaaaaaaaaand btw the current setback is to create a ext4 volume as containerd backend like @jirkadanek did, but on a different location # echo "[+] making sure everything in containerd is empty to let it mount"
# rm -rf /var/lib/rancher/k3s/agent/containerd/*
# echo "[+] create the k3s hierarchy"
# zfs create rpool/k3s
# zfs create -s -V 128GB rpool/k3s/containerd
# echo "[+] wait for the newly created volume to format"
# mkfs.ext4 /dev/rpool/k3s/containerd
# echo "[+] adding the mount to fstab"
# echo "/dev/rpool/k3s/containerd /var/lib/rancher/k3s/agent/containerd ext4 auto 0 0" > /etc/fstab
# echo "[+] manually mounting the volume to the specific mount point"
# mount /dev/rpool/k3s/containerd /var/lib/rancher/k3s/agent/containerd
# echo "[!] if you saw this message without any error it means you're good to go"Keep in mind that this is still not perfect -- so you basically recreated snapshot on ZFS, and so the atomic COW goodies is probably gone, you have no fine control over the data size whatsoever, I think in the future maybe we could get to install k3s plugins to accommodate situations like these. ZFS is a very popular filesystem among Proxmox users after all, and it's a shame such a beautiful piece of work is being insulted like this. |
Beta Was this translation helpful? Give feedback.
-
|
Again,again,again - when you return ZFS snapshotter to containerd (v1.19.5+k3s2) ? Now I must use external containerd from debian testing (Bullseye) with ZFS snapshotter (and calico CNI): |
Beta Was this translation helpful? Give feedback.
-
|
I don't believe we've changed any of our plans around ZFS support, for the reasons @ibuildthecloud described in #66 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
For anyone else that's found this, https://blog.nobugware.com/post/2019/k3s-containterd-zfs/ gives some indication about using an external containerd with k3s. Personally, I plan on just setting up a zfs volume for |
Beta Was this translation helpful? Give feedback.
-
|
Which file system are you running on top of your ZFS volume for |
Beta Was this translation helpful? Give feedback.
-
👆 I use this as well for my single node k3s cluster and find it works great. |
Beta Was this translation helpful? Give feedback.
-
|
@AstraLuma @mhumeSF Can you guys exolain how you did exposed the zpool to the docker container? I deployed k3d but when tried to deploy the zfs-localpv I was getting errors indicating that the docker containers serving as nodes for the k3d cannot work with zfs, cannot see the zpool and etc. I willbe very grateful if someone of you describe how you exposed the zpool to the k3d so that it can be used by the zfs-localpv! |
Beta Was this translation helpful? Give feedback.
-
|
@lord-kyron I don't run k3d and I think k3s in docker with zfs support would be difficult see #66 (comment). I run k3s on a root volume that is ext4. |
Beta Was this translation helpful? Give feedback.
-
I'm not at all. I'm running k3s directly on the host. |
Beta Was this translation helpful? Give feedback.
-
|
Any update on this? It's quite strange to have minimal containerd configuration changes for ZFS because k3s refuses to include them in the distribution. |
Beta Was this translation helpful? Give feedback.
-
|
This works for me using Ubuntu containerd with zfs:
From mcerveny comment above |
Beta Was this translation helpful? Give feedback.
-
No, don't do this. Don't tell other people to do this. Things WILL break when you update k3s and the bin directory gets changed, but your config is now hardcoded pointing at whatever bin directory was in use at the time you ran this command. Or we need to change other things. Use the config template as described in the documentation, and make your changes to the template. |
Beta Was this translation helpful? Give feedback.
-
|
This is configuring Ubuntu containerd, not k3s containerd which is disabled by --container-runtime-endpoint flag. The Ubuntu containerd package doesn't install config.toml so it needs to be generated by this command. The containerd man page says the config is /etc/contairned/config.toml so I don't think it will change. |
Beta Was this translation helpful? Give feedback.
-
|
You could make it more clear that these instructions are for folks using a standalone containerd, not for the containerd embedded in k3s. |
Beta Was this translation helpful? Give feedback.
-
|
or, you know, just use the native snapshotter and it works out of the box as opposed to continuing to inflict incredible pain on people. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe a dumb question, but why not use the Speaking of ZFS snapshotter, I heard not so positive feedback. |
Beta Was this translation helpful? Give feedback.
-
|
No, ZFS is absolutely not ready for production on Linux and you will not only run into performance issues but will end up destroying your system. While this continues to be the case (and don't let anyone continue to lie about it), it's unrelated to k3s support for the filesystem. |
Beta Was this translation helpful? Give feedback.
-
|
Could you provide some details on what is / will go wrong when using ZFS? I suppose you mean the issue lies with the snapshotter, right, not ZFS itself? |
Beta Was this translation helpful? Give feedback.
-
|
For anyone not aware this will likely be upstreamed: truenas/zfs#101 |
Beta Was this translation helpful? Give feedback.
-
|
Very nice to hear, although it may take time to reach (LTS) Linux distros. |
Beta Was this translation helpful? Give feedback.
-
|
Speaking of snapshotters, I see than containerd in k3s ships with the Stargz snapshotter thanks to k3s-io/containerd@01a87ab. |
Beta Was this translation helpful? Give feedback.
-
|
Please note that ZFS 2.2.0 with Linux container support is nearing completion: |
Beta Was this translation helpful? Give feedback.
-
|
What kind of "ZFS support" would actually be expected from K3s, now that the |
Beta Was this translation helpful? Give feedback.
-
|
Closing this as we have included the zfs snapshotter plugin for a while now. See: #7660 (comment) |
Beta Was this translation helpful? Give feedback.
-
Deployment on ZFS is currently not possible, because OverlayFS does not work with ZFS:
From the containerd logfile, it appears the ZFS snapshotter is not included:
ZFS support would be awesome! Are there any plans to include that?
Beta Was this translation helpful? Give feedback.
All reactions