Storage types suitable for live migration.

[kelvich]

I've tried to live-migrate VM created in a two-node k3s cluster. Using the masquerade network and dataVolume storage I was able to create a migration. However, I'm getting this error:

Warning  FailedScheduling  3m1s   default-scheduler  0/2 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had volume node affinity conflict. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling.

IIUC, it is because PersistentVolume is using the "local-path" storage class and will not be available on the second node. So I guess I have to try to set up some DFS-backed storage class (I will try). At the same time, as far as I understand, cloud-hypervisor can transfer disk data as part of the migration process. So is it in principle possible to support migration with e.g. containerRootfs?

[cicdteam]

PV solution for bare-metal clusters easy to try: https://longhorn.io
it simple to install by helm https://github.com/longhorn/longhorn/tree/master/chart

[fengye87]

@kelvich Yes, currently we don't support migration with local disks, a shared storage is required to make live migration work. There's plenty of options there, any NFS can work. Or if you'er looking for something more production ready, may I recommend you IOMesh, a battle-tested and blazing-fast block storage for Kubernetes.

As for the migration with local disks, we may or may not support it in the future, as this is not quite efficient and may take an unexpectedly long time to finish the migration. A shared storage is strongly recommended for live migration and it also helps in VM recovery situation. When a host dies along with all its VMs, these VMs can be started in another host if all its disks are in a shared storage.