Is there any examples for using a Block pvc as root disk?

[weixiao-huang]

I try to write a vm.yaml below

apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
metadata:
  name: ubuntu-container-rootfs
spec:
  instance:
    memory:
      size: 4Gi
    kernel:
      image: smartxworks/virtink-kernel-5.15.12
      cmdline: "console=ttyS0 root=/dev/vda rw"
    disks:
      - name: ubuntu
      - name: cloud-init
    interfaces:
      - name: pod
  volumes:
    - name: ubuntu
      persistentVolumeClaim:
        claimName: test-vm
    - name: cloud-init
      cloudInit:
        userData: |-
          #cloud-config
          password: password
          chpasswd: { expire: False }
          ssh_pwauth: True
  networks:
    - name: pod
      pod: {}

and have a PVC test-vm like

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vm
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      pv: test-vm
  storageClassName: csi-rbd-storageclass
  volumeMode: Block
  volumeName: test-vm
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  phase: Bound

And I want to use the pvc test-vm as root disk for vm. Is there any examples for how to import rootfs into this block pvc device?

[weixiao-huang]

I tried to use

FROM ubuntu:jammy AS rootfs
RUN apt-get update -y && \
    apt-get install -y --no-install-recommends systemd-sysv udev lsb-release cloud-init sudo openssh-server && \
    rm -rf /var/lib/apt/lists/*

to build a docker image ubuntu-rootfs and use

docker create --name ubuntu-rootfs ubuntu-rootfs
docker export ubuntu-rootfs | tar -xvf - -C ./my-rootfs-pvc-mount-dir

which ./my-rootfs-pvc-mount-dir is the mount path where the pvc mounted manually in the host machine.

After doing this, this Block PVC should be used in the VM.

Could I open a PR for adding it into the docs?

On the other hand, the rootfs disk type seems does not support xfs. I use ext4 for successfully creating the VM but by using xfs, I got the error below:

[    0.915391]  driver: virtio_blk
[    0.917481] No filesystem could mount root, tried:
[    0.917481]  ext3
[    0.918974]  ext2
[    0.919595]  ext4
[    0.920285]  vfat
[    0.920910]  msdos
[    0.921529]  iso9660
[    0.922170]  fuseblk
[    0.922860]
[    0.924047] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0)
[    0.926612] Kernel Offset: disabled
[    0.927685] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0) ]---

Could virtink or cloud-hypervisor support xfs rootfs disk type?

[fengye87]

The kernel smartxworks/virtink-kernel-5.15.12 we provided doesn't include the xfs module. We try to keep it minimal, and you can include any kernel modules you like and rebuild it.

As for using PVC (both FS mode and block mode) as the root disk, I'd recommend you packing and publishing your rootfs as a QCOW2 image and then using CDI datavolume to import it. This is a more general and automated way.

[weixiao-huang]

But I could not use a CDI datavolume by using direct kernel boot. I only successfully in the Block PVC imported with

docker create --name ubuntu-rootfs ubuntu-rootfs
docker export ubuntu-rootfs | tar -xvf - -C ./my-rootfs-pvc-mount-dir

I think this method may be added in the docs.

[fengye87]

Direct kernel boot should work with CDI data volumes. Could you share steps to reproduce it if it didn't work? @weixiao-huang

[wavemomo]

I tried to start vm with pvc, but I get an error:

My steps:

1. use hostpath to create a pv：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-hostpath
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  hostPath:
    path: "/mnt/local_pv"

2. create a pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-hostpath
spec:
  storageClassName: manual
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      type: local
  volumeMode: Block
  volumeName: pv-hostpath
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound

3. build a rootfs：
   Dockerfile：

FROM ubuntu:jammy AS rootfs
RUN apt-get update -y && \
   apt-get install -y --no-install-recommends systemd-sysv udev lsb-release cloud-init sudo openssh-server && \
   rm -rf /var/lib/apt/lists/*
RUN apt-get install python3

FROM smartxworks/virtink-container-rootfs-base
COPY --from=rootfs / /rootfs
RUN ln -sf ../run/systemd/resolve/stub-resolv.conf /rootfs/etc/resolv.conf

run the commands：

docker build -t ubuntu-rootfs .
docker create --name ubuntu-rootfs ubuntu-rootfs
docker export ubuntu-rootfs | tar -xvf - -C /mnt/local_pv

4. tried to start vm：

apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
metadata:
  name: test-vm
spec:
  instance:
    memory:
      size: 4Gi
    kernel:
      image: smartxworks/virtink-kernel-5.15.12
      cmdline: "console=ttyS0 root=/dev/vda rw"
    disks:
      - name: ubuntu
      - name: cloud-init
    interfaces:
      - name: pod
  volumes:
    - name: ubuntu
      persistentVolumeClaim:
        claimName: pvc-hostpath
    - name: cloud-init
      cloudInit:
        userData: |-
          #cloud-config
          password: password
          chpasswd: { expire: False }
          ssh_pwauth: True
  networks:
    - name: pod
      pod: {}

Thanks for your reply.

[fengye87]

@wavemomo Although you specified volumeMode: Block in the PV spec, I don't think hostPath PV can be a block PV (correct me if I'm wrong). To use direct kernel boot with a block PV, you need to:

1. Have a raw block mode PV. You can verify if it's a raw block mode PV by using it in a pod's volumeDevices. And it should appear as a raw block device (not a directory) inside the pod.
2. Import your rootfs content using CDI or manually. To manually import your rootfs into a raw block device, you should first format the raw block device and mount it. Then you can copy your rootfs into it. Please be noted, for the image built by the rootfs Dockerfile in our sample, the rootfs content is inside the /rootfs path, not the image root itself.
3. Use the imported PV in the VM.

Importing a rootfs into a PV manually is a non-trivial task. That's why we recommend users to first pack the rootfs as a QCOW2 image and then use CDI to import it. To pack a rootfs into a QCOW2 image, you need to:

1. Build the rootfs image as you would using the Dockerfile.
2. truncate a large enough empty file to be the raw disk file
3. Format and mount that raw disk file
4. Copy your rootfs content into it
5. Unmount it
6. Convert the raw disk file to a QCOW2 file using qemu-img command

[weixiao-huang]

I use Ceph RBD to use Block PVC. By using this, it could work.

[weixiao-huang]

@wavemomo Your Dockerfile is not correct. The Dockerfile I used is

FROM ubuntu:jammy AS rootfs
RUN apt-get update -y && \
    apt-get install -y --no-install-recommends systemd-sysv udev lsb-release cloud-init sudo openssh-server && \
    rm -rf /var/lib/apt/lists/*

[wavemomo]

@fengye87 @weixiao-huang I used CDI to do it. But I still had some problem.

1. use kubectl logs -f vm-test-vm-xxx:
   it looks like successfuly, but the pod will quit later.

2. use kubectl describe vm test-vm :

[wavemomo]

when use the same vm.yaml to start vm again, it will block forever.

[fengye87]

This could be a different problem or bug, would you like to open an issue so that we can track it separately? And when you do, please attach your VM yaml and Dockerfile if any. Thanks. @wavemomo

[wavemomo]

OK.