From d70a96811132f1146c664d9d20a276065df728eb Mon Sep 17 00:00:00 2001 From: Feng Ye Date: Wed, 10 Aug 2022 15:46:06 +0800 Subject: [PATCH] docs: Add doc for VM interfaces and networks --- README.md | 6 +- docs/interfaces_and_networks.md | 196 ++++++++++++++++++++++++++++++++ 2 files changed, 199 insertions(+), 3 deletions(-) create mode 100644 docs/interfaces_and_networks.md diff --git a/README.md b/README.md index 4d34693..eb558ef 100644 --- a/README.md +++ b/README.md @@ -135,13 +135,13 @@ You can also `Shutdown`, `Reset`, `Reboot` or `Pause` a running VM, or `Resume` - [x] VM lifecycle management - [x] Container disks - [x] Direct kernel boot with container rootfs -- [x] Pod network -- [x] Multus CNI networks +- [x] [Pod network](docs/interfaces_and_networks.md#pod-network) +- [x] [Multus CNI networks](docs/interfaces_and_networks.md#multus-network) - [x] Persistent volumes - [x] CDI data volumes - [x] ARM64 support - [ ] VM live migration -- [ ] SR-IOV NIC passthrough +- [ ] [SR-IOV NIC passthrough](docs/interfaces_and_networks.md#sriov-mode) - [ ] GPU passthrough - [ ] Dedicated vCPU allocation - [ ] VM devices hot-plug diff --git a/docs/interfaces_and_networks.md b/docs/interfaces_and_networks.md new file mode 100644 index 0000000..40d25a6 --- /dev/null +++ b/docs/interfaces_and_networks.md @@ -0,0 +1,196 @@ +# Interfaces and Networks + +Connecting a VM to a network consists of two parts. First, networks are specified in `spec.networks`. Then, interfaces backed by the networks are added to the VM by specifying them in `spec.instance.interfaces`. Each interface must have a corresponding network with the same name. + +An interface defines a virtual network interface of a VM. A network specifies the backend of an interface and declares which logical or physical device it is connected to. + +There are multiple ways of configuring an interface as well as a network. + +## Networks + +Networks are configured in `spec.networks`. Each network should declare its type by defining one of the following fields: + +| Type | Description | +| -------- | ------------------------------------------------------------------------------------------------- | +| `pod` | Default Kubernetes network | +| `multus` | Secondary network provided using [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) | + +### `pod` Network + +A `pod` network represents the default pod `eth0` interface configured by cluster network solution that is present in each pod. + +```yaml +apiVersion: virt.virtink.smartx.com/v1alpha1 +kind: VirtualMachine +spec: + instance: + interfaces: + - name: pod + bridge: {} + networks: + - name: pod + pod: {} +``` + +### `multus` Network + +It is also possible to connect VMs to secondary networks using [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni). This assumes that Multus CNI is installed across your cluster and a corresponding `NetworkAttachmentDefinition` CRD was created. + +The following example defines a network which uses the [Open vSwitch CNI plugin](https://github.com/k8snetworkplumbingwg/ovs-cni), which will connect the VM to Open vSwitch's bridge `br1` on the host. Other CNI plugins such as [bridge](https://www.cni.dev/plugins/current/main/bridge/) or [macvlan](https://www.cni.dev/plugins/current/main/macvlan/) might be used as well. For their installation and usage refer to the respective project documentation. + +First the `NetworkAttachmentDefinition` needs to be created. + +```yaml +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: ovs-br1 +spec: + config: | + { + "cniVersion": "0.3.1", + "type": "ovs", + "bridge": "br1" + } +``` + +With following definition, the VM will be connected to the secondary Open vSwitch network. + +```yaml +apiVersion: virt.virtink.smartx.com/v1alpha1 +kind: VirtualMachine +spec: + instance: + interfaces: + - name: ovs + bridge: {} + networks: + - name: ovs + multus: + networkName: ovs-br1 +``` + +## VM Network Interfaces + +VM network interfaces are configured in `spec.instance.interfaces`. They describe properties of virtual interfaces as "seen" inside guest instances. The same network may be connected to a VM in multiple different ways, each with their own connectivity guarantees and characteristics. + +Each interface should declare its type by defining one of the following fields: + +| Type | Description | +| -------- | ---------------------------------------- | +| `bridge` | Connect using a linux bridge | +| `sriov` | Passthrough a SR-IOV PCI device via VFIO | + +### `bridge` Mode + +In `bridge` mode, VMs are connected to the network through a Linux bridge. The pod network IPv4 address is delegated to the VM via DHCPv4. The VM should be configured to use DHCP to acquire IPv4 addresses. + +```yaml +apiVersion: virt.virtink.smartx.com/v1alpha1 +kind: VirtualMachine +spec: + instance: + interfaces: + - name: pod + bridge: {} + networks: + - name: pod + pod: {} +``` + +At this time, `bridge` mode doesn't support additional configuration fields. + +> **Note**: due to IPv4 address delegation, in `bridge` mode the pod doesn't have an IP address configured, which may introduce issues with third-party solutions that may rely on it. For example, Istio may not work in this mode. + +### `sriov` Mode + +In `sriov` mode, VMs are directly exposed to an SR-IOV PCI device, usually allocated by [SR-IOV Network Device Plugin](https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin). The device is passed through into the guest operating system as a host device, using the [VFIO](https://www.kernel.org/doc/html/latest/driver-api/vfio.html#:~:text=The%20VFIO%20driver%20is%20an,non%2Dprivileged%2C%20userspace%20drivers.) userspace interface, to maintain high networking performance. + +#### How to Expose SR-IOV VFs to Virtink + +First you should have [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni), [SR-IOV CNI](https://github.com/k8snetworkplumbingwg/sriov-cni) and [SR-IOV Network Device Plugin](https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin) installed across the cluster. For their installation and usage refer to the respective project documentation. + +Then you should create some VFs on the host's SR-IOV capable device. Please consult the vendor of your device for how to do so. + +To expose SR-IOV VFs to Virtink, each VF's driver should be changed to `vfio-pci`. Below is an example of how to do so: + +```bash +export VF_ADDR=0000:58:01.2 # change it to your VF's PCI address +modprobe vfio_pci +export DRIVER=$(lspci -s $VF_ADDR -k | grep driver | awk '{print $5}') +echo $VF_ADDR > /sys/bus/pci/drivers/$DRIVER/unbind +export VENDOR_ID=$(lspci -s $VF_ADDR -Dn | awk '{split($3,a,":"); print a[1]}') +export DEVICE_ID=$(lspci -s $VF_ADDR -Dn | awk '{split($3,a,":"); print a[2]}') +echo $VENDOR_ID $DEVICE_ID > /sys/bus/pci/drivers/vfio-pci/new_id +``` + +Now create a config for SR-IOV device plugin to capture this VF: + +```bash +echo $VENDOR_ID # make sure it's the vendor ID of your VF +echo $DEVICE_ID # make sure it's the device ID of your VF +cat <