diff --git a/docs/kubevirt-datamover/backup-restore.md b/docs/kubevirt-datamover/backup-restore.md new file mode 100644 index 00000000000..9bd0e841f26 --- /dev/null +++ b/docs/kubevirt-datamover/backup-restore.md @@ -0,0 +1,199 @@ +# Backing Up and Restoring VirtualMachines with KubeVirt DataMover + +This guide walks through a complete backup and restore cycle for a KubeVirt VirtualMachine using the KubeVirt DataMover feature. It assumes you have already enabled and configured KubeVirt DataMover as described in [configuration.md](./configuration.md). + +## Overview + +KubeVirt DataMover backs up VM disks by taking QEMU-level snapshots and tracking changed blocks between backups, instead of relying on CSI volume snapshots. From your point of view as a Velero user, the workflow looks the same as any other Velero backup and restore: you create a `Backup` object, Velero backs up the namespace, and later you create a `Restore` object to bring it back. The difference happens behind the scenes, where the kubevirt-datamover-plugin and kubevirt-datamover-controller take over the disk data movement for you. + +## Step 1: Label the VM for Changed Block Tracking + +CBT has to be turned on per VM, in addition to being enabled at the HCO level. Add the `changedBlockTracking: "true"` label to the VirtualMachine: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: my-vm + namespace: my-vm-namespace + labels: + changedBlockTracking: "true" +spec: + dataVolumeTemplates: + - metadata: + name: my-vm-disk + spec: + pvc: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + volumeMode: Block + source: + registry: + pullMethod: node + url: docker://your-image + running: true + template: + spec: + domain: + devices: + disks: + - disk: + bus: virtio + name: rootdisk + volumes: + - dataVolume: + name: my-vm-disk + name: rootdisk +``` + +Two things matter here for KubeVirt DataMover to work correctly: + +- The disk's `volumeMode` must be `Block`. CBT tracking relies on being able to read raw changed blocks off the underlying volume, which is not available in filesystem-mode PVCs. +- The label goes on the `VirtualMachine`, not on the `DataVolume` or `PersistentVolumeClaim`. + +If you apply the label to an existing VM that is already running, you may need to restart the VM (stop and start it again) for CBT to actually start tracking, depending on your KubeVirt version. Check that CBT is active on the VirtualMachine itself: + +```bash +oc get vm my-vm -n my-vm-namespace -o jsonpath='{.status.changedBlockTracking.state}' +``` + +You should see `Enabled`. If it isn't, try restarting the VM (`virtctl stop` then `virtctl start`), or simply proceed to the backup step below and confirm the first backup succeeds as a full backup. + +## Step 2: Create a volume policy that routes VM disks through KubeVirt DataMover + +Create a ConfigMap containing the volume policy, if you have not already done so as part of your DPA configuration. The ConfigMap must have exactly one entry under `data`, but the key name does not matter, Velero reads whatever single value is there. `policy.yaml` is just the conventional name used in most examples: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: kubevirt-volume-policy + namespace: openshift-adp +data: + policy.yaml: | + version: v1 + volumePolicies: + - conditions: {} + action: + type: custom + parameters: + datamover: kubevirt +``` + +If you would rather keep your policy in a separate file and create the ConfigMap from it, that works the same way: + +```bash +oc create cm kubevirt-volume-policy -n openshift-adp --from-file policy.yaml +``` + +See [configuration.md](./configuration.md#volume-policy-configuration) for more on how volume policy matching works and what to watch out for with catch-all entries. + +## Step 3: Run a backup + +Create a Velero `Backup` that references the namespace containing your VM and the volume policy ConfigMap: + +```yaml +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: my-vm-backup + namespace: openshift-adp +spec: + includedNamespaces: + - my-vm-namespace + defaultVolumesToFsBackup: false + snapshotMoveData: true + resourcePolicy: + kind: ConfigMap + name: kubevirt-volume-policy +``` + +Or create the same backup with the `oc oadp` CLI plugin instead of writing the YAML by hand: + +```bash +oc oadp backup create my-vm-backup --include-namespaces my-vm-namespace --resource-policies-configmap kubevirt-volume-policy --snapshot-move-data +``` + +`snapshotMoveData: true` is required. KubeVirt DataMover always moves the backed up data to your object storage location, it does not leave data sitting in an in-cluster snapshot the way a CSI-only backup might. + +Watch the backup progress: + +```bash +oc get backups.velero.io my-vm-backup -n openshift-adp -w +``` + +Behind the scenes, when Velero gets to the VM's disks, the kubevirt-datamover-plugin creates a `DataUpload` custom resource with `spec.datamover: kubevirt`. The kubevirt-datamover-controller picks that up and works through a series of phases: `New`, `Accepted`, `Prepared`, `InProgress`, and finally `Completed` (or `Failed` if something goes wrong). You can watch this directly if you want more granular visibility than the Backup object gives you: + +```bash +oc get datauploads.velero.io -n openshift-adp -w +``` + +Along the way, the controller creates a KubeVirt `VirtualMachineBackup` for your VM to trigger the actual CBT snapshot, and a `VirtualMachineBackupTracker` to record the checkpoint chain for that VM. The `VirtualMachineBackup` is temporary: once a backup finishes, the controller archives its state into your object storage bucket and removes it from the cluster, so do not be surprised if you cannot find it afterward. The `VirtualMachineBackupTracker` behaves differently and is left on the cluster between backups on purpose, so KubeVirt can use it to redefine the VM's libvirt checkpoint across restarts and live migrations. You will see it stick around in the VM's namespace even after a backup completes, that is expected. + +### When a full backup happens automatically + +You don't need to manage full-versus-incremental yourself. The controller decides this on its own, and falls back to a full backup automatically in a few situations: when it can't find or validate a previous checkpoint chain in your BackupStorageLocation (for example, if something in the bucket was deleted or changed outside of normal operation), or when the `maxIncrementalBackups` limit configured on the DPA has been reached for that VM (see [configuration.md](./configuration.md)). Restarting the VM does not force a full backup and does not invalidate its checkpoint chain, a backup taken after a restart stays incremental as normal, because the controller deliberately keeps the VM's `VirtualMachineBackupTracker` on the cluster across restarts rather than deleting it. If that tracker object is ever missing when a new backup starts, either because it was deleted manually or the VM's namespace was recreated, the controller tries to rebuild it from the archived state in object storage first, and only falls back to a full backup if that archive can't be found either. There is currently no supported way to request a one-off full backup directly from the Backup or VirtualMachine object, and manually editing or deleting anything in object storage is not a supported way to reset the chain either. If you need a full backup on demand, lower `maxIncrementalBackups` (either on the DPA or with the per-VM `kubevirt-datamover.io/max-incremental-backups` annotation) so the next backup crosses the limit and falls back to full. + +## Step 4: Confirm the backup completed successfully + +```bash +oc get backups.velero.io my-vm-backup -n openshift-adp -o jsonpath='{.status.phase}' +``` + +You should see `Completed`. Check the `DataUpload` object's phase too, since Velero can sometimes report a backup as complete while individual DataUploads are still finishing up in edge cases: + +```bash +oc get datauploads.velero.io -n openshift-adp -l velero.io/backup-name=my-vm-backup +``` + +## Step 5: Restore the VM + +Delete or otherwise lose your VM (or restore into a different namespace/cluster), then create a Velero `Restore`: + +```yaml +apiVersion: velero.io/v1 +kind: Restore +metadata: + name: my-vm-restore + namespace: openshift-adp +spec: + backupName: my-vm-backup +``` + +```bash +oc apply -f restore.yaml +oc get restore my-vm-restore -n openshift-adp -w +``` + +On the restore path, the kubevirt-datamover-plugin's Restore Item Action creates a `DataDownload` resource, which the controller processes through its own phase sequence. The controller downloads the checkpoint chain from object storage and reconstructs the disk image using `qemu-img rebase`, chaining each incremental checkpoint onto its parent rather than flattening everything onto the full backup in one step. This preserves the same layered structure the backup had, and lets the controller avoid downloading the full backup data again for every incremental restore. + +Once the `DataDownload` reaches `Completed`, the restored PVC is bound and rebound to the new VM created by Velero's standard VM restore path (handled by kubevirt-velero-plugin), and the VM should come up with its data intact. + +### Expect the restored VM to start out halted + +Before the restore, the plugin stops the VM and remembers whether it was running or stopped at backup time. This is expected and not a sign of a failed restore. If the VM had more than one disk, each disk gets its own `DataDownload`, and the controller only puts the VM back into its original run state once every one of those `DataDownload`s for that VM has reached `Completed`. In practice this means a freshly restored multi-disk VM can sit in a Halted or Stopped state for a little while, then start on its own once all of its disks are done. Don't start the VM manually while restores are still in progress, just wait for it to come up by itself. + +Check that the VM started correctly: + +```bash +oc get vm my-vm -n my-vm-namespace +oc get vmi my-vm -n my-vm-namespace +``` + +## Verifying data integrity + +For a meaningful test, write something identifiable to the VM's disk before backing it up (a file, a database record, whatever suits your workload), take the backup, delete the VM, restore it, and confirm the same data is present. This is exactly the pattern OADP's own end-to-end tests use, and it is the best way to build confidence in your specific storage backend and VM configuration before relying on this for production backups. + +## Incremental backup chains and full backups over time + +Left running long enough, a VM will accumulate a chain of incremental backups, each depending on the one before it. There are two mechanisms that eventually force a new full backup, so the chain does not grow forever: + +- **maxIncrementalBackups**: configured DPA-wide or per VM (see [configuration.md](./configuration.md)), this caps how many incrementals can chain together before the controller starts a new full backup automatically. +- **Broken chain detection**: if the controller cannot validate the existing checkpoint chain against what is in the BackupStorageLocation (for example, if an earlier backup or checkpoint was deleted out from under it), it falls back to a full backup rather than failing outright. + +You do not need to manage this yourself under normal operation. It is worth knowing about if you notice a backup taking noticeably longer than expected, since that is usually a sign a full backup happened instead of an incremental one. + +For troubleshooting failed or stuck backups and restores, see [troubleshooting.md](./troubleshooting.md). diff --git a/docs/kubevirt-datamover/configuration.md b/docs/kubevirt-datamover/configuration.md new file mode 100644 index 00000000000..0130305bb25 --- /dev/null +++ b/docs/kubevirt-datamover/configuration.md @@ -0,0 +1,226 @@ +# KubeVirt DataMover Configuration + +This guide explains how to enable and configure the KubeVirt DataMover feature in OADP. KubeVirt DataMover backs up and restores KubeVirt VirtualMachines by using QEMU Changed Block Tracking (CBT) instead of CSI snapshots, which lets you back up VMs on storage backends that do not support CSI snapshots or Container Storage Interface volume clones. + +## How it fits together + +KubeVirt DataMover is made up of three pieces: + +- **kubevirt-datamover-plugin**: a Velero plugin (Backup Item Action, Restore Item Action, Delete Item Action) that intercepts VirtualMachine backups and creates `DataUpload`/`DataDownload` custom resources instead of letting Velero snapshot the underlying PVC directly. +- **kubevirt-datamover-controller**: a separate controller, deployed by the OADP operator, that watches those `DataUpload`/`DataDownload` resources and drives the actual CBT-based backup and restore using KubeVirt's `VirtualMachineBackup`/`VirtualMachineBackupTracker` APIs. +- **oadp-operator**: deploys the controller, wires up RBAC, and exposes configuration through the DataProtectionApplication (DPA) custom resource. + +You do not interact with the plugin or the controller directly. Everything is driven through the DPA and normal Velero `Backup`/`Restore` objects. + +## Prerequisites + +Before enabling KubeVirt DataMover, make sure your cluster meets these requirements: + +- OpenShift Virtualization (HCO) version 1.18 or later. HCO 1.18+ and the `backup.kubevirt.io` CRDs are required for CBT support. +- KubeVirt version 1.8.2 or later (this version includes a fix for a QEMU backup abort race that KubeVirt DataMover depends on). +- Changed Block Tracking enabled at the HCO level (see below). +- The VM you want to back up must be running. Offline (stopped) VM backup is not supported. +- An object storage backend configured as a Velero BackupStorageLocation. Backups that use KubeVirt DataMover must set `snapshotMoveData: true` on the Velero `Backup` object (or `spec.configuration.velero.defaultSnapshotMoveData: true` on the DPA to make it the default for all backups), since KubeVirt DataMover always moves data to object storage rather than leaving it as an in-cluster snapshot. + +### Enabling Changed Block Tracking + +CBT enablement is a two-part configuration on the `HyperConverged` (HCO) custom resource: enabling the feature gate, and telling KubeVirt which VM label to treat as opting a VM into CBT. + +First, enable the `incrementalBackup` feature gate. This is a first-class field on the HCO CR and automatically turns on the underlying `IncrementalBackup` and `UtilityVolumes` feature gates on the KubeVirt CR: + +```bash +oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type merge -p ' +spec: + featureGates: + incrementalBackup: true +' +``` + +Second, configure the label selector KubeVirt uses to decide which VMs have CBT enabled. This field, `changedBlockTrackingLabelSelectors`, lives on the underlying KubeVirt CR that HCO manages, so it has to be injected through a `kubevirt.kubevirt.io/jsonpatch` annotation on the HCO CR rather than set directly: + +```bash +oc annotate hyperconverged kubevirt-hyperconverged -n openshift-cnv --overwrite \ + kubevirt.kubevirt.io/jsonpatch='[{"op":"add","path":"/spec/configuration/changedBlockTrackingLabelSelectors","value":{"virtualMachineLabelSelector":{"matchLabels":{"changedBlockTracking":"true"}}}}]' +``` + +This example selector matches any VM labeled `changedBlockTracking: "true"`, which is the label used throughout this documentation and the sample manifests. You can choose a different label or match expression if you prefer, as long as it stays consistent between this HCO configuration and the labels you put on your VMs. + +Verify the configuration took effect: + +```bash +oc get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv \ + -o jsonpath='{.spec.configuration.changedBlockTrackingLabelSelectors}' +``` + +Expected output: + +```json +{"virtualMachineLabelSelector":{"matchLabels":{"changedBlockTracking":"true"}}} +``` + +Once both of these are in place, KubeVirt supports incremental backups based on changed disk blocks for VMs that carry the matching label, provided the VM's disks use `volumeMode: Block` (CBT tracking does not work with filesystem-mode volumes). See [backup-restore.md](./backup-restore.md) for how to label a VM and confirm CBT is active on it. + +## Enabling the plugin in the DPA + +Add `kubevirt-datamover` to `spec.configuration.velero.defaultPlugins` in your DataProtectionApplication: + +```yaml +apiVersion: oadp.openshift.io/v1alpha1 +kind: DataProtectionApplication +metadata: + name: velero-sample + namespace: openshift-adp +spec: + configuration: + velero: + defaultPlugins: + - openshift + - kubevirt + - kubevirt-datamover + podConfig: + nodeSelector: {} + nodeAgent: + enable: true + uploaderType: kopia + snapshotLocations: [] + backupLocations: + - velero: + provider: aws + default: true + config: + region: us-east-1 + profile: "default" + credential: + key: cloud + name: cloud-credentials + objectStorage: + bucket: my-backup-bucket + prefix: velero + features: {} +``` + +A few important notes about this configuration: + +- Always add both `kubevirt` and `kubevirt-datamover` together. The `kubevirt` plugin (kubevirt-velero-plugin) handles VM metadata and file-level restore concerns, while `kubevirt-datamover` handles the actual disk data movement. The operator will still let you enable `kubevirt-datamover` on its own, but it logs a warning if `kubevirt` is missing, and VM restores will not work correctly without it. +- KubeVirt DataMover can only be enabled on one DPA per cluster. If you try to enable it on a second DPA while another DPA already has it enabled and its controller deployment exists, the OADP operator rejects the DPA with a validation error. This is because the datamover controller is a cluster-scoped singleton, not a per-namespace component. +- You do not need to add anything to `snapshotLocations` for KubeVirt DataMover. It writes data straight to the object storage configured in your `backupLocations`. + +When the plugin is enabled, the OADP operator deploys a `Deployment` named `oadp-kubevirt-datamover-controller-manager` in the same namespace as the DPA. You can confirm it came up correctly: + +```bash +oc get deployment oadp-kubevirt-datamover-controller-manager -n openshift-adp +oc get pods -n openshift-adp -l control-plane=oadp-kubevirt-datamover-controller +``` + +## Tuning controller behavior + +The DPA exposes a small set of tuning knobs under `spec.configuration.kubevirtDatamover`: + +```yaml +apiVersion: oadp.openshift.io/v1alpha1 +kind: DataProtectionApplication +metadata: + name: velero-sample + namespace: openshift-adp +spec: + configuration: + kubevirtDatamover: + maxIncrementalBackups: 10 + maxConcurrentDataMovers: 5 + staleDataUploadThreshold: 3h + velero: + defaultPlugins: + - openshift + - kubevirt + - kubevirt-datamover +``` + +- **maxIncrementalBackups**: the number of incremental (changed-blocks-only) backups the controller will chain together before it forces a full backup for a given VM. Set to `0` (the default) for unlimited incrementals, meaning the controller will keep chaining incremental backups indefinitely unless something else forces a full backup, such as a broken checkpoint chain. You can also override this per VM with the `kubevirt-datamover.io/max-incremental-backups` annotation on the VirtualMachine, which takes priority over the DPA-wide setting. +- **maxConcurrentDataMovers**: the maximum number of active DataUploads the controller will process at the same time, and separately, the maximum number of active DataDownloads it will process at the same time. It's the same configured number applied to both, but DataUploads and DataDownloads are counted independently against it, so a value of `5` allows up to 5 backups and up to 5 restores running concurrently, not 5 total. Set to `0` (the default) for unlimited. If you have a large number of VMs, set this explicitly rather than leaving it unlimited, a Backup that targets many VMs at once will otherwise try to run all of their DataUploads concurrently, which can overload your storage backend or object storage endpoint. (There is an open proposal to change the shipped default from `0` to `3`, see [migtools/kubevirt-datamover-controller#193](https://github.com/migtools/kubevirt-datamover-controller/issues/193). This doc reflects the current default as of this writing.) +- **staleDataUploadThreshold**: how long a `DataUpload` can sit in an active phase (Accepted, Prepared, InProgress) before the controller treats it as stale and stops letting it block newer DataUploads for the same VM. Defaults to 2 hours. Raise this if your VMs have very large disks and backups routinely take longer than 2 hours to complete. + +All three fields are optional. If you leave them unset, the controller uses its built-in defaults. + +You can also override the temporary backup PVC size on a per-VM basis with the `kubevirt-datamover.io/backup-pvc-size` annotation on the VirtualMachine (a Kubernetes quantity, for example `50Gi`). The controller normally calculates this size from the VM's disk, so you only need this if you have a VM where the automatic sizing isn't giving you enough headroom. + +## Volume policy configuration + +Velero decides which backup path to use for a given PVC through volume policies. To route a VM's disks through KubeVirt DataMover rather than a CSI snapshot, add a custom action to your Velero volume policy that targets `kubevirt` as the datamover. The ConfigMap needs to hold exactly one data entry, Velero rejects a ConfigMap with zero or more than one key, but the key itself can be named anything you like. Velero just reads whatever single value is in `data` and treats it as the policy YAML, it does not look for a key called `policy.yaml` specifically. Most examples (including this one) use `policy.yaml` by convention, but if you create the ConfigMap from a file with a different name, that works fine too. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: kubevirt-volume-policy + namespace: openshift-adp +data: + policy.yaml: | + version: v1 + volumePolicies: + - conditions: {} + action: + type: custom + parameters: + datamover: kubevirt +``` + +You can also create the same ConfigMap directly from a policy file on disk instead of writing it inline: + +```bash +oc create cm kubevirt-volume-policy -n openshift-adp --from-file policy.yaml +``` + +An empty `conditions: {}` matches every volume, so combine this with `includedNamespaces` on your `Backup` (or a separate policy entry) if you need to be more selective about which PVCs get routed through KubeVirt DataMover. Velero evaluates volume policy entries in order and stops at the first match, so a catch-all entry like the one above always needs to come last in your `volumePolicies` list. If you put it first, it will match every volume and none of the more specific entries after it will ever be considered. If you want to condition on the CSI driver instead, Velero's `csi.driver` condition requires an exact driver name and does not support wildcards, so a plain `csi: {}` (matches any CSI-backed volume) or a fully spelled out driver name works, but `driver: "*"` does not match anything. + +Reference this ConfigMap from your Velero `Backup` object's `spec.resourcePolicy`: + +```yaml +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: my-vm-backup + namespace: openshift-adp +spec: + includedNamespaces: + - my-vm-namespace + snapshotMoveData: true + resourcePolicy: + kind: ConfigMap + name: kubevirt-volume-policy +``` + +Or reference it directly when creating the backup with the `oc oadp` CLI plugin instead of writing the YAML by hand: + +```bash +oc oadp backup create my-vm-backup --include-namespaces my-vm-namespace --resource-policies-configmap kubevirt-volume-policy --snapshot-move-data +``` + +Velero treats `custom` actions with unrecognized parameters as a signal to hand off data movement to whichever plugin claims the volume, which in this case is the kubevirt-datamover-plugin. + +If you back up a namespace that has both VMs and regular workloads, this policy only changes behavior for volumes attached to VirtualMachines. PVCs that are not owned by a VM do not meet the kubevirt-datamover-plugin's prerequisites, so the plugin will not pick them up even though the volume policy matched them. Velero itself has no way to tell in advance whether a given PVC belongs to a VM, so for non-VM volumes this custom policy effectively behaves the same as a `skip` action: Velero hands the volume off looking for a datamover plugin to claim it, none does, and the volume ends up not being moved by this policy at all. If you need non-VM PVCs in the same namespace to still get backed up through CSI snapshots or File System Backup, keep this custom policy scoped with `includedNamespaces` or a label selector so it only ever matches VM-owned volumes, rather than relying on it to fall back safely for everything else. + +## Storage provider and credential setup + +KubeVirt DataMover writes checkpoint data directly to the object storage bucket configured in your BackupStorageLocation. It supports the same providers OADP already supports for Velero: + +- **AWS S3 and S3-compatible storage**: standard secret-based credentials, or a projected service account token when using AWS STS (the controller automatically requests a token with the `openshift` audience and refreshes it, the same pattern used by the rest of OADP's STS support). +- **Azure Blob Storage**: storage account key, or Azure Workload Identity. If you have Workload Identity configured for OADP already (the `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and federated token secret), the operator automatically propagates the same identity to the datamover controller pod, so there is no separate Azure setup step. +- **Google Cloud Storage**: service account key with signing permissions. + +If your BackupStorageLocation credentials already work for regular Velero backups, they will work for KubeVirt DataMover without any extra configuration, because the controller reads the same BackupStorageLocation and credential secret that Velero uses. + +## Verifying the setup + +Once the DPA is applied and reconciled, check that the DPA condition reports the controller as ready: + +```bash +oc get dpa velero-sample -n openshift-adp -o jsonpath='{.status.conditions}' | jq +``` + +Look for a condition of type `KubevirtDatamoverReady`. If it is not `True`, check the controller pod logs: + +```bash +oc logs -n openshift-adp deployment/oadp-kubevirt-datamover-controller-manager +``` + +At this point you are ready to back up and restore VMs. See [backup-restore.md](./backup-restore.md) for the end-to-end workflow, and [troubleshooting.md](./troubleshooting.md) if something does not work as expected. diff --git a/docs/kubevirt-datamover/troubleshooting.md b/docs/kubevirt-datamover/troubleshooting.md new file mode 100644 index 00000000000..d18937512fc --- /dev/null +++ b/docs/kubevirt-datamover/troubleshooting.md @@ -0,0 +1,114 @@ +# Troubleshooting KubeVirt DataMover + +This guide covers the failure modes you are most likely to run into with KubeVirt DataMover, how to tell them apart, and where to look for more detail. It assumes you have gone through [configuration.md](./configuration.md) and are following the workflow in [backup-restore.md](./backup-restore.md). + +## Where to find logs + +The kubevirt-datamover-controller does the vast majority of the work, so start there: + +```bash +oc logs -n openshift-adp deployment/oadp-kubevirt-datamover-controller-manager +``` + +An important detail here: the datamover and downloader pods that actually move VM disk data are short-lived. When one finishes successfully, the controller captures its logs into the controller's own log output first, then deletes the pod, so you do not need to catch it running to see what happened inside it. Filter for lines mentioning "Datamover pod log" (DataUpload) or "Downloader pod log" (DataDownload), or the specific DataUpload/DataDownload name, if you need to isolate one operation. If a pod fails instead, the controller also captures its logs the same way, but it deliberately leaves the failed pod and its resources in place rather than cleaning them up, so you can still inspect it directly with `oc logs` or `oc describe pod` while you investigate. + +Also check events on the relevant objects, since the controller emits Kubernetes events for major phase transitions and failures: + +```bash +oc get events -n openshift-adp --field-selector involvedObject.kind=DataUpload +oc get events -n my-vm-namespace --field-selector involvedObject.kind=VirtualMachine +``` + +## Common issues + +### Backup stuck in Velero's InProgress phase + +Check the DataUpload phase directly: + +```bash +oc get datauploads.velero.io -n openshift-adp -o wide +oc describe datauploads.velero.io -n openshift-adp +``` + +Compare the DataUpload phase against the reconciliation flow: `New`, `Accepted`, `Prepared`, `InProgress`, then `Completed` or `Failed`. If it is stuck at `New` or `Accepted` for a long time, the controller may be waiting on another DataUpload for the same VM to finish first. Only one active DataUpload per VM is allowed at a time, which prevents two backups from stepping on the same checkpoint chain. Check whether an older DataUpload for the same VM is still active: + +```bash +oc get datauploads.velero.io -n openshift-adp -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,VM:.metadata.annotations.kubevirt-datamover\\.io/vm-name +``` + +If an older DataUpload appears to be stuck rather than genuinely still running, the controller has a built in safety valve for this: after the `staleDataUploadThreshold` (2 hours by default), a DataUpload that is still sitting in an active phase is treated as stale and no longer blocks newer DataUploads for the same VM. If you are seeing this often, it usually means something upstream (KubeVirt, storage, or the object storage endpoint) is failing silently rather than erroring out cleanly. Check the controller logs for the specific VM and look for repeated retries. + +### CBT not enabled or not working + +If the very first backup on a VM you expect to be incremental turns out to be a full backup every single time, or backups are much slower than expected, CBT is probably not actually active for that VM even if you added the label. Confirm: + +1. The VM has the `changedBlockTracking: "true"` label (this goes on the `VirtualMachine`, not the `DataVolume` or PVC). +2. The VM's disk volumes use `volumeMode: Block`. CBT does not work with filesystem-mode volumes. +3. HCO has the CBT feature gate enabled cluster-wide, and the HCO/KubeVirt version meets the minimum (HCO 1.18+, KubeVirt 1.8.2+). +4. The VM was restarted after the label was added, if you added it to an already-running VM. Some KubeVirt versions only pick up CBT at VM start. + +### CBT backup fails with "no space left on device" + +If a DataUpload fails and the underlying VirtualMachineBackup or virt-launcher logs mention `SyncFailed` and `No space left on device`, this is usually not a problem with your actual VM disk, it is the small overlay volume KubeVirt uses internally to track changed blocks running out of room. On some storage backends, that overlay volume gets provisioned at its bare minimum size (around 10-11Mi), which is not enough headroom for VMs with large disks or a lot of write activity between backups. + +The fix is to point that overlay volume at a storage class with a larger minimum allocation, by setting `vmStateStorageClass` on the HyperConverged CR: + +```bash +oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type merge \ + -p '{"spec":{"storage":{"vmStateStorageClass":"standard-csi"}}}' +``` + +Pick a storage class where PVCs round up to at least 1Gi or so (many CSI drivers do this automatically). This is a cluster-wide HCO setting, not something you configure per VM or through the DPA. + +### VM backup reports success but restored disk has no data + +On some VM configurations, most commonly Fedora or RHEL VMs using a DataSource-backed DataVolume with EFI and SMM firmware enabled, a VirtualMachineBackup can report `Done: True` and a `VirtualMachineBackupCompletedSuccessfully` condition while the actual backup PVC ends up empty or the restore comes back with none of the VM's data. If this happens, check the events on the VirtualMachineBackup and its associated pods for `HotplugFailed`, which is the real underlying failure that the top-level status doesn't currently surface clearly. Smaller VMs based on a plain containerdisk (CirrOS test images, for example) are not affected. Until this is fixed, treat a successful-looking VMB status on an EFI/SMM Fedora or RHEL VM with some suspicion and verify restored data directly rather than trusting the status alone. + +### Restore fails partway through + +Check the DataDownload's phase and events the same way you would for a DataUpload: + +```bash +oc get datadownloads.velero.io -n openshift-adp -o wide +oc describe datadownloads.velero.io -n openshift-adp +``` + +A restore failure partway through the checkpoint chain usually means one of the incremental checkpoints referenced in the chain is missing or corrupted in object storage. This can happen if someone manually deleted objects out of the bucket, or if a lifecycle policy on the bucket expired objects that were still referenced by a VM's checkpoint chain. If you use bucket lifecycle rules, make sure they exclude the datamover checkpoint prefix, or align expiration with your actual backup retention policy so you never expire a checkpoint that a stored backup still depends on. + +### Credential or authentication errors + +These show up as errors in the controller log mentioning the object storage provider (AWS, Azure, or GCP), typically at the start of a DataUpload or DataDownload, before any actual data movement happens. Since the controller reads the same BackupStorageLocation and credentials Velero uses, the first thing to check is whether ordinary non-VM Velero backups to the same BackupStorageLocation work. If they do not, the problem is in your BSL/credential configuration generally, not specific to KubeVirt DataMover. + +If ordinary Velero backups work fine but VM backups with KubeVirt DataMover fail on credentials, check that the datamover controller pod actually has the environment variables or projected token it expects. For AWS STS setups and Azure Workload Identity setups, the operator propagates the same environment variables it configures for Velero itself, so compare the Velero deployment's environment against the datamover controller deployment's environment if you suspect a mismatch. + +### DPA validation error when enabling the plugin + +If applying your DPA fails with a message like "only a single instance of KubeVirt DataMover Controller can be installed across the entire cluster," another DPA on the cluster already has `kubevirt-datamover` enabled and its controller deployment already exists. KubeVirt DataMover's controller is a cluster-scoped singleton, not a per-namespace deployment, so only one DPA across the whole cluster can have it enabled at a time. Check for other DPAs: + +```bash +oc get dpa -A +``` + +### Warning about missing kubevirt plugin + +If you see a warning that VM restore requires the `kubevirt` plugin, add `kubevirt` alongside `kubevirt-datamover` in `spec.configuration.velero.defaultPlugins`. `kubevirt-datamover` handles disk data movement, but VM metadata and file-level restore actions are handled by the separate kubevirt-velero-plugin (the `kubevirt` plugin). Both need to be present for a complete VM backup and restore workflow. + +### "Failed freezing guest filesystem" warning during backup + +If the VirtualMachineBackup status includes a warning like `Failed freezing guest filesystem: ... QEMU guest agent is not connected`, and your VM does not have the QEMU guest agent installed and running, this is expected and not a failure. KubeVirt attempts to quiesce (freeze) the guest filesystem for a cleaner backup, but without a guest agent it can't, so it falls back to a crash-consistent backup instead, the same way a hard power-cycle would leave the disk. The backup still completes successfully. If you want quiesced, application-consistent backups, install `qemu-guest-agent` in the guest OS. There is currently no way to require quiescing and fail the backup instead of falling back, that behavior is still under development upstream. + +## Known limitations + +- **VM must be running**: KubeVirt DataMover backs up VMs through CBT, which requires the VM to be running (`spec.running: true`, `status.printableStatus: Running`) at backup time. Offline (stopped) VM backup through this path is not supported. +- **Single active backup per VM**: only one DataUpload can be active for a given VM at a time. Concurrent backups of the same VM are not supported. +- **Block volume mode required**: CBT-based backup only works with `volumeMode: Block` PVCs. Filesystem-mode disks fall back to whatever your volume policy routes them to (typically a CSI snapshot or File System Backup), not KubeVirt DataMover. +- **EFI/SMM Fedora and RHEL VMs may back up zero data without an obvious error**: see "VM backup reports success but restored disk has no data" above. +- **Object storage required**: KubeVirt DataMover always requires `snapshotMoveData: true` and a working BackupStorageLocation. There is no in-cluster-snapshot-only mode. +- **Cluster-wide singleton controller**: only one DPA per cluster can have KubeVirt DataMover enabled. +- **VirtualMachineBackup is transient, VirtualMachineBackupTracker is not**: the controller deletes each `VirtualMachineBackup` after archiving its state to object storage, so if you are scripting around it directly, expect it to disappear once a backup completes; treat the archived JSON in object storage as the durable record, not the live cluster object. The `VirtualMachineBackupTracker` is different: the controller deliberately leaves it on the cluster between backups so KubeVirt can use it to redefine the VM's checkpoint across restarts and live migrations, so seeing it stick around in the VM's namespace long after a backup finishes is expected, not a leak. +- **Restore chains rebuild incrementally**: a restore from a VM with a long incremental chain replays each checkpoint in sequence via `qemu-img rebase`, rather than restoring straight from the full backup. A very long incremental chain can make individual restores slower than you might expect, even though it keeps backups themselves fast. This is a reasonable trade to be aware of when deciding on your `maxIncrementalBackups` setting. +- **No user-triggered full backup**: there is currently no supported way to force a one-off full backup from the Backup or VirtualMachine object. The controller falls back to a full backup automatically when it can't validate the existing checkpoint chain or when `maxIncrementalBackups` is reached. +- **Log tail is capped at 200 lines**: when the controller captures a datamover or downloader pod's logs, it only keeps the last 200 lines. For most failures this is enough, but if you need earlier output from a long-running transfer, you will need to catch the pod while it is still alive with `oc logs`. If the pod failed rather than being canceled or completing normally, the controller leaves it in place rather than deleting it, so `oc logs` and `oc describe pod` still work against it after the fact. +- **Cancellation cleanup is best effort**: canceling a DataUpload or DataDownload tells the controller to clean up the pod and any temporary PVCs it created, but if that cleanup itself fails, the operation still moves to `Canceled` rather than getting stuck, and the cleanup error is only logged, not retried automatically. The datamover/downloader pod is owned by the DataUpload/DataDownload, so Kubernetes garbage collection will remove it once the parent object itself is deleted, even if the controller's own cleanup missed it. The temporary backup PVC does not get the same treatment: it lives in the VM's namespace while the DataUpload/DataDownload lives in `openshift-adp`, and Kubernetes does not allow owner references across namespaces, so that PVC is tracked and cleaned up only by the controller's own reconcile logic, not by garbage collection. If a cleanup failure leaves one behind, it can persist indefinitely even after the parent object is deleted, so it's worth checking for orphaned PVCs in the VM's namespace after canceling an operation, especially if you see something odd there afterward. + +If you run into an issue that is not covered here, the most useful thing to collect before opening a bug report is the full controller log around the time of the failure, plus `oc describe` output for the affected DataUpload or DataDownload and the corresponding Velero Backup or Restore object.