Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add supports for vDPA interface #98

Merged
merged 1 commit into from
Jan 10, 2024
Merged

Conversation

makoto126
Copy link
Contributor

Tested with SR-IOV Network Device Plugin and modified kube-ovn(ovs-dpdk offload to our DPU).
Live migration previously tested in Kubevirt(with qemu).

VM spec

apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    interfaces:
      - name: pod
      - name: vdpa1
        vdpa: { numQueues: 9, iommu: False }
      - name: vdpa2
        vdpa: { numQueues: 9, iommu: False }
  networks:
    - name: pod
      pod: {}
    - name: vdpa1
      multus:
        networkName: offload-ovn1
    - name: vdpa2
      multus:
        networkName: offload-ovn2

SR-IOV device plugin config

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "jaguar",
                "resourcePrefix": "jaguarmicro.com",
                "selectors": {
                    "vendors": ["1f53"],
                    "devices": ["1000"],
                    "drivers": ["jaguar"],
                    "vdpaType": "vhost"
                }
            }
        ]
    }

NetworkAttachmentDefinition Spec

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: offload-ovn1
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: jaguarmicro.com/jaguar
spec:
  config: >-
    {
        "cniVersion": "0.3.0",
        "type": "kube-ovn",
        "server_socket": "self.dpu.com:12345",
        "provider": "offload-ovn1.default.ovn"
    }
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: offload-ovn2
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/resourceName: jaguarmicro.com/jaguar
spec:
  config: >-
    {
        "cniVersion": "0.3.0",
        "type": "kube-ovn",
        "server_socket": "self.dpu.com:12345",
        "provider": "offload-ovn2.default.ovn"
    }

Pod network status

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "cbr0",
          "interface": "eth0",
          "ips": [
              "10.244.2.45"
          ],
          "mac": "5a:61:29:39:0d:29",
          "default": true,
          "dns": {},
          "gateway": [
              "10.244.2.1"
          ]
      },{
          "name": "default/offload-ovn1",
          "interface": "net1",
          "ips": [
              "10.1.0.20"
          ],
          "mac": "00:00:00:03:6C:84",
          "dns": {},
          "device-info": {
              "type": "vdpa",
              "version": "1.1.0",
              "vdpa": {
                  "parent-device": "test1",
                  "driver": "vhost",
                  "path": "/dev/vhost-vdpa-2",
                  "pci-address": "0000:68:00.5"
              }
          }
      },{
          "name": "default/offload-ovn2",
          "interface": "net2",
          "ips": [
              "10.2.0.2"
          ],
          "mac": "00:00:00:50:A4:81",
          "dns": {},
          "device-info": {
              "type": "vdpa",
              "version": "1.1.0",
              "vdpa": {
                  "parent-device": "test2",
                  "driver": "vhost",
                  "path": "/dev/vhost-vdpa-3",
                  "pci-address": "0000:68:00.6"
              }
          }
      }]

VM after dhclient

ubuntu@ubuntu-container-rootfs:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5a:61:29:39:0d:29 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    inet 10.244.2.45/24 metric 100 brd 10.244.2.255 scope global ens4
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:00:00:03:6c:84 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 10.1.0.20/24 brd 10.1.0.255 scope global dynamic ens6
       valid_lft 2305sec preferred_lft 2305sec
4: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:00:00:50:a4:81 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
    inet 10.2.0.2/24 brd 10.2.0.255 scope global dynamic ens7
       valid_lft 2305sec preferred_lft 2305sec

@CLAassistant
Copy link

CLAassistant commented Jan 9, 2024

CLA assistant check
All committers have signed the CLA.

Copy link
Contributor

@fengye87 fengye87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. Please also doc this new feature in this file

@@ -115,6 +116,11 @@ type InterfaceMasquerade struct {
type InterfaceSRIOV struct {
}

type InterfaceVDPA struct {
NumQueues int `json:"numQueues,omitempty"`
Iommu bool `json:"iommu,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better name it IOMMU to keep it in Golang style

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The VdpaConfig struct and other structs in pkg/cloudhypervisor/client.go used Iommu too.

type VdpaConfig struct {
	Id         string `json:"id,omitempty"`
	Iommu      bool   `json:"iommu,omitempty"`
	NumQueues  int    `json:"num_queues"`
	Path       string `json:"path"`
	PciSegment int16  `json:"pci_segment,omitempty"`
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's generated. For non-generated code, better keep them in style

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fengye87 I updated the doc and IOMMU.

Copy link

codecov bot commented Jan 10, 2024

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (30780fe) 34.11% compared to head (0f3d995) 34.08%.

❗ Current head 0f3d995 differs from pull request most recent head a49cb4e. Consider uploading reports for the commit a49cb4e to get more accurate results

Files Patch % Lines
pkg/controller/vm_webhook.go 50.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #98      +/-   ##
==========================================
- Coverage   34.11%   34.08%   -0.04%     
==========================================
  Files           6        6              
  Lines        1952     1957       +5     
==========================================
+ Hits          666      667       +1     
- Misses       1182     1185       +3     
- Partials      104      105       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fengye87 fengye87 merged commit db3f654 into smartxworks:main Jan 10, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants