SmartNIC: Install drivers and Access the SmartNIC

Install Drivers and Access the SmartNIC

What is SmartNIC?

• To keep up with the increasing need for data processing power, SmartNICs are network interface cards that are equipped with an on-board ARM processor.
• Originially, this ARM processor was meant for offloading some parts of data processing to the NIC and thus, save precious cycles on the host CPU.
• However, we use this on-board ARM-based processor with popcorn linux to offload any processes that are compute intensive or for research purposes.
• SmartNICs are categorized into three main types of devices:
  • ASIC-based (easy to program but flexibility is limited to pre-defined capabilities)
  • FPGA-based (difficult to program but supports workload specific optimizations)
  • SoC-based (Easiest programmability and highest flexibility)

Install Drivers on Host

These instructions assume that Ubuntu 20.04 is installed on the host. If you are using some other version of linux, you should install the corresponding drivers from the mellanox website.

Downloads drivers

# cd /opt
# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-5.3-1.0.0.1/MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64.tgz

Install drivers

# tar -xzvf MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64.tgz# cd MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64
# ./mlnxofedinstall --auto-add-kernel-support --without-fw-update

• During the installation, you will be asked a few questions. Just hit 'y' for all of them.

• This process will take some time to complete.

• It is safe to use the ./mlnxofedinstall script again if the installation fails initially. This script will make sure that any previous version of drivers are removed before starting the installation process.

• The script tells us what ethernet interfaces it found and will apply the newly installed drivers. We can see that besides the smartNIC (non-capitalized ‘s’), i.e., ConnectX-5, our SmartNIC (a.k.a. Bluefield) is also found.

• It will also print out all the Mellanox ethernet interfaces that are affected after the installation.

Restart MLNX_OFED's openibd:

# /etc/init.d/openibd restart
Unloading HCA drvier:                [ OK ]
Loading HCA driver and Access Layer: [ OK ]

Install rshim

• rshim is a shim layer that can be used to from the host system to communicate with the SmartNIC.
• It creates dummy interfaces with local IP addresses on both the host and the SmartNIC.
• That way, we can easily SSH into the SmartNIC from the host (we will get back to this later)
• By default, the above install script will also install the rshim drivers. You can confirm this by checking whether you have an rshim0 device in /dev/. If there is one, you are fine.
• In case you do not have, install it separately from the DEBS directory of the drivers we downloaded above.

# cd MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64/DEBS
# dpkg --force-all -i rshim-<version>.deb

Start rshim service

# systemctl enable rshim
# systemctl start rshim

Check status of rshim.service

# systemctl status rshim
● rshim.service - rshim driver for BlueField SoC
     Loaded: loaded (/lib/systemd/system/rshim.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-11-17 00:59:42 EST; 1 weeks 0 days ago
       Docs: man:rshim(8)
   Main PID: 1685 (rshim)
      Tasks: 6 (limit: 18698)
     Memory: 600.0K
     CGroup: /system.slice/rshim.service
             └─1685 /usr/sbin/rshim

Nov 17 00:59:41 BluefieldServer systemd[1]: Starting rshim driver for BlueField SoC...
Nov 17 00:59:42 BluefieldServer systemd[1]: Started rshim driver for BlueField SoC.
Nov 17 00:59:42 BluefieldServer rshim[1685]: Probing pcie-0000:18:00.2
Nov 17 00:59:42 BluefieldServer rshim[1685]: create rshim pcie-0000:18:00.2
Nov 17 00:59:43 BluefieldServer rshim[1685]: rshim0 attached

Sanity Check

Get the device name

# lspci | grep i bluefield

Network configuration

• Install bridge-utils package

# apt-get install bridge-utils

• Write the network configuration

# vi /etc/netplan/01-netcfg.yaml
network:
    version: 2
    renderer: networkd
    ethernets:
        enp1s0f0:
            dhcp4: yes
        tmfifo_net0:
            dhcp4: no
            addresses: [10.4.4.100/24]
    bridges:
        br0:
            dhcp4: no
            addresses: [10.4.4.10/24]
            gateway4: 10.4.4.1
            nameservers:
                addresses:
                - 8.8.8.8
            interfaces:
            - enp1s0f0

• Apply network configuration

# netplan apply

Check the network configuration

• Make sure the output of ifconfig command reflects 01-netcfg.yaml's network configuration.

# ifconfig

Install Operating System on SmartNIC

• This is a good time to start with the OS installation process on the SmartNIC.
• If OS is already installed on the SmartNIC, feel free to skip this installation step.

Hardware and Software Requirements

• Host system with RShim driver installed.
• BlueField NIC inserted in a host PCIe slot.
• USB cable connecting the NIC card and the host.
• Note: From now on, all the commands would be run on the SmartNIC, until mentioned otherwise.

Ubuntu 20.04 installation with MLNX_OFED and DOCA

• A pre-built BFB of Ubuntu 20.04 with MLNX_OFED_LINUX and DOCA packages installed is available on the Nvidia Doca SDK developer zone page.
• To install Ubuntu, run on the hostL:

$ cat <ubuntu.bfb> > /dev/rshim0/boot
  
Username: ubuntu
Password: ubuntu

• We will add a new user later, but for now, we need ubuntu as a user on this sytem.

Disable clout-init network configuration

• Write network: {config: disabled} to the 99-disable-network-config.cfg file.

# vi /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}

Add a new user

• Create a user with the same name as that of host OS on the SmartNIC.
• Add this user to the sudoers list.
• Follow instructions on this page.
• Note: The username on the SmartNIC's OS should be the same as the host's username. This is necessary because Popcorn only runs for the same user on all the available nodes.

Network configuration

• 00-nm.yaml

# vi /etc/netplan/00-nm.yaml
network:
    version: 2
    renderer: NetworkManager

• 50-cloud-init.yaml

# vi /etc/netplan/50-clout-init.yaml
network:
    ethernets:
        tmfifo_net0:
            addresses:
            - 10.4.4.101/24
            dhcp4: false
            nameservers:
                addresses:
                - 10.4.4.100
            routes:
            - metric: 1025
              to: 0.0.0.0/0
              via: 10.4.4.100
    renderer: networkd
    version: 2

• Note: Make sure that the network configuration is correct. If you want a custom configuration, make sure that the addresses are witihin the same subnet. Preferrably, these addresses should be in the same subnet as your main network.

Access the SmartNIC via Host

• From now on, we will run the commonds on the host.

Apply the network configuration

# netplan apply

• If there is an error, due to typos or whatever, use the command in debug mode to find out more details:

# netplan --debug apply

Enable IP forwarding on the host

# echo 1 | tee /proc/sys/net/ipv4/ip_forward

Enable NAT on the internet-facing interface on the host

# iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE

• Note: Replace enp1s0f0 with your main (internet-facing) interface.

IP forwarding

# net.ipv4.conf.eno1.forwarding = 1

• If this is indeed enabled, but still no Internet access, then it might have happened that your default firewall rules are configured to an implicit deny policy. Let’s change that:

# iptables -A FORWARD -o enp1s0f0 -j ACCEPT
# iptables -A FORWARD -m state --state ESTABLISHED,RELATED -i enp1s0f0 -j ACCEPT

Set Google's DNS on the BlueField

# echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Full internet access is enabled

# ping google.com

Test the network

• Try pinging the SmartNIC

# ping -I tmfifo_net0 192.168.100.2 -c2

• You should be able to ping the SmartNIC at this point.

Connect to the SmartNIC via Host

# ssh <username>@10.4.4.101

Congratulations! You have successfully connected to the SmartNIC!

Tips

• It is recommended to keep address to interfaces enp1s0f0 (or whatever your main networking interface is called) and tmfifo_net0 interface within the same subnet both on the host and on the SmartNIC. That way, you don't have to worry about constantly changing the network configuration in the future.
• If you are playing around with the networking configuration, you might not be able to connect via ssh. In that case, use a program like minicom or screen to access the SmartNIC via /dev/rshim0/console file.