SmartNIC: Install popcorn kernel on the SmartNIC setup

Install and run popcorn-linux on the SmartNIC setup

Overview

This guide describes how to set up a testing environment for Popcorn Linux using an x86 host equipped with a SmartNIC. Basically, we will run popcorn on 2 machines - one on x86 host and one for ARM SmartNIC, where the host and the SmartNIC are connected over PCI-e connection. The examples and commands are based on Ubuntu 20.04.

--------------   --------------
|   Node 0   |   |   Node 1   |
|    Host    |   |  SmartNIC  |
|    x86     |   |    arm     |
| 10.4.4.100 |   | 10.4.4.101 |
--------------   --------------
-------------------------------
|  linux-x86      linux-arm   |
|                             |
|           host (x86)        |
|           10.4.4.10         |
-------------------------------

Setup the Host network interface

Backup default netplan config

# cp /etc/netplan/01-netcfg.yaml ~/default_nw_cfg.yaml

Install bridge-utils package

# apt-get install bridge-utils

Write the network configuration Replace enp1s0f0 with your main network interface

# vi /etc/netplan/01-netcfg.yaml
network:
    version: 2
    renderer: networkd
    ethernets:
        enp1s0f0:
            dhcp4: yes
        tmfifo_net0:
            dhcp4: no
            addresses: [10.4.4.100/24]
    bridges:
        br0:
            dhcp4: no
            addresses: [10.4.4.10/24]
            gateway4: 10.4.4.1
            nameservers:
                addresses:
                - 8.8.8.8
            interfaces:
            - enp1s0f0

Apply netplan configuration

# netplan apply

Check the network configuration

# ifconfig

The output should reflect the network configuration that we have written.

Note: In this example we have assigned IPs for a specific subnet. Please use IPs that belong in the same subnet as your network.

Setup the SmartNIC network interface

Disable cloud-init network configuration

Write network: {config: disabled} to the 99-disable-network-config.cfg file.

# vi /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}

Write the network configuration for 00-nm.yaml

# vi /etc/netplan/00-nm.yaml
network:
    version: 2
    renderer: NetworkManager

Write the network configuration for 50-cloud-init.yaml

# vi /etc/netplan/50-clout-init.yaml
network:
    ethernets:
        tmfifo_net0:
            addresses:
            - 10.4.4.101/24
            dhcp4: false
            nameservers:
                addresses:
                - 10.4.4.100
            routes:
            - metric: 1025
              to: 0.0.0.0/0
              via: 10.4.4.100
    renderer: networkd
    version: 2

Apply netplan configuration

# netplan apply

Check the network configuration

# ifconfig

The output should reflect the network configuration that we have written.

Note: In this example we have assigned IPs for a specific subnet. Please use IPs that belong in the same subnet as your network.

Prepare for the setup on Host

Let's start with updating apt repository and install some utilities:

$ sudo apt-get update
$ sudo apt-get install build-essential libssl-dev libncursesw5-dev git curl bc

Retrieve the popcorn kernel source code

$ git clone --depth=1 -b main --single-branch https://github.com/ssrg-vt/popcorn-kernel.git
$ cd popcorn-kernel

Kernel config

$ cat /boot/config-$(uname -r) > .config

Note: You can use your own config but make sure the following options are disabled: - CONFIG_SWAP - CONFIG_TRANSPARENT_HUGEPAGE - CONFIG_CMA - CONFIG_MIGRATION - CONFIG_COMPACTION - CONFIG_KSM - CONFIG_MEM_SOFT_DIRTY Page should be 4 KB. ARM should select ARM64_4K_PAGES PPC should use CONFIG_PPC_4K_PAGES.

Add popcorn-specific config options to the end of your .config file

#
# Popcorn Distributed Execution Support
#
CONFIG_ARCH_SUPPORTS_POPCORN=y
CONFIG_POPCORN=y
CONFIG_POPCORN_DEBUG=y
CONFIG_POPCORN_DEBUG_PROCESS_SERVER=y
# CONFIG_POPCORN_DEBUG_PAGE_SERVER is not set
# CONFIG_POPCORN_DEBUG_VMA_SERVER is not set
# CONFIG_POPCORN_DEBUG_VERBOSE is not set
CONFIG_POPCORN_CHECK_SANITY=y
# CONFIG_POPCORN_REMOTE_INFO is not set
CONFIG_POPCORN_STAT=y
# CONFIG_POPCORN_STAT_PGFAULTS is not set
CONFIG_POPCORN_KMSG=y
CONFIG_POPCORN_KMSG_SOCKET=m
CONFIG_POPCORN_KMSG_RDMA=m
CONFIG_POPCORN_KMSG_TEST=m
CONFIG_POPCORN_DEBUG_MSG_LAYER=y

Update the .config file

$./update_config.sh

Build the kernel

$ make -j 8
$ sudo make modules_install
$ sudo make install

Reboot with popcorn kernel and check using

uname -r

Access the SmartNIC via Host

From now on, we will run the commonds on the host.

Apply the network configuration

# netplan apply

If there is an error, due to typos or whatever, use the command in debug mode to find out more details:

# netplan --debug apply

Enable IP forwarding on the host

# echo 1 | tee /proc/sys/net/ipv4/ip_forward

Enable NAT on the internet-facing interface on the host

# iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE

Note: Replace enp1s0f0 with your main (internet-facing) interface.

IP forwarding

# net.ipv4.conf.eno1.forwarding = 1

If this is indeed enabled, but still no Internet access, then it might have happened that your default firewall rules are configured to an implicit deny policy. Let’s change that:

# iptables -A FORWARD -o enp1s0f0 -j ACCEPT
# iptables -A FORWARD -m state --state ESTABLISHED,RELATED -i enp1s0f0 -j ACCEPT

Set Google's DNS on the BlueField

# echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Full internet access is enabled

# ping google.com

Test the network

Try pinging the SmartNIC

# ping -I tmfifo_net0 192.168.100.2 -c2

You should be able to ping the SmartNIC at this point.

Connect to the SmartNIC via Host

# ssh <username>@10.4.4.101

Prepare for the setup on SmartNIC

Make sure you are ssh-ed into the SmartNIC before you proceed.
Let's start with updating apt repository and install some utilities:

$ sudo apt-get update
$ sudo apt-get install build-essential libssl-dev libncursesw5-dev git curl bc

Retrieve the popcorn kernel source code

$ git clone --depth=1 -b main --single-branch https://github.com/ssrg-vt/popcorn-kernel.git
$ cd popcorn-kernel

Kernel config

$ cat /boot/config-$(uname -r) > .config