The VLAB

Purpose of the VLAB

The Virtual Lab (VLAB) is designed to allow a team of people to share remote access to individual FPGA development boards. The VLAB provides user access control, mutual exclusion, load balancing, statistics, and logging.

The VLAB was developed to support two main use cases:

• To allow a development team to share access to a small number of expensive development boards. Also this has the benefit that these boards can be safely in a server room instead of on someone's desk.
• To support an external access teaching framework, allowing boards to be served at a centralised location but accessed securely by many external users.

Overall organisation

The VLAB is structured as follows:

The main entry point for clients is the relay server. The client software connects to the relay (using ssh) and requests access to an FPGA board of a given type. The relay authenticates the user, and if they are allowed to access the board type that they have requested, a free FPGA board of the requested type is selected and the user is forwarded on to the boardserver responsible for serving it.

The VLAB makes use of Docker and consists of three Docker images:

• Relay - the main entrypoint to the VLAB.
• Boardserver - a boardserver container is created for each hosted FPGA, and is responsible for actually serving the FPGA.
• Web - provides statistics tracking in a handy web interface.

These containers can be hosted on any physical server, and may be all the same server in small deployments.

relay and web are always running. boardserver instances are started by udev rules which detect when a supported FPGA device is connected, read its serial number, and use this to launch an instance of the boardserver container to serve the FPGA. For the purpose of this documentation, a physical server which is configured to look for FPGAs and launch instances of the boardserver container are called board hosts.

Client Installation and Use

The client script (vlab.py) can be downloaded directly from GitHub. It has no external dependencies, apart from Python 3 itself.

To use the client you must have a keyfile which should have been sent to the client by the VLAB administrator. Connect to the VLAB using:

./vlab.py -k keyfile -b requested_board_class

If the VLAB relay server or port are not the defaults defined in the script they can be specified with the -r and -p options respectively. Also if you must use a different username to the currently logged in user you can use -u.

./vlab.py -r a_servername -p 2223 -u vlabuser -k keyfile -b requested_board_class

These commands will request a free board of the board class requested_board_class and, if one is free, connect to it. Local port 12345 will be connected to the remote hardware server for that board, so the Xilinx tools can be pointed at tcp:localhost:12345 to program and debug the FPGA, whilst the terminal will show the serial UART input and output.

The terminal uses GNU screen so to disconnect press and release Ctrl-A, then press Ctrl-K.

By default, the relay server will allocate boards using a least-recently-unlocked scheme to balance load, i.e. whichever board has the oldest unlock time will be preferred. Boards that have been attached to the system but never allocated to a user will be used preferentially over those that have previously been used.

Installation

Relay Server Installation

The relay server is tested on Ubuntu Linux, but should work on similar operating systems. Docker makes installation simple. Ensure that Docker is installed and clone the VLAB repository:

git clone https://github.com/RTSYork/VLAB

First, you need to edit vlab.conf. This file describes the FPGA boards in your lab and the users who can access them. This is described in more detail here.

You then need to set up ssh keypairs for use inside the VLAB. To do this run:

./manage.py generatekeys --internal

This will construct a keypair which is used by the relay server to communicate with the boardservers. This will be stored in /keys/ and should be protected appropriately. When you edited vlab.conf you should have added some users to the system. These users also need keypairs which can be generated by running:

./manage.py generatekeys --allnew

This command creates a keypair for all users that are mentioned in vlab.conf and does not already have a key pair in the /keys/ directory. These are stored as /keys/$username and /keys/$username.pub for the private and public keys respectively. The user needs the private key in order to use the VLAB. Should you need to recreate these, you can simply delete the pair and rerun generatekeys --allnew.

Before building the Docker containers, the Xilinx Hardware Server archive must be downloaded so it can be installed into the board server container. Download the Vivado 64-bit Hardware Server for Linux for the edition of the Xilinx tools you are using (use the latest relevant version if using the VLAB with multiple Xilinx installs) and place the Xilinx_HW_Server_Lin_xxxx.x_xxxx_xxxx.tar.gz file in the boardserver folder within the VLAB repository.

Once configuration and keys are set up and the hardware server archive is in place, build the VLAB containers with:

./manage.py build

This instructs Docker to build the images for the relay server, web server, and board servers. Finally to start the service use:

./manage.py start

The images will need to be rebuilt if changes are make to the internal keypair, but changes to the configuration or user keypairs only require the containers to be restarted. The board server image will need rebuilding and re-loading onto the board hosts if you need to update the Xilinx Hardware Server version.

You likely want to configure the VLAB containers to start automatically on boot. There are many way to do this covered in the Docker documentation. For example, on a systemd-based system, create the file /etc/systemd/system/vlab.service:

[Unit]
Description=VLAB
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker-compose -f /opt/VLAB/docker-compose.yml up --force-recreate
ExecStop=/usr/bin/docker-compose -f /opt/VLAB/docker-compose.yml stop

[Install]
WantedBy=default.target

The service can be started and stopped with systemctl start vlab.service and systemctl stop vlab.service, or made to restart every boot with systemctl enable vlab.service.

Clients will ssh to the relay container hosted here, so its port (2222 by default) should be externally-visible. If you need to use another port then you can edit the port mapping for the relay service in docker-compose.yml. Change the line:

	- "2222:22"

This line says that port 22 in the container should be mapped to port 2222 on the host machine. If another port is required it can be changed. Remember that users will have to use the -p option to specify the non-default port:

./vlab.py -r relayserver -p 2223 -k mykey.vlabkey -b zybo

Board Host Installation

Prerequisites

As well as Docker, the board host server depends on Python 3 and the 'redis' package, and certain FPGA configurations require the fxload firmware downloader and libusb. Task Spooler (tsp) is also required for queueing requests to register and deregister boards with the relay server as they are connected and disconnected.

From a standard Ubuntu Server 18.04 installation, these can be installed with:

sudo apt install fxload libusb-dev python3-redis task-spooler

Installation

As described previously, board hosts are the servers to which the actual FPGA boards are connected. There can be multiple board hosts, and can be the same machine that the relay server is running. udev rules recognise when FPGAs are attached and instances of the boardserver container are launched to serve it.

To set up a board host, first create a vlab user on the which has access to the docker group. For example, on the new board host:

sudo adduser vlab
sudo usermod -a -G docker vlab
sudo mkdir /home/vlab/.ssh
sudo chown vlab:vlab /home/vlab/.ssh
sudo chmod 700 /home/vlab/.ssh

And copy the internal VLAB public key for that user. From the relay server where the keys were generated:

scp boardserver/authorized_keys vlab@newboardserver:.ssh/

If you wish to use Digilent's boards on your board host you need to install the Digilent tools. Go to the Digilent website and download the Runtime and Utilities. These can be installed using standard methods from DEB or RPM packages (recommended), or from the .zip version as follows:

unzip digilent.adept.runtime_*.zip
unzip digilent.adept.utilities_*.zip
sudo mkdir -p /etc/hotplug/usb/
cd digilent.adept.runtime_*/
sudo ./install.sh silent=1
cd ../digilent.adept.utilities_*/
yes "" | sudo ./install.sh 

This should add a utility called dadutil to your $PATH which can be used to enumerate Digilent boards connected to the board host.

The VLAB scripts also require the Xilinx Software Command-line Tool (XSCT) installed on the board host, which is mapped into the board server container and used to reset FPGA boards on connection/disconnection. This can be installed using the Xilinx Software Development Kit Standalone WebInstall Client, which is freely downloadable but requires registering a Xilinx login.

To install these tools from the command line (to /tools/Xilinx), run the following:

sudo mkdir /tools
sudo chown $USER:$USER /tools
chmod +x Xilinx_SDK_2019.1_0524_1430_Lin64.bin
./Xilinx_SDK_2019.1_0524_1430_Lin64.bin -- -b AuthTokenGen
./Xilinx_SDK_2019.1_0524_1430_Lin64.bin -- -a XilinxEULA,3rdPartyEULA,WebTalkTerms -b Install -e "Xilinx\ Software\ Command-Line\ Tool\ \(XSCT\)"

Then clone/download the host directory from this repository to the new board host, and run the install.sh script with the path to the installed Xilinx SDK tools. For example:

cd host
sudo ./install.sh /tools/Xilinx/SDK/2019.1

Once installed, edit /opt/VLAB/boardhost.conf to set the hostname/IP and port of the relay server.

Finally, send the board server Docker image from where you built it to the new board host using the helper script. From the machine where the Docker image was built, run the following (where newboardhost is the hostname or IP of the new board host):

./boardserver-update.sh newboardhost

This script can also be used to update the board servers when the image is changed/updated, and can take multiple board hosts (including localhost) as arguments.

Alternatively this step can be performed manually by saving, transferring and loading the Docker image, as follows:

docker save -o boardserver.tar vlab/boardserver
scp boardserver.tar newboardhost:
ssh newboardhost docker load -i boardserver.tar

Supporting FPGA Boards on the Board Hosts

When a new FPGA board is connected to a board host the following takes place:

1. udev recognises a device connection event for both the TTY and JTAG, and starts executing its rules (in /etc/udev/rules.d/).
2. The appropriate rule executes /opt/VLAB/boardevent.sh attach and passes it the serial number. The script is executed asynchronously through the at now command, in order to break out of udev's network sandbox.
3. The boardevent.sh attach script schedules the execution of boardattach.sh using Task Spooler, to ensure an orderly FIFO queue of attach events.
4. When run from the spool queue, boardattach.sh checks to see if the board's Docker container has already been started by other means, and if not runs boardattach.py.
5. boardattach.py sends the serial number to the relay server, which looks it up to see if it is a device detailed in vlab.conf, and if so registers it.
   • If not, the unknown device is rejected and logged.
   • If it is, the relay server informs boardconnected.py which type of device it is, and so which container should be launched to host it. Currently only one type of container is required, but more diverse boards may require other containers in the future.

Upon board disconnection (according to udev), the board host will kill the associated board server container and deregister it from the relay server.

Each supported FPGA must have a unique serial number, and that serial number must be readable by udev (and must match for both JTAG and TTY devices) It is sometimes the case that all FPGA development boards of the same type come from the factory with the same serial number, so a utility program may be required to set a unique serial number first. Instructions to program JTAG and UART serials on certain boards can be found at https://wiki.york.ac.uk/display/RTS/Setting+Up+FPGA+Boards.

The board host also runs a cronjob each minute to try to register any board that are attached but don't have board server containers running for them. Each board server container will also periodically attempt to re-register with the relay server each minute using a cronjob. The relay server will attempt to SSH to each of its registered board servers each minute, and remove any that are unreachable.

Configuration

The VLAB is configured by vlab.conf, which describes two things:

1. The FPGAs in the system
   • FPGAs are identified by their serial number. udev rules on the host machine detect when a device is connected, read its serial number, and use this to launch an instance of the boardserver container to serve the FPGA.
   • If an unknown FPGA serial number is detected then no boardserver will be created. The logfile log/attachdetach.log will contain a line detailing the unknown serial number.
   • Each FPGA also has an assigned board class. Users request board classes, rather than specific boards. This allows for FPGAs to be grouped together and users load balanced across them.
2. The users in the system
   • Users have an overlord flag, which if set means that they can access all boards on the VLAB.
   • Otherwise, users are given a list of the board classes which they are allowed to access. Users may only connect to board classes they are allowed to.

vlab.conf is volume mapped into the relay container. After editing vlab.conf the relay container must be restarted with:

./manage.py start

vlab.conf is in JSON format. An example is shown below:

{
	"users": {
		"example_user": {"allowedboards": ["boardclass_a", "boardclass_b"]},
		"overlord_user": {"overlord": true}
	},

	"boards": {
		"exampleboardserialnumber": {"class": "boardclass_a", "type": "standard"}
	}
}

In this example there is one board with a serial number exampleboardserialnumber assigned to the boardclass boardclass_a. "type" in the board definition is a string used to tell the VLAB which drivers are required to interact with the board. Currently "type" is not used because all supported boards can be served from the same container.

Resetting boards on disconnect

When a user disconnects from an FPGA their design will remain active. The VLAB also supports shutting down a hosted FPGA when the user disconnects. This can be useful if, for example, the board is connected to an Ethernet network and so it is not desirable to have designs active when they are not being tested.

Resetting the boards requires that the boardserver containers have access to the Xilinx command line tools, by installing them on each board host (see above).

Once installed, create a symlink on the board host called xsct in /opt/VLAB/ which points to the Xilinx SDK install folder. The board host install script will create this symlink automatically if the path to the Xilinx tools is is specified as an argument.

To create this symlink this manually, run the following (change if your install paths or version are different):

sudo ln -s /tools/Xilinx/SDK/2019.1 /opt/VLAB/xsct 

Then add "reset: "true" to the board definition in vlab.conf. For example

"boards": {
	"210279777433": {"class": "vlab_zybo", "type": "zybo", "reset": "true"}
}

Now when a user connects or disconnects from the defined board, a full system reset will be issued, and in the case of Zynq-based boards the ARM cores shut down.

Common Issues

Asking for root password when relay connects to the board server

This happens when the relay cannot read the keys/id_rsa key, which is used for authentication between the relay and boardserver containers. Ensure that the keys in /keys/ are readable by the relay Docker container.

Asking for vlab password when relay connects to the board host

This can happen when the relay cannot read the keys/id_rsa key, which is used for authentication. Ensure that the keys in /keys/ are readable by the relay Docker container. Also check that the correct public key file has been installed with appropriate permissions to /home/vlab/.ssh/authorized_keys on the board host.

Permission denied: '/vlab/log/access.log'

This error can happen when trying to access the VLAB or run any command that writes to a log from the relay. Ensure that the log directory inside the VLAB folder (or just the log files, if they are already created) is writable from the Docker container.