README.md

OrkhestraFPGAStream - Accelerated database operations on an FPGA with a dynamic stream processing interface

OrkhestraFPGAStream is a middleware for orchestrating a dynamic stream processing pipeline. Currently, it only supports DBMS data streams with DBMStoDSPI, which uses FPGA modules with our Dynamic Stream Processing Interface (DSPI). This software stack is meant to be coupled with a SQL parser or a DBMS. Then the software stack would execute the given query plan with an available FPGA using the available accelerator module library. How the data flows through the FPGA stack is shown in the graph below:

As the image shows, the input would be a query plan graph from a DBMS to the software stack, which will use FOS and the modules described below to accelerate the query. The image is slightly out of date, but it gives a rough overview.

Currently supported data types:

• 32 bit integer
• Fixed length string
• SQL Date
• SQL Decimal(15,2)
• NULL

Currently supported modules:

• Filter
• Join
• Linear Sort
• Merge Sort
• Addition
• Multiplication
• Global Aggregation Sum
• DMA

The DMA is a compulsory module which will set up the interface. The sort modules are meant to be used together. The join module is for sort-merge join operations. So it will only join sorted streams. The addition, multiplication and sum modules work with 64 bit decimal values.

The interface specification documentation will be available later.

How does it work?

To run the program use the following command:

dbmstodspi -i INPUT_DEF.json -c CONFIG.ini

The first file defines the query plan. The second file contains json filepaths to define which bitstreams are available with their respective constraints. Examples can be found here:

• INPUT_DEF.json
• CONFIG.ini

More info on the how the data is mapped to the interface and how the data is processed can be found here.

How to use the source?

The documentation for this project is created using CMake with Doxygen. To turn the documentation build option on you need to change the main CMakeLists.txt file and then you can find the output in the build directory under the doc_doxygen folder. The FOS library and test library aren't included in the generated documentation.

Formatting is done according to the clang-tidy options you can find here.

Testing for this code is done using googletest and is automatically downloaded when the code is built with testing options turned on.

Quick steps to running the benchmark from zero:

• Get a ZCU102 SD card image here or build it yourself
• Install the OS to an 16GB SD card. Further explanation here or just use the following command:

dd if=<fos_image.img> of=<SD card image> bs=4M status=progress

• Boot the board from the SD card
• Login with default username xilinx password xilinx (change the password)
• Allocate UDMA buffer space and possibly extend CMA space as described in the tutorial here
• Build OrkhestraFPGAStream using CMake
• Create the dataset
  • Get a TPC-H data generator
  • Use data generation script
• Run benchmark perl script