Skip to content
This repository has been archived by the owner on Jun 20, 2022. It is now read-only.

Latest commit

 

History

History
63 lines (43 loc) · 1.47 KB

README.md

File metadata and controls

63 lines (43 loc) · 1.47 KB
Raw

Bolson

A JSON to Arrow IPC converter and Pulsar publishing tool.

Goal

Functionality

  • Bolson receives newline-separated JSONs over a TCP connection.
  • The JSONs are converted to Arrow RecordBatches.
  • The Arrow RecordBatches are serialized to an Arrow IPC message.
  • The IPC messages are published to a Pulsar broker.

Performance

  • The implementation aims to achieve high throughput and low latency.
  • The implementation allows using FPGA accelerators for more performance.

Build

To build Bolson, make sure your system adheres to the following requirements:

  • Toolchain:
    • CMake 3.14+
    • A C++17 compiler.
  • Dependencies:

Build Bolson as follows:

git clone https://github.com/teratide/bolson.git
cd bolson
mkdir build
cd build
cmake ..
make

Usage

There are two subcommands, stream and bench.

More detailed options can be found by running:

bolson --help <subcommand>

FPGA-accelerated parsing

To enable FPGA-accelerated parsing, continue to read here.

FAQ

  • Why is it called Bolson?
    • The name is inspired by the "Bolson Pupfish", which sounds a bit like " JSON publish". It's a working title.