Engines, queries, and data for dynamic Datalog computation
This repository maintains a list of dynamic Datalog engines, those that update query outputs in response to changes in the input fact sets.
The repository also maintains several paired Datalog queries and input data, meant to exercise dynamic Datalog execution engines in non-trivial ways. These can be found in the ./problems/ subdirectory. These problems may also be independently interesting for evaluating non-dynamic Datalog engines.
- Differential Dataflow - A Rust framework for incremental data-parallel computation.
- Declarative Dataflow - An interpreted layer on top of Differential Dataflow.
- Differential Datalog - A compiler targeting Differential Dataflow.
- IncA - A framework for incremental program analysis with a Datalog-like language.
We also use the excellent Soufflé as a non-dynamic baseline.
- crdt - A CRDT implementation of a shared text editor, from Martin Kleppmann.
- doop - A fragment of the Doop program analysis tool, from Yannis Smaragdakis.
- galen - An inference task in the GALEN medical ontology, from John Liagouris.
For each problem we record reported times for various systems in various configurations, both to perform any query-specific compilation and then execution. These measurements are meant to be representative rather than definitive. All of the systems support multiple worker threads, and could be run in a variety of configurations on a variety of hardware platforms.
Soufflé can often benefit from join planning help; without this help it can take orders of magnitude longer than it could. Such help is currently provided for the CRDT and Doop benchmarks, and measurements for other queries could improve in the future (especially for those runs in which it did not finish in 1,000 seconds). The Soufflé measurements are all directly using the query.dl file from the decompressed input directory, either with the -c flag (for compilation) or without (for interpretation).
Unlike other measurements, the Differential Dataflow measurements are for hand-written code in a larger language, and can reflect implementation and optimizations not easily available within Datalog. The code for each problem is in the differential/ directory.
| Engine | Compilation | Evaluation | Cores | Notes |
|---|---|---|---|---|
| Soufflé (interpreted) | 0s | 1000s+ (DNF) | 1 | Laptop |
| Soufflé (compiled) | 10.15s | 294.73s | 1 | Laptop |
| Differential Dataflow | 166.26s | 3.44s | 1 | Laptop |
| Declarative Dataflow | 0s | |||
| Differential Datalog | ||||
| IncA |
The CRDT benchmark contains stratified negation, as well as several "maximization" idioms represented in Datalog using a quadratic number of facts, which can be more efficiently implemented as data-parallel reduce operations.
| Engine | Compilation | Evaluation | Cores | Notes |
|---|---|---|---|---|
| Soufflé (interpreted) | 0s | 762.14s | 1 | Laptop |
| Soufflé (compiled) | 93.43s | 111.76s | 1 | Laptop |
| Differential Dataflow | 237.43s | 161.58s | 1 | Laptop |
| Declarative Dataflow | 0s | |||
| Differential Datalog | ||||
| IncA |
The DOOP benchmark is just really quite large.
| Engine | Compilation | Evaluation | Cores | Notes |
|---|---|---|---|---|
| Soufflé (interpreted) | 0s | 1000s+ (DNF) | 1 | Laptop |
| Soufflé (compiled) | 9.31s | 198.19s | 1 | Laptop |
| Differential Dataflow | 111.59s | 123.54s | 1 | Laptop |
| Declarative Dataflow | 0s | |||
| Differential Datalog | ||||
| IncA |
The GALEN benchmark contains joins on highly skewed keys, for which correct or adaptive join orders are important. The most problematic rule (IR4) is presented in the least problematic order for binary joins.