Cairo HDP is a collection of Cairo0 programs designed to verify inclusion proofs and perform computations on the data. These computations can be verified on-chain, enabling trustless operations on any historical data from Ethereum or integrated EVM chains.
docker build -t hdp-cairo . && docker run hdp-cairo
To install the required dependencies and set up the Python virtual environment, run:
make setup
Ensure you run the Cairo program from the virtual environment. To activate the virtual environment, execute:
source venv/bin/activate
Before running the program, prepare the input data. The inputs are provided via the hdp_input.json
file located in the root directory of the HDP project. These inputs can be generated using the HDP CLI. Example inputs are available in tests/hdp/fixtures
.
To run the program, use:
make run-hdp
The program will output the results root and tasks root. These roots can be used to extract the results from the on-chain contract.
Cairo HDP operates in three main stages. First, it verifies the passed state. Upon validation, it executes the defined tasks on the data. Finally, the tasks and results are added to a Merkle tree, returning the respective roots as output.
Verification involves several sequential steps:
The first step is to verify the validity of the passed headers by recreating the MMR root, proving each header's inclusion in the MMR. Since the Herodotus header accumulator stores every Ethereum header, it verifies the headers' validity.
The second step is to verify the passed account and storage slot data's validity by checking MPT proofs against the state root from the respective header.
Currently, the following operators are available:
min
: Returns the minimum value of the passed data.max
: Returns the maximum value of the passed data.sum
: Returns the sum of the passed data.avg
: Returns the average of the passed data.count_if
: Returns the number of elements that satisfy a condition.slr
: Returns the best-fit linear regression predicted point for the supplied data.
These operations can be performed on any verified field, including non-numerical values like addresses or hashes, such as the parent_hash
of a header.
In the final step, the results and tasks are added to a Merkle tree, and the roots of these trees are returned as output. These roots can be used to extract the results from the on-chain contract, enabling multiple aggregations in a single execution.
To add a new aggregation function:
- Add the function to
src/tasks/aggregate_functions
. - Integrate the function into the datalake tasks handler by updating the parameter decoder and the
execute
function. - Define the
fetch_trait
function for this aggregation functionality.
HDP can dynamically load Cairo1 programs at runtime, allowing the creation of Cairo1 modules with aggregate function logic. To add a Cairo1 module:
- Create a new Scarb project in
src/cairo1/
. - Add the new aggregation function file to
src/tasks/aggregate_functions
. - Define the
fetch_trait
function appropriate for this aggregation functionality.
The fetch_trait
is an abstract template containing datalake-specific data fetching functions. Each aggregate function must implement this template individually.
Some tests require Ethereum Mainnet RPC calls. Ensure an environment variable named RPC_URL_MAINNET
is set.
To run the tests (from the virtual environment), execute:
make test-full
- Merkelize: Extract data and add it to a Merkle tree.
- Transaction Datalake: A datalake focused on transactions.
- Iterative Dynamic Layout Datalake: Iterate through a dynamic layout, such as a Solidity mapping.
- Multi Task Executions: Run multiple tasks in a single execution.
- Bloom Filter Aggregate: Generate a bloom filter from the data.
Herodotus Dev Ltd - 2024