Skip to content

babybear-labs/benchmark

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

138 Commits
.idea
.idea
 
 
circom
circom
 
 
documentation
documentation
 
 
halo2 @ ba4d9c5
halo2 @ ba4d9c5
 
 
jolt
jolt
 
 
nexus
nexus
 
 
powdr
powdr
 
 
reports
reports
 
 
risc-zero
risc-zero
 
 
sp1
sp1
 
 
zokrates
zokrates
 
 
.gitmodules
.gitmodules
 
 
Readme.md
Readme.md
 
 

Repository files navigation

ZK Benchmark

Overview

This repository hosts the source code and resources for benchmarking various Zero-Knowledge (ZK) proofs. It builds upon the original repositories of Halo2, RISC0, Nexus, SP1, Jolt, Circom, and Powdr with our implementations of proofs including but not limited to SHA256, Fibonacci, and Poseidon operations.

Our aim is to benchmark the performance of these proofs, comparing them in terms of time, space, and other metrics. This comprehensive comparison will not only highlight the most popular ZK proofs and their implementations but also deepen our understanding of their inner workings, developer friendliness, and proving systems.

Goals

  • Benchmark Operations:

    • Fibonacci: Variants 1, 10, 100, 1000, 10000
    • SHA256: Variants 32b, 1k, 10k Bytes
    • Poseidon: Variants 32b, 100b

  • Additional Benchmarks:

    • Arithmetic Operations: Addition, Subtraction, Multiplication, Division
    • Loop: Variants 10, 100, 1000
    • Hash Functions: Pederson, RPO, Keccak (Variants 1k, 10k Bytes)
    • Merkle Tree: Generation, Membership/Inclusion, Merge
    • nth Prime: Variants 1, 10, 100, 1000, 10000
    • Cryptographic Verifications: ECDSA, Elliptic Curve, BLS Verification, BLS Aggregation

General overview of features of proving schemes

Proof System Setup Complexity Post-Quantum Resistance Scalability Parallel Execution Features
Halo2 Transparent No (ECC based) High Moderate Recursive SNARKs, Custom gates, Plonkish arithmetization
Circom (Groth16) Trusted Setup No (Pairing-based) Moderate High Efficient SNARK generation for circuits, good for small proof
Risc Zero Transparent Yes High High General-purpose zkVM, RISC-V architecture, supports arbitrary computations (STARK + AIR + FRI + RISC V)
Jolt Both Yes Very High Very High Sumcheck and lookup arguments, optimized for high-performance proof generation, upcoming twist and shout ( Spartan Sumcheck + R1CS + Zero morph + RISC V )
Nexus zkVM Transparent Partial Moderate High Privacy-focused zkVM, Ethereum compatibility, supports smart contract verification (R1CS + Relazxed R1CS + Nova + Hyper Nova + Risetto255 + RISC V )
SP1 Transparent Yes Very High Very High Optimized for rollups, efficient parallel proving, GPU-optimised, supports high-throughput applications, supports arbitrary computations, whitepaper. ZK: Plonky3 (STARK + AIR + FRI + RISC V), Multi set hashing, towards real-time proving with jagged PCS
Powdr Transparent Yes High High Developer-friendly, multiple proof systems, and zk-continuations for unbounded execution, minimal setup
Polygon Miden Transparent Yes High High STARK-based (Winterfell) zkVM, designed for program execution proofs, high scalability
Aleo SnarkVM Trusted Setup No (Pairing-based) Moderate High Privacy-preserving SNARK-based zkVM, supports private smart contracts
Zokrates Trusted Setup No (Pairing-based) Moderate High High-level DSL for zk-SNARKs, extensive developer tooling, supports Groth16 and PLONK
Delphinus ZKWASM Transparent Yes High High WebAssembly-based zkVM, supports proving WASM execution, ideal for cross-platform apps
Lita Valida Transparent Yes Moderate Moderate Modular chip, ensures the correctness of zk circuits, supports custom proofs (STARK + AIR + FRI + LITA ISA)
Eigen ZKVM Transparent Yes Very High High High-performance zkVM, designed for Ethereum scaling, supports recursive proofs
CairoVM Transparent Yes Very High High STARK-based VM, designed for StarkNet, supports general-purpose computation
Noir Transparent Yes (STARK-based) High High High-level zkDSL, supports general-purpose zk circuits, STARK-based backend
Ola VM Transparent Yes High High STARK-based, Privacy-preserving zkVM, supports custom circuits, programmable, scalable, and private proof
Triton VM Transparent Yes High High Efficient recursive verification, Algebraic Execution Tables (AET) and Arithmetic Intermediate Representations (AIR) with a STARK proof system.
Lurk Transparent Yes Moderate Moderate zk-LISP interpreter, supports verifiable computations, ideal for symbolic execution
Ceno Transparent Yes High High Non-uniform prover based on GKR Protocol, Segment and Parallel Zero-knowledge Virtual Machine
Expander Transparent Yes Very High Very High GKR + Libra, Parallel computing
OpenVM Transparent Yes High High Modular architecture, supports general-purpose zkVM, adaptable to various use cases
Brevis Transparent Yes Very High High Focused on proof aggregation and recursion, modular, glue-and-coprocessor architecture, multiple proving backends, Coprocessor Integration (STARK / CIRCLESTARK + AIR + FRI + RISC V)
PetraVM Transparent Yes High High Binius based

The key differences between these proof systems include their setup complexity, proof features, post-quantum resistance, scalability, and parallel execution capabilities. Some use recursive proofs, while others focus on efficiency or general-purpose functionality. The systems also vary in their post-quantum resistance and ability to scale and parallelize.

Proving Schemes

Name Type Benchmark Code Reports
Jolt zkVM code report
Powdr zkVM code report
Circom HDL code report
Nexus zkVM code report
Halo2 Proof System code report
Risc Zero zkVM code report
SP1 zkVM code report
Polygon Miden zkVM TBA TBA
Aleo SnarkVM zkVM TBA TBA
Zokrates DSL TBA TBA
Delphinus Lab ZKWASM zkVM TBA TBA
Lita Valida zkVM TBA TBA
Eigen ZKVM zkVM TBA TBA
CairoVM zkVM TBA TBA
Noir DSL TBA TBA
Ola VM zkVM TBA TBA
Triton VM zkVM TBA TBA
Lurk zkVM TBA TBA
Ceno zkVM TBA TBA
Expander zkVM TBA TBA
OpenVM zkVM TBA TBA
Brevis Pico zkVM TBA TBA
  • zkVM: Zero-Knowledge Virtual Machine, a virtual machine designed to execute programs written in standard programming languages while generating zero-knowledge proofs of their correctness.
  • HDL: Hardware Description Language, used for circuit descriptions.
  • DSL: Domain-Specific Language, designed specifically for writing ZKP programs.
  • Proof System: Framework for creating and verifying zk-proofs
  • TBA: To Be Added

Result Summary

Only selected operations are shown. The detailed reports have metrics for other variants and additional operations.

Fibonacci

10000 th number

Proof System Prover Time (s) Cycles Verifier Time (s) Prover Memory (KB) Proof Size (B)
Halo2 0.196 - 0.004 9.8 1664
Circom 1.75 9999 0.81 466280 805
Risc Zero 6.37 65536 - - 206182
Jolt 21.73 280287 0.01 - 452398
Nexus (max 100) 35.2 - 2.4 - 47.9 MB
SP1 18.87 69101 0.174 - 2656912
Powdr 8.64 2990 - - -

SHA256

1 KB Input

Proof System Prover Time (s) Cycles Verifier Time (s) Prover Memory (KB) Proof Size (B)
Halo2 14.78 - 0.13 1134 4064
Circom 46.07 540736 1.14 3920848 805
Risc Zero 2.5 65536 - - 210157
Jolt 2.199 62231 0.052 - 401116
Nexus 30+ mins - - - -
SP1 17.6 71249 0.172 - 265691
Powdr 9.07 73731 - - -

Poseidon

32 Byte input

Proof System Prover Time (s) Cycles Verifier Time (s) Prover Memory (KB) Proof Size (B)
Halo2 8.74 - 0.086 25 2144
Circom 1.19 4184 0.72 373560 804
Risc Zero 5.47 524288 - - 256742
Jolt 91.38 554595 0.19 - 477746
SP1 112.5 39479 0.509 - 2876912
Powdr 21.54 286652 - - -

Additional reports in progress.

Metrics

  • Prover Time
  • Verifier Time
  • Prover Space
  • Verifier Space
  • Proof Size
  • Parallel Execution
  • CPU Usage
  • Memory Usage
  • Verifier Gas Consumption
  • Cycles Count
  • Developer Friendliness

Reports

The detailed reports are in the reports directory.

Structure

flowchart TD
    %% Global Components
    BO("Benchmark Orchestrator"):::orchestrator
    REP("Reporting/Aggregation Module"):::report
    DOC("Documentation Component"):::doc

    %% Proof System Modules
    subgraph "Proof System Modules"
        %% Circom Module with internal details
        subgraph "Circom Module"
            COV("Overview"):::module
            C1("Circuits"):::subcomponent
            C2("Compilation&Proving Scripts"):::subcomponent
            C3("Test Scripts"):::subcomponent
            COV --> C1
            C1 --> C2
            C2 --> C3
        end
        H("Halo2 Module"):::module
        J("Jolt Module"):::module
        N("Nexus Module"):::module
        P("Powdr Module"):::module
        RZ("RISC Zero Module"):::module
        SP("SP1 Module"):::module
        Z("Zokrates Module"):::module
    end

    %% Data Flow Connections from Orchestrator to Modules
    BO -->|"triggers"| COV
    BO -->|"triggers"| H
    BO -->|"triggers"| J
    BO -->|"triggers"| N
    BO -->|"triggers"| P
    BO -->|"triggers"| RZ
    BO -->|"triggers"| SP
    BO -->|"triggers"| Z

    %% Modules output benchmark results to Reporting Module
    COV -->|"outputs"| REP
    H -->|"outputs"| REP
    J -->|"outputs"| REP
    N -->|"outputs"| REP
    P -->|"outputs"| REP
    RZ -->|"outputs"| REP
    SP -->|"outputs"| REP
    Z -->|"outputs"| REP

    %% Documentation guides Orchestrator and Modules
    DOC ---|"guides"| BO
    DOC ---|"guides"| COV
    DOC ---|"guides"| H
    DOC ---|"guides"| J
    DOC ---|"guides"| N
    DOC ---|"guides"| P
    DOC ---|"guides"| RZ
    DOC ---|"guides"| SP
    DOC ---|"guides"| Z

    %% Click Events for Component Mapping
    click COV "https://github.com/babybear-labs/benchmark/tree/main/circom"
    click C1 "https://github.com/babybear-labs/benchmark/tree/main/circom/circuits"
    click C2 "https://github.com/babybear-labs/benchmark/blob/main/circom/scripts/compile_circuit.sh"
    click C3 "https://github.com/babybear-labs/benchmark/blob/main/circom/test_fibonacci.py"
    click H "https://github.com/babybear-labs/benchmark/tree/main/halo2"
    click J "https://github.com/babybear-labs/benchmark/tree/main/jolt"
    click N "https://github.com/babybear-labs/benchmark/tree/main/nexus"
    click P "https://github.com/babybear-labs/benchmark/tree/main/powdr"
    click RZ "https://github.com/babybear-labs/benchmark/tree/main/risc-zero"
    click SP "https://github.com/babybear-labs/benchmark/tree/main/sp1"
    click Z "https://github.com/babybear-labs/benchmark/tree/main/zokrates"
    click REP "https://github.com/babybear-labs/benchmark/tree/main/reports"
    click DOC "https://github.com/babybear-labs/benchmark/blob/main/documentation/Documentation.pdf"

    %% Styles
    classDef orchestrator fill:#FFCCFF,stroke:#990099,stroke-width:2px;
    classDef module fill:#BBDEFF,stroke:#0D47A1,stroke-width:2px;
    classDef subcomponent fill:#E3F2FD,stroke:#1565C0,stroke-width:1px;
    classDef report fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px;
    classDef doc fill:#FFE0B2,stroke:#E65100,stroke-width:2px;
Loading

Benchmark Machine Specifications

  • Macbook M1 Pro: Core 8, Memory 8 GB
  • AlmaLinux 8.10: Core 16, Memory 32 GB, Disk 1 TB
  • Windows 11: Core 16, Memory 32 GB, Disk 1 TB
  • Intel(R): Core 4 - Memory 8GB
  • Macbook M2 Pro: Core 16, Memory 16 GB

Installation

Each project has its own README file with instructions on installing the dependencies and running the benchmarks.

Roadmap

Please refer to the issues.

Contributors

This project is part of the ZK and Scaling Bootcamp organized by the Encode club.

Other benchmarks

Acknowledgments

Special thanks to the ZK and Scaling Bootcamp team for organizing this event and providing us with the opportunity to learn and contribute to the ZK space.

About

Benchmark of ZKVMs and Proving Schemes

Resources

Readme
Activity
Custom properties

Stars

29 stars

Watchers

2 watching

Forks

7 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 5

Languages