test(benchmarks): improve bechmarks to have complete performance data (…

…privacy-scaling-explorations#326) * test(benchmarks): improve bechmarks to have complete performance data - Added Sparse Merkle Tree to the benchmark suites to complete the available merkle tree implementations in zk-kit. - Added operation to the benchmark suites (proof generation, proof verification, update, delete - Changed the "imt.ts" name to "benchmark-merkle-trees.ts" - Added README.md documentation about the benchmark suite - Added log managment with winston library to handle any error produced during the benchmark withtout break the benchmark execution re privacy-scaling-explorations#311 * fix(benchmarks): remove package winston and to-hex libraries to fix validation issues in GH remove package winston and to-hex libraries to fix validation issues in GH re privacy-scaling-explorations#311 * docs(benchmarks): added benchmarks summary Added benchmarks summary tables for different sizes of samples and conclusion ideas re # 311 * Update benchmarks/README.md Co-authored-by: sripwoud <[email protected]> * Update benchmarks/benchmark-merkle-trees.ts Co-authored-by: sripwoud <[email protected]> * Update benchmarks/README.md Co-authored-by: sripwoud <[email protected]> * Update benchmarks/benchmark-merkle-trees.ts Co-authored-by: sripwoud <[email protected]> * Update benchmarks/README.md Co-authored-by: sripwoud <[email protected]> * refactor(benchmarks): added winston to package.json and refactored index.ts Added winston to package.json and refactored index.ts to run the samples with a loop managed by an array re privacy-scaling-explorations#311 * Update benchmarks/benchmark-merkle-trees.ts Co-authored-by: sripwoud <[email protected]> * fix(benchmarks): added counter initialization mistake Changed to initialize from -1 instead of 0 to fix the logic to start from the first samples element re privacy-scaling-explorations#311 * docs(benchmarks): changed the usage instructions to point the exisitin script Change the usage instructions of benchmarks from the npx command to use the exisiting script in the package.json re privacy-scaling-explorations#311 * style(benchmarks): benchmark-merkle-trees.ts changes from prettier validation Minimal change in the code to add an space to a instruction from prettier lib suggestion * fix(benchmarks): added the new version of yarn.lock after the run the yarn command in my local environment plus add the new dev dependency winston the yarn.lock have changed some lib versions and added winston re privacy-scaling-explorations#311 --------- Co-authored-by: sripwoud <[email protected]> Co-authored-by: Cedoor <[email protected]>
ursulabauer · Sep 25, 2024 · 9adf8bf · 9adf8bf
1 parent 71db371
commit 9adf8bf
Show file tree

Hide file tree

Showing 7 changed files with 1,723 additions and 1,192 deletions.
diff --git a/README.md b/README.md
@@ -513,3 +513,25 @@ You can see some examples in the `benchmarks` folder. All you have to do is crea
 | **Incremental**      | [@zk-kit/imt](https://github.com/privacy-scaling-explorations/zk-kit/tree/main/packages/imt)           | Fastest for incremental updates.        | [Semaphore V3](https://github.com/semaphore-protocol/semaphore/tree/v3.15.2), [Worldcoin](https://github.com/worldcoin) |
 | **Lean Incremental** | [@zk-kit/lean-imt](https://github.com/privacy-scaling-explorations/zk-kit/tree/main/packages/lean-imt) | Optimized for lightweight environments. | [Semaphore V4](https://github.com/semaphore-protocol/semaphore), [Zupass](https://github.com/proofcarryingdata/zupass)  |
 | **Sparse**           | [@zk-kit/smt](https://github.com/privacy-scaling-explorations/zk-kit/tree/main/packages/smt)           | Handles very large sets efficiently.    | [Iden3](https://github.com/iden3)                                                                                       |
+
+Following benchmarks data of zk-kit Merkle Trees implementations:
+|8 leafs|insert|delete|update|generate proof|verify proof|
+|--|:--:|:--:|:--:|:--:|:--:|
+|fastest|IMT|IMT ~ SparseMT|LeanIMT|LeanIMT|IMT|
+|slowest|LeanIMT|IMT ~ SparseMT|IMT|SparseMT|SparseMT|
+
+| 128 leafs | insert  |  delete  | update  | generate proof | verify proof |
+| --------- | :-----: | :------: | :-----: | :------------: | :----------: |
+| fastest   |   IMT   | SparseMT | LeanIMT |    LeanIMT     |   SparseMT   |
+| slowest   | LeanIMT |   IMT    |   IMT   |      IMT       |     IMT      |
+
+| 1024 leafs |  insert  |  delete  | update  | generate proof | verify proof |
+| ---------- | :------: | :------: | :-----: | :------------: | :----------: |
+| fastest    | SparseMT | SparseMT | LeanIMT |    LeanIMT     |   SparseMT   |
+| slowest    | LeanIMT  |   IMT    |   IMT   |      IMT       |     IMT      |
+
+From the benchmark data we can take another criteria to evaluate which Merkle tree should be used:
+
+-   IMT have the best performance for medium and small size insert related operations.
+-   LeanIMT have the best performance for all the merkle tree sizes for update and generate proof related operations.
+-   Sparse is good for larger data insert, delete and verify proof.
diff --git a/benchmarks/README.md b/benchmarks/README.md
@@ -0,0 +1,99 @@
+# ZK-Kit: Merkle Tree Benchmarks
+
+This project is part of the `zk-kit` repository and includes benchmarking tests for three Merkle Tree implementations:
+
+-   **Incremental Merkle Tree (IMT)** from `@zk-kit/imt`
+-   **Lean Incremental Merkle Tree (LeanIMT)** from `@zk-kit/lean-imt`
+-   **Sparse Merkle Tree (SMT)** from `@zk-kit/smt`
+
+The benchmark compares the performance of each Merkle Tree implementation for operations like adding leaves, generating proofs, and verifying proofs.
+
+## Table of Contents
+
+-   [Installation](#installation)
+-   [Usage](#usage)
+-   [Benchmark](#benchmark)
+-   [Results](#results)
+-   [Dependencies](#dependencies)
+
+---
+
+## Installation
+
+Before running the benchmark, ensure that you have Node.js version 20 installed and the required dependencies set up.
+
+1. **Clone the Repository**:
+    ```bash
+    git clone https://github.com/privacy-scaling-explorations/zk-kit.git
+    cd zk-kit
+    yarn
+    ```
+
+## Usage
+
+To run the benchmark, use the following script:
+
+```bash
+yarn run benchmarks
+```
+
+## Benchmark
+
+The benchmark includes the following tests for each Merkle Tree implementation:
+
+1. Add Leaves: Measures the time it takes to insert a number of leaves into the tree.
+2. Generate Proofs: Measures the performance of generating Merkle proofs.
+3. Verify Proofs: Measures how quickly the generated proofs can be verified.
+4. Update Leaves: Measures how quickly the leaves can be updated.
+5. Delete Leaves: Measures how quickly the leaves can be deleted. The Lean Merkle tree is excluded because it does not have deleted method implemented.
+
+## Benchmark Structure.
+
+-   The benchmarks are defined using the [benny](https://caderek.github.io/benny/) library.
+-   Results are saved to .html files in the benchmarks/results directory for visualization and further analysis.
+
+## Results
+
+The results for the benchmarking process will be saved in the following formats:
+
+Charts: Performance charts for each operation are saved as .html files.
+Tables: Detailed benchmark results in tabular form are also saved.
+Results can be found in the benchmarks/results directory after running the benchmark.
+
+## Example of Benchmark Execution
+
+Here’s a typical output from running the benchmark:
+
+```bash
+
+Suite: add-merkle-trees-1000
+IMT - Add 1000 leaves x 12,345 ops/sec ±2.31% (92 runs sampled)
+LeanIMT - Add 1000 leaves x 9,876 ops/sec ±1.87% (87 runs sampled)
+SparseMT - Add 1000 leaves x 7,654 ops/sec ±3.14% (75 runs sampled)
+Fastest is IMT - Add 1000 leaves
+
+Suite: proof-generation-merkle-trees-1000
+IMT - Generated 500 proofs x 4,321 ops/sec ±2.02% (80 runs sampled)
+LeanIMT - Generated 500 proofs x 3,456 ops/sec ±1.75% (78 runs sampled)
+SparseMT - Generated 500 proofs x 2,789 ops/sec ±2.63% (68 runs sampled)
+Fastest is IMT - Generated 500 proofs
+```
+
+The Benchmarks suggested in the index.ts are for **8, 128 and 1024 leafs** to see how each Merkle tree for different sizes of trees
+perform because their theorical expected behavior described [here](https://github.com/privacy-scaling-explorations/zk-kit?tab=readme-ov-file#i-need-to-use-a-merkle-tree-to-prove-the-inclusion-or-exclusion-of-data-elements-within-a-set-which-type-of-merkle-tree-should-i-use).
+
+## Dependencies
+
+This benchmark depends on the following packages:
+
+-   @zk-kit/imt: Incremental Merkle Tree implementation.
+-   @zk-kit/lean-imt: Lean Incremental Merkle Tree implementation.
+-   @zk-kit/smt: Sparse Merkle Tree implementation.
+-   benny: A benchmark library for testing performance.
+-   ts-node: TypeScript execution environment for Node.js.
+-   winston: Logger used for capturing errors and logs during execution.
+
+## Notes
+
+-   Node.js Version: The benchmarks are tested and run with Node.js version 20. Ensure you have the correct version installed.
+-   Logging: Logs, including any errors that occur during benchmarking, are captured using the winston logger and saved to error.log in the root directory.