bench: generate random inputs for each benchmark iteration #25

shekhirin · 2025-06-17T13:21:57Z

Previously benchmarks reused the same static data for all iterations, which could lead to unrealistic performance measurements due to cache effects and lack of input variation.

Now we use proptest::arbitrary to generate random data.

Previously benchmarks reused the same static data for all iterations, which could lead to unrealistic performance measurements due to cache effects and lack of input variation. Changes: - Add rand dependency for random data generation - Replace static data setup with iter_batched pattern - Generate fresh random inputs for each benchmark iteration - Update all 20+ benchmark functions to use random inputs - Remove unused helper functions This ensures more realistic and robust performance measurements by testing with varied inputs on each iteration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

codspeed-hq · 2025-06-17T13:23:14Z

CodSpeed Performance Report

Merging #25 will degrade performances by 98.5%

_{Comparing alexey/fix-benchmark-random-inputs (a4c8884) with main (866d3f7)}

Summary

❌ 62 regressions
✅ 8 untouched benchmarks
🆕 20 new benchmarks
⁉️ 45 dropped benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
❌	`clone[16]`	335.3 ns	8,894.7 ns	-96.23%
❌	`clone[32]`	306.4 ns	8,893.3 ns	-96.55%
❌	`clone[64]`	306.4 ns	8,893.3 ns	-96.55%
❌	`clone[8]`	335.8 ns	8,950.3 ns	-96.25%
❌	`cmp[16]`	305.8 ns	8,861.4 ns	-96.55%
❌	`cmp[32]`	303.6 ns	8,550 ns	-96.45%
❌	`cmp[64]`	303.6 ns	8,579.2 ns	-96.46%
❌	`cmp[8]`	305.8 ns	8,825.6 ns	-96.53%
❌	`common_prefix_length[16]`	251.4 ns	10,220.3 ns	-97.54%
❌	`common_prefix_length[32]`	295.8 ns	13,048.1 ns	-97.73%
❌	`common_prefix_length[64]`	384.7 ns	16,682.8 ns	-97.69%
❌	`common_prefix_length[8]`	200 ns	9,120.3 ns	-97.81%
❌	`ends_with[16]`	276.9 ns	8,887.5 ns	-96.88%
❌	`ends_with[32]`	276.9 ns	8,870.3 ns	-96.88%
❌	`ends_with[64]`	276.9 ns	8,955 ns	-96.91%
❌	`ends_with[8]`	276.9 ns	8,782.8 ns	-96.85%
❌	`extend[16]`	466.4 ns	11,050 ns	-95.78%
❌	`extend[32]`	466.7 ns	11,105.6 ns	-95.8%
❌	`extend[8]`	496.1 ns	11,190.3 ns	-95.57%
❌	`first[16]`	150.8 ns	4,367.5 ns	-96.55%
...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

DaniPopes · 2025-06-17T13:34:43Z

benches/bench.rs

-            b.iter(|| Nibbles::from_nibbles(black_box(data)))
+        group.bench_function(BenchmarkId::from_parameter(size), |b| {
+            b.iter_batched(
+                || generate_nibbles_random(size),


makes sense but this will only work locally; codspeed only uses 1 input regardless of the function used. See ruint benchmarks for a wrapper to run multiple times on codspeed

benches/prelude.rs

benches/bench.rs

benches/prelude.rs

benches/bench.rs

benches/prelude.rs

benches/bench.rs

DaniPopes · 2025-06-19T11:13:35Z

benches/prelude.rs

+#[inline]
+fn manual_batch<T, U>(mut setup: impl FnMut() -> T, mut f: impl FnMut(&T) -> U) -> impl FnMut(T) {
+    let inputs =
+        criterion::black_box((0..CODSPEED_BATCH_SIZE).map(|_| setup()).collect::<Vec<_>>());


use std::hint::black_box

shekhirin and others added 2 commits June 17, 2025 13:19

fmt

0a3f472

shekhirin marked this pull request as ready for review June 17, 2025 13:24

shekhirin changed the title ~~fix: generate random inputs for each benchmark iteration~~ test: generate random inputs for each benchmark iteration Jun 17, 2025

shekhirin changed the title ~~test: generate random inputs for each benchmark iteration~~ bench: generate random inputs for each benchmark iteration Jun 17, 2025

DaniPopes reviewed Jun 17, 2025

View reviewed changes

shekhirin force-pushed the alexey/fix-benchmark-random-inputs branch 4 times, most recently from df4a3bb to 9020723 Compare June 17, 2025 17:04