Does each Sample in a Benchmark have the same number of evals?

[gdalle]

It would seem natural but since each Sample carries its own evals field I wanted to double-check

[LilithHafner]

Yes, that is the case, at least for now. I don't want to guarantee it because we might want to add a more adaptive evals selection method. There is a special case in the display logic to alert you if a benchmark has variable sample count:

julia> bench = @be 1+1
Benchmark: 5636 samples with 13167 evaluations
min    1.136 ns
median 1.206 ns
mean   1.205 ns
max    3.709 ns

julia> only(unique([s.evals for s in bench.samples]))
13167.0

julia> samp = @b rand sqrt
1.981 ns

julia> bench.samples[1] = samp # This is a strange thing to do...
1.981 ns

julia> bench
Benchmark: 5636 samples with variable evaluations
min    1.136 ns
median 1.206 ns
mean   1.205 ns
max    3.709 ns

[gdalle]

Good to know, thanks!
For context, I use Chairmarks within DifferentiationInterfaceTest, and to work around the current absence of ChairmarkGroup I defined an ad-hoc type for results, which I finally got around to document
https://gdalle.github.io/DifferentiationInterface.jl/DifferentiationInterfaceTest/dev/api/#DifferentiationInterfaceTest.DifferentiationBenchmarkDataRow