way to run and compare all benchmarks in a crate

[andrewgazelka]

right now we have to do

cargo bench -q -p daft-minhash --bench=minhash -- compare target/benchmarks/minhash
cargo bench -q -p daft-minhash --bench=windowed -- compare target/benchmarks/windowed

It would be nice if we could instead do

cargo bench -q -p daft-minhash -- compare target/benchmarks

Right now my workflow combines tango-bench and cargo export

[bazhenov]

This is neat idea I would really love to implement, because I have very similar use case when I ran several benchmarks (from different crates) using shell-script.

Unfortunately, it will not work because cargo bench actually run not only benches but also unit tests. When you will pass any argument that standard harness is not supports (like -t) it will fail the whole cargo bench process. You can check it yourself, run following command in the context of a project that has at least 1 unit test. It should fail.

$ cargo bench -- compare -t 0.1
    Finished `bench` profile [optimized] target(s) in 0.04s
     Running unittests src/lib.rs (target/release/deps/pmem-dbbad5b7c1126fc4)
error: Unrecognized option: 't'
error: bench failed, to rerun pass `-p pmem --lib`

One workaround I can think of is to implement separate runner cargo tango that will run only benches.

[andrewgazelka]

having cargo tango would be really nice. I want to make a CI runner for this and being able to only run tango benches is a requirement.

[andrewgazelka]

here's an example of where I was using tango... but it could be better if I was able to see all benches that are made with tango

hyperion-mc/hyperion#201

[andrewgazelka]

another example... I just re-integrated it. Right now I am using a matrix

hyperion-mc/hyperion#781