Missing benchmark

[meisterT]

Description of the problem / feature request:

bazel-bench did not run on all specified bazel commits.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Ran the following command:

bazel run :benchmark -- --bazel_commits ffad58728d850b465a070d25b037ea1bced0f91f,4782b8872c3bef8df56f3714ffd005dd3dcc95f7 --bazel_source /usr/local/google/home/twerth/bazel-up --collect_memory  --data_directory /tmp/bazel-g1_2 --project_commits ffad58728d850b465a070d25b037ea1bced0f91f --project_source /usr/local/google/home/twerth/bazel-up --runs 20 --collect_json_profile -- build --compilation_mode=opt //src:bazel

Note that I am using the same local repository as bazel and project source - is that causing any problems?

From the logs:

INFO: Running command line: bazel-out/k8-py2-fastbuild/bin/benchmark --bazel_commits ffad58728d850b465a070d25b037ea1bced0f91f,4782b8872c3bef8df56f3714ffd005dd3dcc95f7 --bazel_source /usr/local/google/home/twerth/bazel-up --collect_memory --data_directory /tmp/bazel-g1_2 --project_commits ffad58728d850b465a070d25b037ea1bced0f91f --project_source /usr/local/google/home/twerth/bazel-up --runs 20 --collect_json_profile -- build '--compilation_mode=opt' //src:bazel
...
I0619 07:56:51.347815 139708403218240 logger.py:33] Preparing bazelbuild/bazel repository.
I0619 07:56:51.347987 139708403218240 logger.py:33] Path /tmp/.bazel-bench/bazel exists. Updating...
I0619 07:56:51.811319 139708403218240 logger.py:33] Preparing /usr/local/google/home/twerth/bazel-up clone.
I0619 07:56:51.811598 139708403218240 logger.py:33] Path /tmp/.bazel-bench/project-clones/cbf1f465934e5495d5024f0a2bd99ea5 exists. Updating...
I0619 07:56:52.261014 139708403218240 logger.py:33] Binary exists at /tmp/.bazel-bench/bazel-bin/ffad58728d850b465a070d25b037ea1bced0f91f/bazel, reusing...
I0619 07:56:52.328670 139708403218240 logger.py:33] Pre-fetching external dependencies & exporting build event json to /tmp/.bazel-bench/out/build_env.json...

So it is already missing in the preparation phase.

[meisterT]

As one more data point: running rm -rf /tmp/.bazel-bench/ brings us back the missing benchmark on the next run. After that it's missing again.

[joeleba]

As discussed offline, bazel-bench goes through the process of building binary & running benchmark for each commit sequentially, instead of building every binary upfront. So it's WAI that the log shows the creation of 1 binary followed by the benchmarking for that binary.