Empty code coverage collection for llvm lit test

[yyyokata]

Description of the bug:

While i'm trying to test a llvm/mlir based repo with lit test configured as below:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/glob_lit_test.bzl
CMD:bazel coverage //workspace/... --action_env=COVERAGE_GCOV_OPTIONS=-b --instrumentation_filter=/workspace[/:] --combined_report=lcov
All coverage.dat for lit tests are empty.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Ubuntu 20.04

What is the output of bazel info release?

release 5.3.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

My walkaround:
Since lit test is a wrap of py_test, I tried the walkaround mentioned in #15098, but changed collect_cc_coverage.sh a few:

function init_gcov() {
  # Symlink the gcov tool such with a link called gcov. Clang comes with a tool
  # called llvm-cov, which behaves like gcov if symlinked in this way (otherwise
  # we would need to invoke it with "llvm-cov gcov").
  # For more details see https://llvm.org/docs/CommandGuide/llvm-cov.html.
  GCOV="${COVERAGE_DIR}/gcov"
  if [ ! -f "${COVERAGE_GCOV_PATH}" ]; then
    echo "GCov does not exist at the given path: '${COVERAGE_GCOV_PATH}'"
    exit 1
  fi
  # When using a tool from a toolchain COVERAGE_GCOV_PATH will be a relative
  # path. To make it work on different working directories it's required to
  # convert the path to an absolute one.
  COVERAGE_GCOV_PATH_ABS="$(cd "${COVERAGE_GCOV_PATH%/*}" && pwd)/${COVERAGE_GCOV_PATH##*/}"
  ln -s "${COVERAGE_GCOV_PATH_ABS}" "${GCOV}"

 + # init bazel-out
 + BAZEL_OUT_PATH="$(realpath ${RUNFILES_DIR}/bazel-out)"
 + ln -s ${BAZEL_OUT_PATH} $(dirname ${GCOV})
}

After link bazel-out to gcov path, coverage for single test can be generated properly.
For multi-test, it seems like there is still another issue, in https://github.com/bazelbuild/bazel/blob/master/tools/test/collect_cc_coverage.sh#L151, gcov will generate json/gcov files under CMD's current directory, which is shared by all tests but not under their own runfile path, the logic below raises a bad file move under gcov 9:

if [[ $gcov_major_version -ge 9 ]]; then
   mv -- *.gcov.json.gz "$(dirname "$output_file")/$(dirname ${gcno_path})"
  # if multi tests are runing and generating gcov.json at the same time, and one of them mv everything away,
  # all other tests will gain an error like this: mv: cannot stat '*.gcov.json.gz': No such file or directory
  # and will end like this:
  # Jul 03, 2023 6:49:18 AM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
  # SEVERE: File 
 # /root/.cache/bazel/_bazel_root/284ccfd4b38efa0b6060c8f1f0461411/execroot/org_genesis/_coverage/genesis/Compiler/Strategies/Test/UtilTest/test/bazel-out/k8-opt/bin/genesis/CodeGen/Dialect/LinalgExt/IR/_objs/LinalgExtDialect/LinalgExtTransformOps.gcda.gcov.json.gz could not be parsed due to xxxxx
else
   cat -- *.gcov >> "$output_file"
  # i do believe cat/rm will also cause some undefined behaviours.
   rm -- *.gcov
fi

So I changed this part to code below and no more error raised.

function gcov_coverage() {
  local output_file="${1}"; shift
+  pushd $(dirname "$output_file")
  # We'll save the standard output of each the gcov command in this log.
  local gcov_log="$output_file.gcov.log"

  # Copy .gcno files next to their corresponding .gcda files in $COVERAGE_DIR
  # because gcov expects them to be in the same directory.
  while read -r line; do
    if [[ ${line: -4} == "gcno" ]]; then
      gcno_path=${line}
      local gcda="${COVERAGE_DIR}/$(dirname ${gcno_path})/$(basename ${gcno_path} .gcno).gcda"
      if [[ -f "$gcda" ]]; then
          if [ ! -f "${COVERAGE_DIR}/${gcno_path}" ]; then
              mkdir -p "${COVERAGE_DIR}/$(dirname ${gcno_path})"
              cp "$ROOT/${gcno_path}" "${COVERAGE_DIR}/${gcno_path}"
          fi
          "${GCOV}" -i $COVERAGE_GCOV_OPTIONS -o "$(dirname ${gcda})" "${gcda}"
      fi
    fi
  done < "${COVERAGE_MANIFEST}"
 + popd
}

[Pavank1992]

Hi @yyyokata, Could you please help us with minimal steps to reproduce the issue with sample code. It will help us to narrow down the issue. Thanks!

[c-mita]

With your "workaround", do you get coverage data? Does this happen with any py_test?

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.