Support test coverage reporting without git or a git repository present

[dblandin]

Context

Among our four existing official test reporters for ruby, python, javascript, and php, there is inconsistent support for submitting test coverage reports without git and a git repository present. While we've worked with community contributors to add partial support. this has not been prioritized internally as a general test reporter feature.

There is also inconsistent support from CI providers for commit information within the CI environment. We currently check both the environment and git for information required to generate a coverage report payload.

If the CI provider checks out the source code into a detached HEAD, we can't rely on its branch, so we also look for it in the environment via our CI mappings. Conversely, if we can't determine the git branch from the CI environment, we have to rely on git for that information.

Generating file blob IDs for source files included within the test coverage report in order to compare test coverage data with source listings is particularly more reliable when done from git. There are ways of generating file blob IDs from the file content (and most of our existing test reporters include such a fallback) but we've found such solutions to be occasionally incompatible with git's internal methods.

Related Issues & PRs

codeclimate/ruby-test-reporter#183
codeclimate/ruby-test-reporter#177
codeclimate/ruby-test-reporter#171
codeclimate/ruby-test-reporter#168
codeclimate/ruby-test-reporter#156
codeclimate/ruby-test-reporter#116
codeclimate/ruby-test-reporter#115
codeclimate/ruby-test-reporter#89
codeclimate/ruby-test-reporter#57
codeclimate/ruby-test-reporter#56
codeclimate/ruby-test-reporter#38
codeclimate/javascript-test-reporter#44

Next Steps

We should determine whether or not to support test coverage reporting without git and a git repository present. I'm personally not 100% clear of what happens when the file blob IDs mismatch. I assume we simply don't render file coverage or file coverage annotations for that file on codeclimate.com. Do we know how often these mismatches take place?

I'll hold back any thoughts on implementation until we've addressed these basic "should we" questions.

[ale7714]

You're right, if the blob IDs don't match we don't render the coverage for a source file. Currently we don't have any metric for blob IDs mismatches.

We use git to get the blob id for constants on builder. Wouldn't this discussion require taking in consideration builder? To have consistent blob ids everywhere?

[gdiggs]

Possibility here: use libgit2. @markbates talk to @pbrisbin before starting on this for better definition

[pbrisbin]

It looks like work already started in #102. I think this is the right approach based on all the various caveats, so closing this probably means just getting that over the line.

[ale7714]

@markbates i updated the #102 PR to fix specs. The PR got really big because of the dep update I mentioned the other day. Feel free to update the PR if you need to.

This PR also addresses the fallback blob id issue #99. Gets the blob from the file content by using a function from the go git library.

cc @gordondiggs @pbrisbin

[markbates]

@ale7714 i just merged in #102 does that resolve this ticket as well?

[ale7714]

yes! it does :)