Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add code coverage into the CI #251

Open
slaperche-scality opened this issue Oct 12, 2018 · 0 comments
Open

Add code coverage into the CI #251

slaperche-scality opened this issue Oct 12, 2018 · 0 comments

Comments

@slaperche-scality
Copy link
Contributor

slaperche-scality commented Oct 12, 2018

It would be really nice to compute the code coverage in our CI pipeline, especially since we can upload the result to CircleCI's artifacts directory (BTW, we should propably do the same for our benchmark).

That being said, before getting there we have several changes to make if we want to do it.

Add coverage tooling (easy)

We should install a recent lcov in our Docker image for testing.
Note that it must be a recent lcov because older version don't work with GCC 8.

Make the test suite faster

In order to have a comprehensive coverage, we should run the test suite 3 times:

  • once without SIMD
  • once with SSE
  • once with AVX

Otherwise, some code path won't be covered by the test suite.

In order to keep a bearable user experience, we should remove the stuff that unecessarily slow don our test suite.

First, we should get rid of test_ec.sh:

  • it's way too slow (you can't run the test suite in Debug mode, and I'm not sure Release mode give an accurate coverage)
  • it's too coarse grained

test_ec.sh is based on ec_test, which could be good for integration test but not for unit testing.

I think we should convert as much as possible the test from test_ec.sh into unit tests, and for the few thing that do require a full run then we can add some integration test.

Next, we should remove most of the loop that run the tests thousands of time with random input:

  • that make the tests non-reproducible
  • that's not the role of unit test: if we want to test our behavior with random input to detect numeric instabilities or other bugs this should be done by using instrumentation-guided fuzzing (such as AFL), not by randomly adding randomness hither and yon.

Finally, we should move the benchmark into it's own CircleCI job and update the testing job to run the testing suite thrice before collecting the code coverage results.

slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
That way, we are sure that every codepath is covered and we won't
silently break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
That way, we are sure that every codepath is covered and we won't
silently break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
That way, we are sure that every codepath is covered and we won't
silently break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
That way, we are sure that every codepath is covered and we won't
silently break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 12, 2018
That way, we are sure that every codepath is covered and we won't
silently break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 15, 2018
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 15, 2018
That way, we are sure that every codepath is covered and we won't
silently break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 15, 2018
That way, it can be ran in parallel with the test suite.

Refs: scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 15, 2018
Now, we are sure that every codepath is covered and we won't silently
break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 16, 2018
That way, it can be ran in parallel with the test suite.

Refs: scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 16, 2018
Now, we are sure that every codepath is covered and we won't silently
break one of them.

Refs: scality#222, scality#251
slaperche-scality added a commit to slaperche-scality/quadiron that referenced this issue Oct 16, 2018
Now, we are sure that every codepath is covered and we won't silently
break one of them.

Refs: scality#222, scality#251
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant