Measure code coverage

[callahad]

We should find a way to measure the code coverage of our unit tests (e.g., by running our tests under coverage.py or using Twisted Trial's built in --coverage flag)

[H-Shay]

I updated tox.ini in my fork to use commands = trial --coverage tests to use the built-in --coverage flag. It ran all the tests successfully with this change. If this is all that needs to be done let me know and I will open a PR with the change.

[anoadragon453]

@callahad I'm a bit concerned that the coverage for trialin Synapse today isn't really used by anyone on the team, and at least I find it to be more noise than anything else. We talked about having a different interface in the past that allowed tracking coverage changes over time, as well as allowing to peer into a file and see the coverage line-for-line, which may be more useful that just seeing a snapshot of things at the file level.

It would be nice if they were placed elsewhere other than the end of the unit test output, as I often find myself needing to scroll up past it to get to the more immediate need: the failing test output.

This probably isn't the right issue to discuss making the output more useful, but I wonder if we want to hold back on adding coverage output to other projects (i.e Sygnal) until we've got a meaningful setup for it on Synapse.

[callahad]

We talked about having a different interface in the past that allowed tracking coverage changes over time, as well as allowing to peer into a file and see the coverage line-for-line

This is exactly where I want to end up. It seems like Trial's built-in reporting doesn't do much for us in that regard, but coverage.py has a nice coverage html command which spits out a pretty enough report:

which you can then click around in to see line-by-line information:

I wonder if we want to hold back on adding coverage output to other projects (i.e Sygnal) until we've got a meaningful setup for it on Synapse.

I think it might be nice to figure things out in a lower stakes repo (Sygnal) before pushing that rock all the way up the hill that is Synapse 🙂

[callahad]

As a specific end goal, I'd love to see us upload coverage data to CodeCov or Coveralls which would provide the pretty longitudinal graphs, ability to click around into files, etc.

But for now, I think the first milestone is to set up coverage.py and get it working in a Tox environment.

It seems like running coverage run --source=sygnal -m twisted.trial tests && coverage html is 90% of the battle, but maybe there's a way to avoid needing to pass so many command line args by setting up a config file or something?

Then it's on to getting those reports added to Pull Requests -- so selecting between CodeCov and Coveralls, adding report generation it into our GitHub Actions workflow, making sure it runs on the right triggers, etc. I think CodeCov is likely to be easier as it supports uploading coverage information on public repositories without requiring a secret GitHub access token, but I'm open to being convinced that Coveralls is a better fit.

[H-Shay]

I can take a look at making this happen.