Feature Request: Comments to specify code intended to be covered by test

[rbairwell]

It feel it would be beneficial for developers (such as myself ;) ) to be able to mark what a test is intended to "cover" so that the code coverage ignores anything other than that listed when recording coverage data. This will prevent code being recorded as being tested when it does not have a specific test targeting it.

Background

I know Devel::Cover does support files being selected for coverage when it is called and also supports the source code being marked as Uncoverable , but there is no way of do anything like this from the test suite itself - where it would be easiest for the "test designer" to say what is being tested. There is no way to limit the subroutine/functions/methods included.

Proposal

Whilst I have included below how PHPUnit does this in the PHP world, I know it won't map well to the Perl world (as in PHPUnit each test tends to be in its own method/function). I therefore propose that tests can call the following the following:

Devel::Cover->covers_packages(qw/MyApp::Application MyApp::Application::Thing MyApp::Application::SubModule::*/) # limits coverage to any lines in those packages
Devel::Cover->covers_functions(qw/MyApp::Application::method MyApp::Application::thing MyApp::Application::main/); # limits coverage to just listed methods
Devel::Cover->covers_reset(); # resets the full list of "coverable" items.
Devel::Cover->covers_ignore_packages(qw/MyApp::Application::Untestable/); # Any packages listed here will be ignored during coverage recording. Takes precedent over covers_packages
Devel::Cover->covers_ignore_functions(qw/MyApp::Application::SubModule::UntestableMethod/); # Any methods listed here will be ignored recording. Takes precedent over covers_functions. If a package is listed in "covers_packages" but then a method in it is marked as a test as "covers_ignore_function", then that method will be excluded.

If any covers_* restrictions are in place, then nothing is eligible for coverage unless specified.

The limits stay in place from the moment they are called until they are reset, until a new test file is loaded or they are removed (ideally, they should be restricted to the BLOCK they are defined in, but this would be a big chunk of work I feel).

Proposed example usage

package MyApp::Maintest;
use Test2::V0;
use MyApp::Application;
use MyApp::Application::Deeper;
use Devel::Cover; 

Devel::Cover->covers_functions(qw/MyApp::Application::something/);
ok(MyApp::Application->something(),'Should be included in coverage report');
ok(MyApp::Application->otherthing(),'Should not be included');

Devel::Cover->covers_reset(); 

ok(MyApp::Application::Deeper-new(),'Should be included as no restrictions at all in place');
done_testing();

would only record any action taken within the MyApp::Application::something method, even if it called something else - unless it was then called by MyApp::Application::Deeper-new() .

Proposed example implementation

I see it working similar to (if I'm reading the code correctly)

• The above methods will set variables within Devel::Cover
• when get_cover is called, (within the $start block), it then checks these variables and if there is anything listed in the variables it sets a flag of "no cover" unless the $Sub_name or $File is within the functions/packages listed. If the "no cover" flag is set, then the set_subroutine/add_subroutine_cover sections are not called.

However, I'm just really coming back to Perl after 15years or so so it is quite possible I am misreading how Devel::Cover actually works.

In other languages

PHP using PHPUnit

PHPUnit (the "main" PHP Unit Testing toolkit) has the ability to indicate what classes (packages) and/or methods are intended to be covered by a single test (anything called outside that is isn't counted towards the coverage). This helps ensure that the test coverage actually reflects what you intend to be testing.

For example in PHPUnit 10.2 using PHP8's attribute sysytem:

#[CoversClass(Invoice::class)]
#[UsesClass(Money::class)]
final class InvoiceTest extends TestCase

indicates that the class "InvoiceTest" will cover "Invoice::class" (see the Code Coverage Attributes Appendix for details of "CoversClass", "CoversFunction", "CoversNothing") and that the code will also use the Money::class class (this is an optional extra to prevent unintentionally covering/running code).

Previous versions of PHPUnit only supported comment based "DocBlock" annotations (as PHP before v8 didn't support attributes) so code utilising older versions such as PHPUnit 9.6 will have examples such as:

/**
 * @covers \Invoice
 * @uses \Money
 */
final class InvoiceTest extends TestCase

which does the same thing as above, but it also has examples for the "method/function" level checks:

<?php
use PHPUnit\Framework\TestCase;

final class BankAccountTest extends TestCase
{

Thanks

Do you think this could be possible? Thank you for considering it either way ;)

[pjcj]

Thanks for thinking about this and writing it up nicely.

I think we have the packages part covered. Well, we do at a file level which is similar and often the same.

Just trying to think about how this might be implemented and there are a couple of options. We could filter at collection time or at report time. Collection time would be nicer, but I worry that the cost might be quite expensive. But report time might be difficult.

Potentially a solution here could interact with #331