Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/filtering-execution-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,22 @@ behaviour, which does not affect generators at all. If you also use either
`-g`/`--generator-index`, or `-p`/`--path-filter`, you will get the new
behaviour, which can also filter generator elements.

When path filters are in effect, the console and compact reporters print
the active selection at the start of the run, next to the `Filters:` line,
e.g.

```text
Filters: "foo"
Path filters:
- Section: "A"
- Generator: "0"
```

The filters are listed one per line, labelled as `Section` or `Generator`,
in the order in which they were specified. This makes it easy to see exactly
what has been selected when a filter combination ends up running no
assertions.

Both the new and old filter behaviours include some potentially surprising
things:
* Code outside of sections being skipped will still be executed. E.g.
Expand Down
6 changes: 6 additions & 0 deletions src/catch2/reporters/catch_reporter_compact.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,12 @@ class AssertionPrinter {
<< m_config->testSpec()
<< '\n';
}
if ( !m_config->getPathFilters().empty() ) {
m_stream << m_colour->guardColour( Colour::BrightYellow )
<< "Path filters:\n"
<< serializePathFilters( m_config->getPathFilters() )
<< '\n';
}
m_stream << "RNG seed: " << getSeed() << '\n'
<< std::flush;
}
Expand Down
5 changes: 5 additions & 0 deletions src/catch2/reporters/catch_reporter_console.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,11 @@ void ConsoleReporter::testRunStarting(TestRunInfo const& _testRunInfo) {
m_stream << m_colour->guardColour( Colour::BrightYellow ) << "Filters: "
<< m_config->testSpec() << '\n';
}
if ( !m_config->getPathFilters().empty() ) {
m_stream << m_colour->guardColour( Colour::BrightYellow )
<< "Path filters:\n"
<< serializePathFilters( m_config->getPathFilters() ) << '\n';
}
m_stream << "Randomness seeded to: " << getSeed() << '\n'
<< std::flush;
}
Expand Down
23 changes: 23 additions & 0 deletions src/catch2/reporters/catch_reporter_helpers.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
#include <catch2/reporters/catch_reporter_helpers.hpp>
#include <catch2/interfaces/catch_interfaces_config.hpp>
#include <catch2/internal/catch_console_width.hpp>
#include <catch2/internal/catch_path_filter.hpp>
#include <catch2/internal/catch_errno_guard.hpp>
#include <catch2/internal/catch_textflow.hpp>
#include <catch2/internal/catch_reusable_string_stream.hpp>
Expand Down Expand Up @@ -101,6 +102,28 @@ namespace Catch {
return serialized;
}

std::string serializePathFilters( std::vector<PathFilter> const& filters ) {
ReusableStringStream rss;
bool first = true;
for ( auto const& filter : filters ) {
if ( !first ) {
rss << '\n';
}
first = false;
rss << " - ";
switch ( filter.type ) {
case PathFilter::For::Section:
rss << "Section: ";
break;
case PathFilter::For::Generator:
rss << "Generator: ";
break;
}
rss << '"' << filter.filter << '"';
}
return rss.str();
}

std::ostream& operator<<( std::ostream& out, lineOfChars value ) {
for ( size_t idx = 0; idx < CATCH_CONFIG_CONSOLE_WIDTH - 1; ++idx ) {
out.put( value.c );
Expand Down
13 changes: 13 additions & 0 deletions src/catch2/reporters/catch_reporter_helpers.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ namespace Catch {
class IConfig;
class TestCaseHandle;
class ColourImpl;
struct PathFilter;

// Returns double formatted as %.3f (format expected on output)
std::string getFormattedDuration( double duration );
Expand All @@ -30,6 +31,18 @@ namespace Catch {

std::string serializeFilters( std::vector<std::string> const& filters );

/**
* Serializes the active section/generator path filters into a
* human-readable, indented list, one filter per line and in the order
* in which they were specified, e.g.
*
* - Section: "A"
* - Generator: "0"
*
* The returned string has no trailing newline.
*/
std::string serializePathFilters( std::vector<PathFilter> const& filters );

struct lineOfChars {
char c;
constexpr lineOfChars( char c_ ): c( c_ ) {}
Expand Down
27 changes: 27 additions & 0 deletions tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -666,6 +666,33 @@ foreach(reporterName # "Automake" - the simple .trs format does not support any
)
endforeach()

# The console and compact reporters print the active section/generator
# selection (path filters), so that it is obvious what has been selected.
# The exact serialization is unit-tested in `serializePathFilters ...`, so
# here we only check that both reporters actually emit the line, using a
# section filter for one and a generator filter for the other.
add_test(NAME "Reporters:PathFilters:Section:compact"
COMMAND
$<TARGET_FILE:SelfTest> "Tracker"
--reporter compact
--section "section name"
)
set_tests_properties("Reporters:PathFilters:Section:compact"
PROPERTIES
PASS_REGULAR_EXPRESSION "- Section: \"section name\""
)

add_test(NAME "Reporters:PathFilters:Generator:console"
COMMAND
$<TARGET_FILE:SelfTest> "Generators internals"
--reporter console
--generator-index 0
)
set_tests_properties("Reporters:PathFilters:Generator:console"
PROPERTIES
PASS_REGULAR_EXPRESSION "- Generator: \"0\""
)

add_test(NAME "Bazel::RngSeedEnvVar::JustEnv"
COMMAND
$<TARGET_FILE:SelfTest> "Factorials are computed"
Expand Down
1 change: 1 addition & 0 deletions tests/SelfTest/Baselines/automake.sw.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,7 @@ b1!
:test-result: SKIP sections can be skipped dynamically at runtime
:test-result: FAIL send a single char to INFO
:test-result: FAIL sends information to INFO
:test-result: PASS serializePathFilters serializes the section/generator selection
:test-result: PASS shortened hide tags are split apart
:test-result: SKIP skipped tests can optionally provide a reason
:test-result: PASS splitString
Expand Down
1 change: 1 addition & 0 deletions tests/SelfTest/Baselines/automake.sw.multi.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,7 @@
:test-result: SKIP sections can be skipped dynamically at runtime
:test-result: FAIL send a single char to INFO
:test-result: FAIL sends information to INFO
:test-result: PASS serializePathFilters serializes the section/generator selection
:test-result: PASS shortened hide tags are split apart
:test-result: SKIP skipped tests can optionally provide a reason
:test-result: PASS splitString
Expand Down
18 changes: 16 additions & 2 deletions tests/SelfTest/Baselines/compact.sw.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2831,6 +2831,20 @@ Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: failed: false with 1 message: '3'
Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7'
Reporters.tests.cpp:<line number>: passed: serializePathFilters( {} ).empty() for: true
Reporters.tests.cpp:<line number>: passed: serializePathFilters( filters ) == " - Section: \"a section\"" for: " - Section: "a section""
==
" - Section: "a section""
Reporters.tests.cpp:<line number>: passed: serializePathFilters( filters ) == " - Generator: \"0\"" for: " - Generator: "0""
==
" - Generator: "0""
Reporters.tests.cpp:<line number>: passed: serializePathFilters( filters ) == " - Section: \"A\"\n" " - Generator: \"1\"\n" " - Section: \"B\"" for: " - Section: "A"
- Generator: "1"
- Section: "B""
==
" - Section: "A"
- Generator: "1"
- Section: "B""
Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
Expand Down Expand Up @@ -3000,7 +3014,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 452 | 332 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2420 | 2219 passed | 158 failed | 43 failed as expected


18 changes: 16 additions & 2 deletions tests/SelfTest/Baselines/compact.sw.multi.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2820,6 +2820,20 @@ Skip.tests.cpp:<line number>: skipped:
Skip.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: failed: false with 1 message: '3'
Message.tests.cpp:<line number>: failed: false with 2 messages: 'hi' and 'i := 7'
Reporters.tests.cpp:<line number>: passed: serializePathFilters( {} ).empty() for: true
Reporters.tests.cpp:<line number>: passed: serializePathFilters( filters ) == " - Section: \"a section\"" for: " - Section: "a section""
==
" - Section: "a section""
Reporters.tests.cpp:<line number>: passed: serializePathFilters( filters ) == " - Generator: \"0\"" for: " - Generator: "0""
==
" - Generator: "0""
Reporters.tests.cpp:<line number>: passed: serializePathFilters( filters ) == " - Section: \"A\"\n" " - Generator: \"1\"\n" " - Section: \"B\"" for: " - Section: "A"
- Generator: "1"
- Section: "B""
==
" - Section: "A"
- Generator: "1"
- Section: "B""
Tag.tests.cpp:<line number>: passed: testcase.tags, VectorContains( Tag( "magic-tag" ) ) && VectorContains( Tag( "."_catch_sr ) ) for: { {?}, {?} } ( Contains: {?} and Contains: {?} )
Skip.tests.cpp:<line number>: skipped: 'skipping because answer = 43'
StringManip.tests.cpp:<line number>: passed: splitStringRef("", ','), Equals(std::vector<StringRef>()) for: { } Equals: { }
Expand Down Expand Up @@ -2989,7 +3003,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 452 | 332 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2420 | 2219 passed | 158 failed | 43 failed as expected


4 changes: 2 additions & 2 deletions tests/SelfTest/Baselines/console.std.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1743,6 +1743,6 @@ due to unexpected exception with message:
Why would you throw a std::string?

===============================================================================
test cases: 451 | 349 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2394 | 2215 passed | 136 failed | 43 failed as expected
test cases: 452 | 350 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2398 | 2219 passed | 136 failed | 43 failed as expected

62 changes: 60 additions & 2 deletions tests/SelfTest/Baselines/console.sw.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18942,6 +18942,64 @@ with messages:
hi
i := 7

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
No filters serialize to an empty string
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( {} ).empty() )
with expansion:
true

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
Section filters are labelled and quoted
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( filters ) == " - Section: \"a section\"" )
with expansion:
" - Section: "a section""
==
" - Section: "a section""

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
Generator filters are labelled and quoted
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( filters ) == " - Generator: \"0\"" )
with expansion:
" - Generator: "0""
==
" - Generator: "0""

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
Multiple filters keep their order, one per line
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( filters ) == " - Section: \"A\"\n" " - Generator: \"1\"\n" " - Section: \"B\"" )
with expansion:
" - Section: "A"
- Generator: "1"
- Section: "B""
==
" - Section: "A"
- Generator: "1"
- Section: "B""

-------------------------------------------------------------------------------
shortened hide tags are split apart
-------------------------------------------------------------------------------
Expand Down Expand Up @@ -20134,6 +20192,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED:

===============================================================================
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 452 | 332 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2420 | 2219 passed | 158 failed | 43 failed as expected

62 changes: 60 additions & 2 deletions tests/SelfTest/Baselines/console.sw.multi.approved.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18931,6 +18931,64 @@ with messages:
hi
i := 7

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
No filters serialize to an empty string
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( {} ).empty() )
with expansion:
true

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
Section filters are labelled and quoted
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( filters ) == " - Section: \"a section\"" )
with expansion:
" - Section: "a section""
==
" - Section: "a section""

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
Generator filters are labelled and quoted
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( filters ) == " - Generator: \"0\"" )
with expansion:
" - Generator: "0""
==
" - Generator: "0""

-------------------------------------------------------------------------------
serializePathFilters serializes the section/generator selection
Multiple filters keep their order, one per line
-------------------------------------------------------------------------------
Reporters.tests.cpp:<line number>
...............................................................................

Reporters.tests.cpp:<line number>: PASSED:
REQUIRE( serializePathFilters( filters ) == " - Section: \"A\"\n" " - Generator: \"1\"\n" " - Section: \"B\"" )
with expansion:
" - Section: "A"
- Generator: "1"
- Section: "B""
==
" - Section: "A"
- Generator: "1"
- Section: "B""

-------------------------------------------------------------------------------
shortened hide tags are split apart
-------------------------------------------------------------------------------
Expand Down Expand Up @@ -20123,6 +20181,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED:

===============================================================================
test cases: 451 | 331 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2416 | 2215 passed | 158 failed | 43 failed as expected
test cases: 452 | 332 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2420 | 2219 passed | 158 failed | 43 failed as expected

Loading