Support "expected failures"

[jimporter]

It'd be nice to be able to support expected failures so that it's easy to keep your tests green even when something isn't quite finished (e.g. you haven't ported feature X to platform Y).

[darenw]

This has my vote! For now, I just hack in #if 0 / #endif around works in progress.

[jimporter]

I'm open to suggestions on how the API should look. So far, I have three possibilities:

1. Add a new .xfail_test member function that works like .test except that it expects failure; likewise an xfail_suite (and subsuite).
2. Add a new attribute called xfail that has some special handling for it. I like how this would look the most, but it would probably require more work. I'm also not sure how this would interact with the attribute filters on the command line; should you be able to add/remove the xfail status for certain attributes?
3. Add an all-new argument to tests/suites that work sort of like attributes. This gets some of the benefits of (2), but it's a little simpler. However, it would add even more overloads to the various functions, and they're a little hairy as it is.

One nice thing about (2) and (3) is that it'd be easy to decide at runtime whether to xfail a test in the suite builder (e.g. based on the current environment variables).

I'm sort of leaning towards (2) right now, provided I can decide what to do with filters. Maybe add all-new options that say "xfail tests with these attributes" and "don't xfail tests with these other attributes"? Maybe nothing for now?

[jimporter]

This has my vote! For now, I just hack in #if 0 / #endif around works in progress.

You can also use the skip attribute in this case. It'll skip the test unless you explicitly run tests with the skip attribute. Here's an example, which I added to skip a test that fails on OS X because of a buggy pselect(2) implementation:

mettle/test/posix/test_subprocess.cpp

Lines 83 to 88 in 0952f37

	attributes sigtest_attrs;
	#ifdef __APPLE__
	sigtest_attrs.insert(skip("pselect(2) is buggy on OS X"));
	#endif
	_.test("read until signal", sigtest_attrs, [](read_into_fixture &f) {

[jimporter]

@darenw What are your opinions on what should happen if an xfailed test actually passes? Should that be treated as ok, or should it report an error?

[darenw]

I'd assume such a test is a work in progress, or in need of debugging (tomorrow maybe), or is testing a part of my project which is a work in progress or needing debugging, so the test passing or not passing is pure accident.

If I had a specific reason to expect a test to always fail until something is fixed, that's just a regular ol' fail. But if it's a nuisance, two things I'd think of doing: 1) hack in a logical negation in the test, and hope to remember it's there later, or 2) disable or comment out the test for now, and hope to remember to restore it later. You can guess what the problem is with my approach.

In an ideal world, I'd make the test "temporarily disabled" and have it show not red or green, but violet or brown or gray or something. Then it's visible, and can't be mistaken for a 'pass', and can't hold balk up the assembly line, so to speak, for coming up 'fail'. But I wouldn't be interested in verifying that the unmodified test always come up "fail", in any case.

[jimporter]

Hmm. On the one hand, the benefit of reporting an error if an xfailed test passes is that it alerts you to the fact that you fixed the test. However, that doesn't help if the test is intermittently failing (e.g. because of a race condition).

In an ideal world, I'd make the test "temporarily disabled" and have it show not red or green, but violet or brown or gray or something.

This is actually very close to how skipped tests work now. If you skip a test, it shows up as blue and is reported at the end alongside any failures. However, skipped tests don't cause the whole run to fail. For instance, see the OSX log from Travis.

If I do support xfailed tests, I want them to be sufficiently different from skipped tests that there's a reason for both to exist. Maybe just having xfailed tests get run (and not caring about the result) is enough of a difference from skipped tests?