Investigation of test results in XML vs Console

[im-konge]

Signed-off-by: Lukas Kral [email protected]

This is more discussion or issue description than real PR with some changes.

How you can see in this code, I changed a little bit asserts in rerun suite. The main problem that I found is (from my POV) wrong counting of tests count. Or not displaying the real tests run count.

When we take a suite (class) where we have 3 tests, that 2 of them will run twice (one rerun), the count will be like this (how is it in current tests) : Tests run: 5, errors: 1, failures: 1, skipped: 0 ... -> but this is not really the truth. It can be problem in translation, but I would like to know how many tests (not with reruns) I ran -> 3. Then if failures > 0 the test should be flaky etc.

So I assume to add something like tests count: 3, tests run: 5 ... to the console reporter and tests: 3, runs: 5 to the XML reporter.

Second problem that I found is that results from XML file != results from console reporter. I have some examples when I was talking to @Tibor17.

WDYT?

[Tibor17]

Hi @qingzhouluo
Hi @agudian
You are the authors of the feature re-run. Can you please comment on @im-konge 's request?
How did you manage to count the test in case of re-run was enabled?
Is it necessary to include the flakes in "Tests run:" ?

[qingzhouluo]

We include the "number of run" information to give users some ideas about how many total runs there are, so they can better understand the flakiness of their test suite. For example, if I have one single test run 100 times, and it failed once, "test run: 100, failure: 1" will help me to understand how serve the issue is.

Adding some extra information about the total number of test count also sounds good to me, thanks.

[michalxo]

Hello! Is there any progress on this issue please?
We're experiencing the same problem with re-run feature.
Thank you

I have opened https://issues.apache.org/jira/browse/SUREFIRE-1903