Fix junitxml count for teardown errors#14501
Conversation
nicoddemus
left a comment
There was a problem hiding this comment.
Thanks, left some comments!
| fnode.assert_attr(message='failed on teardown with "ValueError: Error reason"') | ||
| assert "ValueError" in fnode.toxml() | ||
|
|
||
| def test_teardown_error_after_pass_counts_as_one_test( |
There was a problem hiding this comment.
I believe we should convert these tests to use run_and_parse instead of calling the hooks manually.
There was a problem hiding this comment.
We should assert that the tests="X" header item matches the number of <testcase> items in the XML.
There was a problem hiding this comment.
Given this should be an invariant, I applied this diff to run_and_parse in main:
diff --git a/testing/test_junitxml.py b/testing/test_junitxml.py
index 5a603c05b..f043a002f 100644
--- a/testing/test_junitxml.py
+++ b/testing/test_junitxml.py
@@ -49,7 +49,11 @@ class RunAndParse:
with xml_path.open(encoding="utf-8") as f:
self.schema.validate(f)
xmldoc = minidom.parse(str(xml_path))
- return result, DomDocument(xmldoc)
+ doc = DomDocument(xmldoc)
+ testcase_nodes = doc.find_by_tag("testcase")
+ test_suite_node = doc.get_first_by_tag("testsuite")
+ test_suite_node.assert_attr(name="pytest", tests=len(testcase_nodes))
+ return result, docAnd that alone caused 9 out of 137 tests to fail already. We might not even need additional tests, an existing test might already be catching the original issue, it wasn't testing the tests header against the number of testcases in the file.
| # schema. | ||
| self.finalize(close_report) | ||
| self.cnt_double_fail_tests += 1 | ||
| elif ( |
There was a problem hiding this comment.
Please add a comment explanation this case, possibly with a link to the issue.
Use run_and_parse for the teardown regression and assert the testsuite count matches emitted testcase nodes. Keep the teardown-only adjustment only for the pass-plus-teardown-error case where both reports share a testcase node. Co-authored-by: OpenAI GPT-5 <noreply@openai.com>
2a63e72 to
549f134
Compare
|
Rebased onto current Verification run locally: uv run --with-editable . --extra dev python -m pytest testing/test_junitxml.py testing/test_unittest.py::test_setup_inheritance_skipping testing/test_unittest.py::test_pdb_teardown_skipped_for_classes testing/test_debugging.py::TestPDB::test_pdb_unittest_skip -q --junitxml=/tmp/pytest-14501-after-rebase.xml
# 142 passed, 3 skipped |
Fixes #3850
Summary
testscount when a test passes during call but fails during teardownrun_and_parsethat the testsuitetestsattribute matches the emitted<testcase>nodesTests
.venv/bin/python -m pytest testing/test_junitxml.py -q.venv/bin/pre-commit run --color never --files src/_pytest/junitxml.py testing/test_junitxml.py changelog/3850.bugfix.rstgit diff --checkAI assistance was used while iterating on the patch and tests. I reviewed the change and can explain it.