Skip to content

Fix junitxml count for teardown errors#14501

Open
puneetdixit200 wants to merge 3 commits into
pytest-dev:mainfrom
puneetdixit200:junitxml-teardown-count
Open

Fix junitxml count for teardown errors#14501
puneetdixit200 wants to merge 3 commits into
pytest-dev:mainfrom
puneetdixit200:junitxml-teardown-count

Conversation

@puneetdixit200
Copy link
Copy Markdown

@puneetdixit200 puneetdixit200 commented May 21, 2026

Fixes #3850

Summary

  • correct the JUnit XML tests count when a test passes during call but fails during teardown
  • assert in run_and_parse that the testsuite tests attribute matches the emitted <testcase> nodes
  • keep call-failure plus teardown-error as separate testcase nodes, while the pass-plus-teardown-error path stays one testcase node
  • add a changelog entry

Tests

  • .venv/bin/python -m pytest testing/test_junitxml.py -q
  • .venv/bin/pre-commit run --color never --files src/_pytest/junitxml.py testing/test_junitxml.py changelog/3850.bugfix.rst
  • git diff --check

AI assistance was used while iterating on the patch and tests. I reviewed the change and can explain it.

@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label May 21, 2026
Copy link
Copy Markdown
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, left some comments!

Comment thread testing/test_junitxml.py
fnode.assert_attr(message='failed on teardown with "ValueError: Error reason"')
assert "ValueError" in fnode.toxml()

def test_teardown_error_after_pass_counts_as_one_test(
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should convert these tests to use run_and_parse instead of calling the hooks manually.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should assert that the tests="X" header item matches the number of <testcase> items in the XML.

Copy link
Copy Markdown
Member

@nicoddemus nicoddemus May 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this should be an invariant, I applied this diff to run_and_parse in main:

diff --git a/testing/test_junitxml.py b/testing/test_junitxml.py
index 5a603c05b..f043a002f 100644
--- a/testing/test_junitxml.py
+++ b/testing/test_junitxml.py
@@ -49,7 +49,11 @@ class RunAndParse:
             with xml_path.open(encoding="utf-8") as f:
                 self.schema.validate(f)
         xmldoc = minidom.parse(str(xml_path))
-        return result, DomDocument(xmldoc)
+        doc = DomDocument(xmldoc)
+        testcase_nodes = doc.find_by_tag("testcase")
+        test_suite_node = doc.get_first_by_tag("testsuite")
+        test_suite_node.assert_attr(name="pytest", tests=len(testcase_nodes))
+        return result, doc

And that alone caused 9 out of 137 tests to fail already. We might not even need additional tests, an existing test might already be catching the original issue, it wasn't testing the tests header against the number of testcases in the file.

Comment thread src/_pytest/junitxml.py
# schema.
self.finalize(close_report)
self.cnt_double_fail_tests += 1
elif (
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment explanation this case, possibly with a link to the issue.

Puneet Dixit and others added 3 commits May 23, 2026 20:25
Use run_and_parse for the teardown regression and assert the testsuite count matches emitted testcase nodes. Keep the teardown-only adjustment only for the pass-plus-teardown-error case where both reports share a testcase node.

Co-authored-by: OpenAI GPT-5 <noreply@openai.com>
@puneetdixit200 puneetdixit200 force-pushed the junitxml-teardown-count branch from 2a63e72 to 549f134 Compare May 23, 2026 14:57
@puneetdixit200
Copy link
Copy Markdown
Author

Rebased onto current main to pick up the unittest skip fixture fix from #14505.

Verification run locally:

uv run --with-editable . --extra dev python -m pytest testing/test_junitxml.py testing/test_unittest.py::test_setup_inheritance_skipping testing/test_unittest.py::test_pdb_teardown_skipped_for_classes testing/test_debugging.py::TestPDB::test_pdb_unittest_skip -q --junitxml=/tmp/pytest-14501-after-rebase.xml
# 142 passed, 3 skipped

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

junit xml "tests" number seems wrong if only teardown fails

2 participants