Repository navigation
[CI] full-build.yml doesn't fail when many tests are skipped #5690
Description
Description
The retired Jenkins osx and windows pipelines failed the build when more than 15 tests were skipped (xunit skipped(failureThreshold: '15') in cbci_jenkins_libs vars/openstudio_incremental_develop_osx.groovy:427 and _windows.groovy:394). full-build.yml has no equivalent, and CTest counts skipped tests as passing.
If a change makes a fixture skip widely (for example, tests that skip when Python or Ruby isn't found on macOS), the Full Build job stays green where Jenkins would have failed it. Since #5681, Full Build is the only macOS/Windows test gate on PRs.
Possible fix
After the CTest step, count Status="notrun" / skipped entries in the Testing/<tag>/Test.xml file the step already writes (-T test), and fail above a threshold. Starting at 15 matches Jenkins; checking what the current per-platform skip count is first would give a better number.
Found in the review of #5681.
🤖 Generated with Claude Code
Activity
anchapin commentedon Nov 10, 2026
Measured on last night's successful develop run (42733776935, all six platforms green), from the
Testing-*artifacts:Skip count today: 10 on every platform where the full results survived (macOS x64, macOS arm64, Ubuntu 24.04 ARM64). All 10 are
Disabled(gtestDISABLED_tests), and they're the same 10 everywhere:EnergyPlusFixture.ForwardTranslator_ScheduleFixedInterval_TwoPoint,..._ScheduleVariableInterval_500_Shifted,..._ScheduleVariableInterval_DaysTimeSeries,..._ScheduleVariableInterval_Hourly_Shifted,EnergyPlusFixture.ReverseTranslator_RunPeriodControlSpecialDays,EnergyPlusFixture.ReverseTranslator_SizingSystem,ISOModelFixture.ForwardTranslator,ModelFixture.Issue_1322,ModelFixture.Issue_1683,ModelFixture.ShatteredModel_Existing_3424The only runtime skip mechanism is
SKIP_RETURN_CODE 5on pytest tests (python/Pytest.cmake:127, pytest's "no tests collected"). So Jenkins' threshold of 15 leaves room for 5 runtime skips; a check that counts only non-Disablednotrun tests could use a much tighter threshold (even 0).The check can't just read
Test.xml. Eachctest -T testcall in the CTest step writes to the sameTesting/<tag>/Test.xml, so the artifact only holds the last call:Test.xmlcontains one test,OpenStudioCLI.test_measure_manager, which means it failed in the parallel run and the--rerun-failedcall overwrote the full results. (That test flaked on three platforms in one night, which may deserve its own issue.)LastTest.log, which the test-summary step tails, has the same problem.So the skip check needs each ctest call to write its own results, e.g.
--output-junit <group>.xmlper call (CMake ≥ 3.21), then count across the files. That would also fix the artifacts' incomplete results.