[Tooling] Test fastlane trainer refactoring for Swift Testing support

[AliSoftware]

Important

Do not merge

This is to test some work I'm doing on fastlane to make trainer support parsing .xcresult bundles that reports test results of tests implemented with Swift Testing (since as of today it seems that trainer only supports test results from XCTests?)

[dangermattic]

	1 Message
📖	This PR is still a Draft: some checks will be skipped.

Generated by 🚫 Danger

[wpmobilebot]

📲 You can test the changes from this Pull Request in WooCommerce iOS by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce iOS
	Build Number	pr15241-6bf7d53
	Version	21.7
	Bundle ID	com.automattic.alpha.woocommerce
	Commit	6bf7d53
	App Center Build	WooCommerce - Prototype Builds #13136

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

[AliSoftware]

cc @mokagio so you dont rush updating all the repos with the trainer workaround we've been trying on WCiOS, given said workaround is likely to not work after all, and might have just returned us to the status quo.

In other words, the workaround might still hide the same failures, especially from tests written in Swift Testing; and the first iteration of the hack looked like it fixed it bc it made the CI go red… but that was misleading (as it was reporting retries)… then the second iteration of the workaround probably returned us to the same behavior as before the hack, since run_tests already uses the Trainer's class implementation internally to report test failure or success, so reading the hash it returns is the same as reading trainer's returned hash already.

Instead, this PR is adopting the implementation of my refactoring of fastlane's trainer to use the new way of parsing the .xcresult files, using the modern Xcode 16+ version of xcresulttool, to support Swift Testing, instead of relying on --legacy like run_tests and trainer are currently doing). This will hopefully be a safer solution by ensuring it detects SwiftTesting failures reported in the .xcresult file and not just XCTest failures. It seems to work with a demo project I've crafted as well as with WCiOS (though I'd love to do more testing on it to account for all possible edge cases people could encounter).

[AliSoftware]

Note that:

• This PR was cut from a version of trunk before the workaround was applied, so it does not rely on it
• The "Unit Tests" job is expected to fail, and report 2 failures:
  • one from test_it_returns_days_if_the_dates_are_between_two_and_seven_days_apart() in AgeTests, an XCTest, run 3 times and failed on each of its retries
  • one from itemsInCartLabel(_:_:) in CartViewHelperTests, which is a @Test(arguments:…) parametrized SwiftTesting test, and is expected to fail (with 3 retries) in only 1 of the 4 different arguments being tested
• Since the run-unit-tests.sh script uses annotate_test_failures, those 2 errors are expected to be reported in Buildkite Annotation¹. Though since annotate_test_failures was not implemented with SwiftTesting arguments feature in mind, it will consider the test to be flaky (because it failed 1 run but passed the other 3, and it didn't distinguish tests with different argument properties²).
• Buildkite Test Analytics will still only see 1 failure out of the 2 tests, because Buildkite's TestCollector Swift package only supports XCTest but not SwiftTesting

Footnotes

1. That made me remember why we used trainer in the first place after all, it was probably for this annotation ↩

2. That makes me think that maybe I should include the name of the arguments in parentheses in the test name when I make trainer generate the XML, so most tools reading the JUnit file (be it our annotate_test_failures or Buildkite's TestEngine if we upload the XML to it instead of using their TestCollector package) would consider them different test cases… ↩

[AliSoftware]

Update: I made the fix in trainer to generate different <testcase name=…> for test case runs that took different arguments, and now they're treated properly by annotate_test_failure (and so should they by other tools like TestAnalytics too)

[AliSoftware]

Created the PR in fastlane to propose my refactoring of trainer supporting SwiftTesting for review: fastlane/fastlane#29463