Phase 1: Test runner consolidation on Microsoft.Testing.Platform#16998
Draft
Evangelink wants to merge 1 commit into
Draft
Phase 1: Test runner consolidation on Microsoft.Testing.Platform#16998Evangelink wants to merge 1 commit into
Evangelink wants to merge 1 commit into
Conversation
Documents the staged plan to retire Arcade's parallel test-runner code
paths in favour of Microsoft.Testing.Platform (MTP) and dotnet test.
This first PR makes no behavior changes for repos that do not opt into
the deprecated paths. It adds:
* Documentation/TestRunnerConsolidation.md - full 5-phase plan,
rationale, and impact estimate.
* MSBuild deprecation warnings (suppressible) for:
- ARCADE0001: explicit UsingToolXUnit (already documented as
deprecated in DefaultVersions.props but never warned).
- ARCADE0002: UseVSTestRunner=true.
- ARCADE0010: XUnitProject items in Helix projects (use MTPProject
instead).
* Deletes the orphaned tools/xunit-reporter/xunit-reporter.py Helix
script which has self-marked itself deprecated since 2021 and is no
longer invoked from any Arcade target.
Defaults flips (Phase 2) and code deletions (Phases 3-5) will follow in
separate PRs after arcade-validation runs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR starts “Phase 1” of consolidating Arcade’s test-runner infrastructure around Microsoft.Testing.Platform by documenting the multi-phase plan, emitting opt-in deprecation warnings for legacy runner switches, and removing an unreferenced legacy Helix reporter script.
Changes:
- Add a detailed consolidation plan document (
Documentation/TestRunnerConsolidation.md). - Emit new MSBuild deprecation warnings (
ARCADE0001,ARCADE0002,ARCADE0010) when deprecated test-runner properties/items are used. - Delete the deprecated/unreferenced
xunit-reporter.pyHelix script.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.DotNet.Helix/Sdk/tools/xunit-runner/XUnitRunner.targets | Adds ARCADE0010 warning when XUnitProject is used. |
| src/Microsoft.DotNet.Helix/Sdk/tools/xunit-reporter/xunit-reporter.py | Removes deprecated, apparently orphaned reporter script. |
| src/Microsoft.DotNet.Arcade.Sdk/tools/Tests.targets | Adds ARCADE0001/ARCADE0002 warnings for deprecated runner options. |
| src/Microsoft.DotNet.Arcade.Sdk/tools/DefaultVersions.props | Adds a flag intended to detect explicit UsingToolXUnit usage. |
| Documentation/TestRunnerConsolidation.md | New documentation describing the rationale and phased plan. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 5/5 changed files
- Comments generated: 3
Comment on lines
+42
to
+44
| <Warning Code="ARCADE0001" | ||
| Text="The 'UsingToolXUnit' property is deprecated and will be removed in a future Arcade release. Use the 'TestRunnerName' property (e.g. TestRunnerName=XUnitV3) to select a test runner. See Documentation/TestRunnerConsolidation.md." | ||
| Condition="'$(_UsingToolXUnitExplicitlySet)' == 'true'" /> |
Comment on lines
+34
to
+38
| We warn when a project opts into a code path that is scheduled for | ||
| removal so that downstream repos can migrate before the next major | ||
| Arcade release. Suppress by setting: | ||
| <NoWarn>$(NoWarn);ARCADE0001;ARCADE0002</NoWarn> | ||
| --> |
|
|
||
| ## Plan | ||
|
|
||
| This is staged so each phase is independently revertable. |
Member
|
I absolutely agree with this direction. We should keep the Arcade SDK infra current and not legacy. If that means doing some manual work in consuming repos then that's fine. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Opens the discussion proposed in follow-up to a maintainer conversation on consolidating Arcade's test-runner infrastructure on top of Microsoft.Testing.Platform (MTP) and the standard
dotnet testorchestration. With xUnit v3 (MTP by default), MSTest 4.x (MTP by default), NUnit/TUnit MTP runners,dotnet test --use-testing-platform, and VSTest in maintenance mode, most of Arcade's bespoke runner targets are now redundant.Today the Arcade SDK ships five parallel runner targets files (
XUnit,XUnitV3,MSTest,VSTest,Microsoft.Testing.Platform) and the Helix SDK ships two parallel work-item generators (CreateXUnitWorkItems+xunit-runner/payload,CreateMTPWorkItems+mtp-runner/). The MTP path covers everything the legacy paths do; once downstream repos migrate, ~1000 LoC and ~8 documented public properties can be deleted with no functional loss.What this PR does (Phase 1 — signal direction, zero behavior change for migrated repos)
Documentation/TestRunnerConsolidation.mdwith the full six-phase plan, rationale and impact estimate.ARCADE0001— explicitUsingToolXUnit(property is already documented as deprecated inDefaultVersions.propsbut never warned).ARCADE0002—UseVSTestRunner=true.ARCADE0010—XUnitProjectitems in Helix projects (useMTPProjectinstead).src/Microsoft.DotNet.Helix/Sdk/tools/xunit-reporter/xunit-reporter.pyscript, which has been self-marked deprecated since 2021 and is not referenced from any current Arcade target (verified withgit grep).No behavior change for any repo that does not already set the deprecated properties. The defaults flip (Phase 2) and the actual file deletions / target collapses (Phases 3–6) will be separate PRs gated on arcade-validation.
Out of scope
The pieces of Arcade test infra that are not replaceable by
dotnet testare explicitly preserved:SendHelixJob, queue selection, correlation payloads,EnableHelixJobMonitor).test-configuration.json.Validation
.props/.targetsfiles verified.SuppressArcadeTestRunnerDeprecationWarnings=true(SDK side) andSuppressArcadeXUnitProjectDeprecationWarning=true(Helix SDK side) so downstream repos can roll out the migration on their own cadence.