Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix#54231
Draft
MichaelSimons wants to merge 9 commits into
Draft
Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix#54231MichaelSimons wants to merge 9 commits into
MichaelSimons wants to merge 9 commits into
Conversation
Add Stopwatch-based timing around each phase of the format verifier: - SDK build (build.cmd -c Release) - Git clone (init + fetch + checkout) - Repo restore (Build.ps1 -restore) - Solution restore (dotnet restore) - Format workspace run - Format folder run This is a no-op change to collect baseline timings for CI optimization analysis. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…elix Replace the YAML/PowerShell format verification pipeline legs with XUnit integration tests that run on Helix. Each repo gets its own test class (for automatic Helix sharding via PartitionByClass), and the pre-built SDK from Helix is used directly — eliminating ~50 min of redundant SDK builds per CI run and ~99 min of machine time per PR. - Add test/dotnet-format.IntegrationTests/ with base class pattern - Add PartitionByClass support to AssemblyScheduler for dynamic sharding - Add PR vs CI filtering via DOTNET_SDK_PR_VALIDATION env var - Remove eng/dotnet-format/ infrastructure and pipeline references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
@mwiemer-microsoft, @phil-allen-msft - I had copilot convert the dotnet-format integration tests to helix in order to save compute as well as make the tests just list any other test. As part of this work, we only run a subset of the tests in PR (as discussed with Phil earlier). This is WIP but wanted to share to get your early feedback. |
These tests were previously excluded from CI (never ran in Helix). Now that they are included, skip the 3 that have pre-existing failures until they can be investigated and fixed. - TestUseVarCodeStyle_AppliesWhenNotUsingVar (Roslyn IDE0007 behavior change) - FSharpProjectsDoNotCreateException (exit code changed from 1 to 0) - CSharpTemplateProject_LoadWithNoDiagnostics(blazorwasm) (CS0012 missing assembly) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…TypesTests Pass TestContext.Current.CancellationToken to RunAsync() calls in CA1033InterfaceWithDefaultImplementationsCSharpAsync and CA1033InterfaceWithExpressionBodiedDefaultImplementationsCSharpAsync test methods to fix xUnit1051 errors introduced by dotnet#50339. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ConditionalFactAttribute in xUnit v3 made Skip obsolete with error: true, requiring AlwaysSkip for unconditional skips. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Target repos have known vulnerable transitive dependencies that cause NU1902/NU1903 warnings-as-errors during restore. These are irrelevant for format verification, so disable NuGetAudit during restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The tests mutate Environment.CurrentDirectory (process-global shared state) which causes intermittent failures when tests run in parallel. Disable parallelization to prevent race conditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Arcade-based repos have global.json pinning a specific SDK version that isn't available on Helix agents. Use the repo's own build script (eng/build.sh or eng/Build.ps1) to install the correct SDK and perform restore, matching the original pipeline behavior. Non-Arcade repos (e.g. project-system) override UseRepoBuildScript=false to use dotnet restore directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Converts the dotnet-format integration test pipeline legs (YAML + PowerShell) into XUnit integration tests that run on Helix. This eliminates redundant SDK builds and reduces PR validation cost.
Time Savings
Measured from build 1412916 with timing instrumentation:
In PR validation: ~107 minutes of machine time saved.
In CI (non-PR): ~49 minutes of SDK build time eliminated.
What changed
test/dotnet-format.IntegrationTests/with one test class per repo (SDK, Roslyn, MSBuild, ProjectSystem, AspNetCore, EfCore)FormatIntegrationTestBasehandles clone, restore, and format execution. Adding or removing a repo requires only adding or deleting a ~20-line class.PartitionByClass="true"metadata causesAssemblySchedulerto discover test classes at scheduling time and create one Helix work item per class automatically.DOTNET_SDK_PR_VALIDATIONenv var andRunInPRValidationproperty control which tests run in PR validation. This infrastructure is extensible — any future test class can opt into or out of PR runs by setting this property, enabling broader use of PR-conditional test filtering beyond format tests.eng/dotnet-format/(YAML template, PowerShell verifier, CMD wrapper, RSP file) and removed pipeline references from.vsts-ci.yml/.vsts-pr.yml.Design
Each repo is a separate test class inheriting from
FormatIntegrationTestBase:This one-class-per-repo pattern enables:
RunInPRValidationpattern can be adopted by other test classes in the future to conditionally exclude expensive tests from PR validation