Skip to content

Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix#54231

Draft
MichaelSimons wants to merge 9 commits into
dotnet:mainfrom
MichaelSimons:format-timings
Draft

Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix#54231
MichaelSimons wants to merge 9 commits into
dotnet:mainfrom
MichaelSimons:format-timings

Conversation

@MichaelSimons
Copy link
Copy Markdown
Member

@MichaelSimons MichaelSimons commented May 7, 2026

Summary

Converts the dotnet-format integration test pipeline legs (YAML + PowerShell) into XUnit integration tests that run on Helix. This eliminates redundant SDK builds and reduces PR validation cost.

Time Savings

Measured from build 1412916 with timing instrumentation:

Leg Total Leg Time SDK Build Time (eliminated)
sdk 19m 03s 8m 02s
Roslyn 28m 50s 8m 26s
project-system 12m 31s 8m 43s
msbuild 17m 02s 8m 25s
efcore 15m 34s 7m 19s
aspnetcore 25m 34s 8m 27s
Total ~118 min ~49 min

In PR validation: ~107 minutes of machine time saved.

  • Previously all 6 legs ran in PR (~118 min total machine time, ~29 min critical path)
  • Now only the SDK leg runs in PR, and without the SDK build (~19m - 8m = ~11m)
  • Savings: ~99 min from dropping 5 legs + ~8 min from eliminating the SDK build in the remaining leg

In CI (non-PR): ~49 minutes of SDK build time eliminated.

  • All 6 legs still run but use the pre-built SDK already deployed to Helix instead of rebuilding it each time

What changed

  • New test project: test/dotnet-format.IntegrationTests/ with one test class per repo (SDK, Roslyn, MSBuild, ProjectSystem, AspNetCore, EfCore)
  • Base class pattern: FormatIntegrationTestBase handles clone, restore, and format execution. Adding or removing a repo requires only adding or deleting a ~20-line class.
  • Dynamic Helix sharding: PartitionByClass="true" metadata causes AssemblyScheduler to discover test classes at scheduling time and create one Helix work item per class automatically.
  • PR vs CI filtering: DOTNET_SDK_PR_VALIDATION env var and RunInPRValidation property control which tests run in PR validation. This infrastructure is extensible — any future test class can opt into or out of PR runs by setting this property, enabling broader use of PR-conditional test filtering beyond format tests.
  • Removed old infrastructure: Deleted eng/dotnet-format/ (YAML template, PowerShell verifier, CMD wrapper, RSP file) and removed pipeline references from .vsts-ci.yml / .vsts-pr.yml.

Design

Each repo is a separate test class inheriting from FormatIntegrationTestBase:

public class SdkFormatTests : FormatIntegrationTestBase
{
    protected override string RepoUrl => "https://github.com/dotnet/sdk";
    protected override string Sha => "e6bc966cc3d1348265b0831c6daca23267169d8f";
    protected override string TargetSolution => "sdk.slnx";
    protected override bool RunInPRValidation => true;
}

This one-class-per-repo pattern enables:

  1. Helix sharding — each class becomes its own work item (parallel execution)
  2. Zero maintenance — adding or removing a repo only requires adding or deleting a class file; sharding adapts automatically
  3. Extensible PR filtering — the RunInPRValidation pattern can be adopted by other test classes in the future to conditionally exclude expensive tests from PR validation

Add Stopwatch-based timing around each phase of the format verifier:
- SDK build (build.cmd -c Release)
- Git clone (init + fetch + checkout)
- Repo restore (Build.ps1 -restore)
- Solution restore (dotnet restore)
- Format workspace run
- Format folder run

This is a no-op change to collect baseline timings for CI optimization analysis.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@MichaelSimons MichaelSimons changed the title Add timing instrumentation to dotnet-format integration tests Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix May 8, 2026
…elix

Replace the YAML/PowerShell format verification pipeline legs with XUnit
integration tests that run on Helix. Each repo gets its own test class
(for automatic Helix sharding via PartitionByClass), and the pre-built
SDK from Helix is used directly — eliminating ~50 min of redundant SDK
builds per CI run and ~99 min of machine time per PR.

- Add test/dotnet-format.IntegrationTests/ with base class pattern
- Add PartitionByClass support to AssemblyScheduler for dynamic sharding
- Add PR vs CI filtering via DOTNET_SDK_PR_VALIDATION env var
- Remove eng/dotnet-format/ infrastructure and pipeline references

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@MichaelSimons
Copy link
Copy Markdown
Member Author

@mwiemer-microsoft, @phil-allen-msft - I had copilot convert the dotnet-format integration tests to helix in order to save compute as well as make the tests just list any other test. As part of this work, we only run a subset of the tests in PR (as discussed with Phil earlier). This is WIP but wanted to share to get your early feedback.

MichaelSimons and others added 7 commits May 11, 2026 14:38
These tests were previously excluded from CI (never ran in Helix).
Now that they are included, skip the 3 that have pre-existing failures
until they can be investigated and fixed.

- TestUseVarCodeStyle_AppliesWhenNotUsingVar (Roslyn IDE0007 behavior change)
- FSharpProjectsDoNotCreateException (exit code changed from 1 to 0)
- CSharpTemplateProject_LoadWithNoDiagnostics(blazorwasm) (CS0012 missing assembly)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…TypesTests

Pass TestContext.Current.CancellationToken to RunAsync() calls in
CA1033InterfaceWithDefaultImplementationsCSharpAsync and
CA1033InterfaceWithExpressionBodiedDefaultImplementationsCSharpAsync
test methods to fix xUnit1051 errors introduced by dotnet#50339.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ConditionalFactAttribute in xUnit v3 made Skip obsolete with error: true,
requiring AlwaysSkip for unconditional skips.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Target repos have known vulnerable transitive dependencies that cause
NU1902/NU1903 warnings-as-errors during restore. These are irrelevant
for format verification, so disable NuGetAudit during restore.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The tests mutate Environment.CurrentDirectory (process-global shared state)
which causes intermittent failures when tests run in parallel. Disable
parallelization to prevent race conditions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Arcade-based repos have global.json pinning a specific SDK version that
isn't available on Helix agents. Use the repo's own build script
(eng/build.sh or eng/Build.ps1) to install the correct SDK and perform
restore, matching the original pipeline behavior.

Non-Arcade repos (e.g. project-system) override UseRepoBuildScript=false
to use dotnet restore directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants