Run installer tests in pipeline #47313

NikolaMilosavljevic · 2025-03-06T16:46:31Z

This completes the installer package testing work - dotnet/runtime#109848

Features:

Adds x64 and arm64 installer test jobs to VMR Validation stage
Fixes arm64-specific test code-paths
Updates test-image repo
Adds filtering of NuGet packages, as the size of PackageArtifacts in official build is about 8GB
Enables running tests with a new property - TestInstallerPackages
Modifies tests.proj to condition test projects based on this new property. This is far from ideal, and will get more complicated as we add more test projects

As we need to support both pipeline and developer scenarios, some decisions were made to support these scenario with a single code-path, i.e.:

Pipeline contains a merged manifest which could have been used for filtering NuGet packages, based on the correct vertical name that corresponds to the target test platform. Vertical name isn't ideal and it could change, so this is something that could be incorporated in the future, but for now we filter NuGet packages based on file patterns. Filtering is needed so we don't need to copy all packages to the test image/container
Instead of copying packages to the image, I explored sharing packages from host to container. This is doable in pipeline as host is a real machine and not a docker container. However, in developer scenarios, host is likely a container. In this scenario, testing utilizes sibling containers. Developer could have packages (and repo clone) in container file-system or in a mounted docker volume. It would be possible to mount the same volume in a test container, but we cannot share packages from host container's file-system.

This was tested with an earlier dotnet sha: build

I'm running another verification with latest sha, to incorporate some recent pipeline changes: build

PR Overview

This PR adds installer tests to the pipeline and refines test setups on Linux to better handle platform-specific scenarios and package filtering.

Adds YAML pipeline templates for validating installer packages for Linux x64 and arm64.
Updates Linux installer tests to use platform-specific package architectures and improves package filtering logic.
Revises test code constants and downloads for consistency with current runtime dependencies.

Reviewed Changes

File	Description
eng/pipelines/templates/steps/vmr-validate-installers.yml	Introduces tasks to download build artifacts and authenticate feeds for installer tests.
src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs	Updates Linux installer test methods to support refined package filtering and architecture-specific adjustments.
eng/pipelines/templates/stages/vmr-build.yml	Adds new jobs for Linux x64 and Linux arm64 installer validations in the build stage.

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (3)

src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs:28

[nitpick] Consider renaming _excludeLinuxArch to a more descriptive name (e.g., _oppositeArch) to clearly indicate that it holds the architecture to exclude for package filtering.

private readonly string _excludeLinuxArch;

src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs:122

Using .Wait() on an asynchronous task can cause deadlock issues or hide exceptions; consider using GetAwaiter().GetResult() for better exception propagation.

DownloadFileAsync(NetStandard21RpmPackage, Path.Combine(_contextDir, Path.GetFileName(NetStandard21RpmPackage))).Wait();

src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs:98

[nitpick] The use of 'aarch64' for RPM packages versus 'arm64' for others might be confusing; consider standardizing the naming for clarity if possible.

string packageArchitecture = Config.Architecture == "x64" ? "x64" : packageType == PackageType.Rpm ? "aarch64" : "arm64";

ViktorHofer · 2025-03-06T17:17:40Z

eng/pipelines/templates/steps/vmr-validate-installers.yml

+        extraBuildProperties="$extraBuildProperties /p:TestDebPackages=true"
+      fi
+
+      ./build.sh \


This additional entry point invocation worries me. The VMR is supposed to receive properties like TargetOS, TargetArchitecture, ContinuousIntegrationBuild, etc that help us distinguish different OSs, architectures, CI vs dev, etc.

Could this yml import the steps/vmr-build.yml which handles all that?

That would be ideal. But, we don't currently use that for running tests - the only such instance also has its own custom invocation:

sdk/eng/pipelines/templates/jobs/vmr-build.yml

Lines 421 to 432 in d833ffb

- ${{ if eq(parameters.runTests, 'True') }}:

- script: build.cmd

$(baseArguments)

$(targetArguments)

$(buildPassArguments)

${{ parameters.extraProperties }}

-test

-excludeCIBinarylog

/bl:artifacts/log/Release/Test.binlog

displayName: Run Tests

workingDirectory: ${{ variables.sourcesPath }}

timeoutInMinutes: ${{ variables.runTestsTimeout }}

I'll think some more about this.

For this to work I would use jobs/vmr-build.yml template, but first I'd need to modify it to enable test-only mode (skipBuild or testOnly), which would run tests but skip the build. Today, we allow tests, as an addition to regular build step.

This would also, likely, fix the issue with missing test status, as test logs are currently not uploaded from this job. The alternative is to add test-log publishing to the new job.

the only such instance also has its own custom invocation

Right, we need the additional invocation as we can't build and run tests from a single invocation today. It also better separates the actions so I don't see that as a problem. That additional invocation in vmr-build.yml is kept in sync with the build invocation so that isn't a problem. It receives the properties that the build invocation receives. If it doesn't then that's a bug.

My feedback here was around not introducing another YML entry point for the VMR. vmr-build.yml should be able to handle the different scenarios (building and testing today). The vmr-final-join has its own entry point and I think that should also get consolidated eventually.

Yeah, this makes sense. I will add the test-only feature to jobs/vmr-build.yml and use that yml as a template for the new installer-testing job - thus removing the separate build.sh invocation.

OK. Three ways to refactor this.

Option 1 is for steps template to run just one build steps - either regular build or tests. It seems clunky, as we'd need to supply some property that specifies the build type, regular or test. Based on such property (or properties) we'd only do regular or test build. This way we can share a single entry point across all VMR pipelines.

Option 2 would run build and test (if requested). This implies that we'd also need to run any steps that are executed between main and test builds, i.e.

sdk/eng/pipelines/templates/jobs/vmr-build.yml

Lines 414 to 419 in d956fb2

- ${{ if eq(variables['_EnableDacSigning'], 'true') }}:

# TODO: Once we turn off the dotnet/runtime official build, move these templates into the VMR's eng folder.

- template: ${{ variables['Build.SourcesDirectory'] }}/src/runtime/eng/pipelines/coreclr/templates/remove-diagnostic-certs.yml

parameters:

isOfficialBuild: ${{ variables.isOfficialBuild }}

scriptRoot: '$(Build.SourcesDirectory)/src/runtime'

and

sdk/eng/pipelines/templates/jobs/vmr-build.yml

Lines 556 to 561 in d956fb2

- ${{ if ne(parameters.runOnline, 'True' )}}:

- script: |

set -ex

# Update the owner of the staging directory to the current user

sudo chown -R $(whoami) $(artifactsStagingDir)

displayName: Update owner of artifacts staging directory

Option 3 would move all of these steps to the new build steps yml:

sdk/eng/pipelines/templates/jobs/vmr-build.yml

Lines 383 to 621 in d956fb2

- ${{ if eq(parameters.targetOS, 'windows') }}:

# Node 20.x is a toolset dependency to build aspnetcore

# Keep in sync with aspnetcore: https://github.com/dotnet/aspnetcore/blob/7d5309210d8f7bae8fa074da495e9d009d67f1b4/.azure/pipelines/ci.yml#L719-L722

- task: NodeTool@0

displayName: Install Node 20.x

inputs:

versionSpec: 20.x

- ${{ if eq(variables['_EnableDacSigning'], 'true') }}:

# TODO: Once we turn off the dotnet/runtime official build, move these templates into the VMR's eng folder.

- template: ${{ variables['Build.SourcesDirectory'] }}/src/runtime/eng/pipelines/coreclr/templates/install-diagnostic-certs.yml

parameters:

isOfficialBuild: ${{ variables.isOfficialBuild }}

certNames:

- 'dotnetesrp-diagnostics-aad-ssl-cert'

- 'dotnet-diagnostics-esrp-pki-onecert'

vaultName: 'clrdiag-esrp-id'

azureSubscription: 'diagnostics-esrp-kvcertuser'

scriptRoot: '$(Build.SourcesDirectory)/src/runtime'

- script: build.cmd

$(baseArguments)

$(targetArguments)

$(signArguments)

$(buildPassArguments)

$(ibcArguments)

$(_SignDiagnosticFilesArgs)

${{ parameters.extraProperties }}

displayName: Build

workingDirectory: ${{ variables.sourcesPath }}

- ${{ if eq(variables['_EnableDacSigning'], 'true') }}:

# TODO: Once we turn off the dotnet/runtime official build, move these templates into the VMR's eng folder.

- template: ${{ variables['Build.SourcesDirectory'] }}/src/runtime/eng/pipelines/coreclr/templates/remove-diagnostic-certs.yml

parameters:

isOfficialBuild: ${{ variables.isOfficialBuild }}

scriptRoot: '$(Build.SourcesDirectory)/src/runtime'

- ${{ if eq(parameters.runTests, 'True') }}:

- script: build.cmd

$(baseArguments)

$(targetArguments)

$(buildPassArguments)

${{ parameters.extraProperties }}

-test

-excludeCIBinarylog

/bl:artifacts/log/Release/Test.binlog

displayName: Run Tests

workingDirectory: ${{ variables.sourcesPath }}

timeoutInMinutes: ${{ variables.runTestsTimeout }}

- ${{ else }}:

- ${{ if eq(parameters.targetOS, 'osx') }}:

- script: |

$(sourcesPath)/eng/common/native/install-dependencies.sh osx

displayName: Install dependencies

- ${{ if eq(parameters.buildSourceOnly, 'true') }}:

- script: |

set -ex

customPrepArgs=""

prepSdk=true

if [[ -n '${{ parameters.artifactsRid }}' ]]; then

customPrepArgs="${customPrepArgs} --artifacts-rid ${{ parameters.artifactsRid }}"

fi

if [[ '${{ parameters.withPreviousSDK }}' == 'True' ]]; then

# Source-built artifacts are from CentOS 9 Stream. We want to download them without

# downloading portable versions from the internet.

customPrepArgs="${customPrepArgs} --no-sdk --no-bootstrap"

prepSdk=false

elif [[ '${{ length(parameters.reuseBuildArtifactsFrom) }}' -gt '0' ]]; then

customPrepArgs="${customPrepArgs} --no-sdk --no-artifacts"

prepSdk=false

fi

if [[ "$prepSdk" == "false" ]]; then

mkdir $(sourcesPath)/.dotnet

previousSdkPath="$(sourcesPath)/prereqs/packages/archive/dotnet-sdk-*.tar.gz"

eval tar -ozxf "$previousSdkPath" -C "$(sourcesPath)/.dotnet"

eval rm -f "$previousSdkPath"

echo "##vso[task.setvariable variable=additionalBuildArgs]--with-sdk $(sourcesPath)/.dotnet"

fi

./prep-source-build.sh $customPrepArgs

displayName: Prep the Build

workingDirectory: $(sourcesPath)

- script: |

set -ex

df -h

customEnvVars=""

customPreBuildArgs=""

customBuildArgs="--ci --clean-while-building --prepareMachine -c ${{ parameters.configuration }} $(officialBuildParameter)"

if [[ '${{ parameters.runOnline }}' == 'True' ]]; then

customBuildArgs="$customBuildArgs --online"

else

customPreBuildArgs="$customPreBuildArgs sudo unshare -n"

fi

if [[ '${{ parameters.enablePoison }}' == 'True' ]]; then

customBuildArgs="$customBuildArgs --poison"

fi

if [[ '${{ parameters.buildFromArchive }}' == 'True' ]]; then

customBuildArgs="$customBuildArgs --source-repository https://github.com/dotnet/dotnet"

customBuildArgs="$customBuildArgs --source-version $(git -C "$(vmrPath)" rev-parse HEAD)"

fi

if [[ '${{ parameters.buildSourceOnly }}' == 'True' ]]; then

customBuildArgs="$customBuildArgs --source-only"

extraBuildProperties="$extraBuildProperties /p:ReportSbrpUsage=true"

fi

if [[ '${{ parameters.useMonoRuntime }}' == 'True' ]]; then

customBuildArgs="$customBuildArgs --use-mono-runtime"

fi

if [[ '${{ parameters.sign }}' == 'True' ]] && [[ '${{ parameters.buildSourceOnly }}' != 'True' ]]; then

customBuildArgs="$customBuildArgs --sign"

if [[ '$(_SignType)' == 'real' ]] || [[ '$(_SignType)' == 'test' ]]; then

# Force dry run signing until https://github.com/dotnet/source-build/issues/4793 is resolved - https://github.com/dotnet/source-build/issues/4678

extraBuildProperties="$extraBuildProperties /p:DotNetSignType=$(_SignType) /p:TeamName=$(_TeamName) /p:ForceDryRunSigning=true"

else

extraBuildProperties="$extraBuildProperties /p:ForceDryRunSigning=true"

fi

fi

if [[ -n "${{ parameters.targetRid }}" ]]; then

customBuildArgs="$customBuildArgs --target-rid ${{ parameters.targetRid }}"

fi

if [[ -n "${{ parameters.crossRootFs }}" ]]; then

customEnvVars="$customEnvVars ROOTFS_DIR=${{ parameters.crossRootFs}}"

if [[ '${{ parameters.targetArchitecture }}' != 'wasm' ]]; then

extraBuildProperties="$extraBuildProperties /p:CrossBuild=true"

fi

fi

if [[ ! -z '${{ parameters.targetOS }}' ]]; then

extraBuildProperties="$extraBuildProperties /p:TargetOS=${{ parameters.targetOS }}"

fi

if [[ ! -z '${{ parameters.targetArchitecture }}' ]]; then

extraBuildProperties="$extraBuildProperties /p:TargetArchitecture=${{ parameters.targetArchitecture }}"

fi

if [[ -n "${{ parameters.buildPass }}" ]]; then

extraBuildProperties="$extraBuildProperties /p:DotNetBuildPass=${{ parameters.buildPass }}"

fi

if [[ -n "${{ parameters.extraProperties }}" ]]; then

extraBuildProperties="$extraBuildProperties ${{ parameters.extraProperties }}"

fi

extraBuildProperties="$extraBuildProperties /p:VerticalName=$(Agent.JobName)"

extraBuildProperties="$extraBuildProperties /p:ArtifactsStagingDir=$(artifactsStagingDir)"

buildArgs="$(additionalBuildArgs) $customBuildArgs $extraBuildProperties"

for envVar in $customEnvVars; do

customEnvVarsWithBashSyntax="$customEnvVarsWithBashSyntax export $envVar;"

done

eval $customEnvVarsWithBashSyntax

$customPreBuildArgs ./build.sh $buildArgs

displayName: Build

workingDirectory: $(sourcesPath)

- ${{ if ne(parameters.runOnline, 'True' )}}:

- script: |

set -ex

# Update the owner of the staging directory to the current user

sudo chown -R $(whoami) $(artifactsStagingDir)

displayName: Update owner of artifacts staging directory

# Only run tests if enabled

- ${{ if eq(parameters.runTests, 'True') }}:

# Setup the NuGet sources used by the tests to use private feeds. This is necessary when testing internal-only product

# builds where the packages are only available in the private feeds. This allows the tests to restore from those feeds.

- ${{ if eq(variables['System.TeamProject'], 'internal') }}:

- task: Bash@3

displayName: Setup Private Feeds Credentials

inputs:

filePath: $(sourcesPath)/src/sdk/eng/common/SetupNugetSources.sh

arguments: $(sourcesPath)/src/sdk/NuGet.config $Token

env:

Token: $(dn-bot-dnceng-artifact-feeds-rw)

- script: cp "$(sourcesPath)/src/sdk/NuGet.config" "$(sourcesPath)/test/Microsoft.DotNet.SourceBuild.Tests/assets/online.NuGet.Config"

displayName: Copy Test NuGet Config for Smoke Tests

- script: |

set -ex

customPreBuildArgs=''

customBuildArgs=''

extraBuildProperties=''

customBuildArgs="--ci --prepareMachine -c ${{ parameters.configuration }} $(officialBuildParameter)"

if [[ '${{ parameters.runOnline }}' == 'False' ]]; then

customPreBuildArgs="$customPreBuildArgs sudo"

fi

if [[ ! -z '${{ parameters.targetOS }}' ]]; then

extraBuildProperties="$extraBuildProperties /p:TargetOS=${{ parameters.targetOS }}"

fi

if [[ ! -z '${{ parameters.targetArchitecture }}' ]]; then

extraBuildProperties="$extraBuildProperties /p:TargetArchitecture=${{ parameters.targetArchitecture }}"

fi

if [[ '${{ parameters.buildSourceOnly }}' == 'True' ]]; then

if [[ '${{ parameters.enablePoison }}' == 'True' ]]; then

customBuildArgs="$customBuildArgs --poison"

fi

customBuildArgs="$customBuildArgs --source-only /p:SourceBuildTestsExcludeOmniSharpTests=${{ parameters.excludeOmniSharpTests }}"

fi

if [[ -n "${{ parameters.targetRid }}" ]]; then

customBuildArgs="$customBuildArgs --target-rid ${{ parameters.targetRid }}"

fi

extraBuildProperties="$extraBuildProperties /p:VerticalName=$(Agent.JobName)"

extraBuildProperties="$extraBuildProperties /p:ArtifactsStagingDir=$(artifactsStagingDir)"

if [[ -n "${{ parameters.extraProperties }}" ]]; then

extraBuildProperties="$extraBuildProperties ${{ parameters.extraProperties }}"

fi

cd $(sourcesPath)

$customPreBuildArgs ./build.sh --test --excludeCIBinarylog /bl:artifacts/log/Release/Test.binlog $customBuildArgs $extraBuildProperties $(additionalBuildArgs)

displayName: Run Tests

timeoutInMinutes: ${{ variables.runTestsTimeout }}

Option 3 sounds best but I'm not sure what others think. cc @mmitche

I think I am missing the difference between Option 1 and 3 here.

Option 1 would create a very small yml with a single step to invoke build.cmd or build.sh with various arguments. All conditioning, prep, and all other steps, like installing Node would stay in jobs/vmr-build.yml. This option would require a new parameter that would specify the build type (i.e. regularBuild vs test), so we could run one or the other type of build.

Option 3 would move the whole ~250 line block I linked above into the new steps yml. It keeps most of build and related steps together, and leaves the extra stuff like publishing in the current yml.

I will follow up with refactoring in a separate PR - still working out the issues.

src/SourceBuild/content/test/tests.proj

MichaelSimons · 2025-03-06T17:18:20Z

src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs

@@ -92,31 +95,46 @@ public void DebTest(string repo, string tag)

    private void InitializeContext(PackageType packageType)
    {
+        string packageArchitecture =
+            Config.Architecture == "x64" ?


Consider defining consts for the architectures to help avoid possible bugs.

Fixed with 4c52a0c

MichaelSimons · 2025-03-06T17:25:34Z

src/SourceBuild/content/test/tests.proj

@@ -20,7 +20,11 @@
    <_RunScenarioTests Condition="'$(DotNetBuildPass)' != '' and '$(DotNetBuildPass)' != '1'">false</_RunScenarioTests>
  </PropertyGroup>

-  <ItemGroup>
+  <ItemGroup Condition="'$(TestInstallerPackages)' == 'true'">
+    <ProjectReference Condition="'$(TestRpmPackages)' == 'true' or '$(TestDebPackages)' == 'true'" Include="Microsoft.DotNet.Installer.Tests\Microsoft.DotNet.Installer.Tests.csproj" BuildInParallel="false" />


Are these conditions really necessary here? It seems unnecessary and a bit of a maintenance concern as new installer tests are added.

Oh, I did forget to remove Condition="'$(TestRpmPackages)' == 'true' or '$(TestDebPackages)' == 'true'"

Fixed with 4c52a0c

MichaelSimons · 2025-03-06T17:30:58Z

eng/pipelines/templates/steps/vmr-validate-installers.yml

I was expecting to see tests listed under the tests tab of the build you linked to - https://dev.azure.com/dnceng/internal/_build/results?buildId=2655424&view=ms.vss-test-web.build-test-results-tab

Also do you happen to have build where there is a failure? I am curious what the UX is in this case on seeing what scenario test failed.

Hmm, interesting - will check what's going on.

I do not have the pipeline run, with failing scenario tests. I could share the output of the local run.

Fixed with defc5d2

This will still, likely, get refactored further to utilize shared build.sh invocation from jobs/vmr-build.yml.

There was an issue with search folder so tests weren't being published. This is now fixed with 08f3e6b

Verification build with tests showing up correctly: https://dev.azure.com/dnceng/internal/_build/results?buildId=2659027&view=ms.vss-test-web.build-test-results-tab

It doesn't look like the capture the inner scenario test log file. I'm curious how difficult it will be to diagnose failures when there is a scenario test failure. I am fine with adjusting as needed if this proves to be a problem.

NikolaMilosavljevic · 2025-03-13T19:31:24Z

Updated the branch with latest to pick up the fix for a known tests issue - #47549

NikolaMilosavljevic · 2025-03-14T03:58:26Z

@ViktorHofer @MichaelSimons if you're OK with current PR state, can you approve it? All checks are green and this has been tested in an internal build - the failing job was also failing in the sha I picked for my VMR tests.

I am going to follow up with refactoring changes in a separate PR.

ViktorHofer · 2025-03-14T08:47:09Z

eng/pipelines/templates/steps/vmr-validate-installers.yml

+        extraBuildProperties="$extraBuildProperties /p:TestDebPackages=true"
+      fi
+
+      ./build.sh \


I think you are missing the --ci flag here which is one of the more important args as it controls whether the machine wide package cache gets used or not (for non-source-only builds).

Oh, yeah, I don't use it locally, but other official legs/steps do use it - will fix, thanks.

Fixed with 5dfa459

MichaelSimons · 2025-03-14T16:00:46Z

eng/pipelines/templates/stages/vmr-build.yml

+    - job: ValidateInstallers_Linux_x64
+      displayName: Validate Installers - Linux x64
+      pool: ${{ parameters.pool_Linux }}
+      timeoutInMinutes: 240


This timeout seems excessive. Looks like in your test run it took 20 minutes.

Yeah, I'll adjust - thanks.

Fixed with d9895e4

MichaelSimons · 2025-03-14T16:04:02Z

eng/pipelines/templates/steps/vmr-validate-installers.yml

It doesn't look like the capture the inner scenario test log file. I'm curious how difficult it will be to diagnose failures when there is a scenario test failure. I am fine with adjusting as needed if this proves to be a problem.

MichaelSimons · 2025-03-14T16:07:10Z

src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs

@@ -53,13 +54,15 @@ public LinuxInstallerTests(ITestOutputHelper outputHelper)
        Directory.CreateDirectory(_tmpDir);
        _contextDir = Path.Combine(_tmpDir, Path.GetRandomFileName());
        Directory.CreateDirectory(_contextDir);
+
+        _excludeLinuxArch = Config.Architecture == Architecture.X64 ? "arm64" : "x64";


Would it be wrong to use the tolower string representation of the enum?

No, it would be best to do that - will fix.

Fixed with d9895e4

NikolaMilosavljevic · 2025-03-14T16:23:03Z

Regarding scenario-tests log - I'll do that in the follow-up PR.

MichaelSimons · 2025-03-14T17:07:33Z

src/SourceBuild/content/test/Microsoft.DotNet.Installer.Tests/LinuxInstallerTests.cs

@@ -55,7 +55,9 @@ public LinuxInstallerTests(ITestOutputHelper outputHelper)
        _contextDir = Path.Combine(_tmpDir, Path.GetRandomFileName());
        Directory.CreateDirectory(_contextDir);

-        _excludeLinuxArch = Config.Architecture == Architecture.X64 ? "arm64" : "x64";
+        _excludeLinuxArch = Config.Architecture == Architecture.X64 ?


Could this just be the following? Why is the condition necessary?

Suggested change

_excludeLinuxArch = Config.Architecture == Architecture.X64 ?

_excludeLinuxArch = Architecture.Arm64.ToString().ToLower()

Wrong example... Config.Architecture.ToString().ToLower()

For x64 tests we exclude arm64 packages from copying to the image. For arm64 tests we exclude x64 packages.

I see - taking a step back, my larger question is what will happen if/when additional architectures are added? Ideally this code would be written in a way that would be unaffected.

If we add another supported architecture for testing, we'd need to modify this code and the Architecture enum, at the same time.

This member field as an optimization only. We could just iterate over all supported architectures that are not being tested, in the filtering method. That would incur performance cost.

Ideally, as we discussed before, I think filtering should be modified to consume the merged manifest and filter packages based on vertical name/Id. I can do a follow up PR with that change.

I think filtering should be modified to consume the merged manifest and filter packages based on vertical name/Id. I can do a follow up PR with that change.

Sounds good to me.

NikolaMilosavljevic · 2025-03-20T15:06:35Z

I'll need to resolve conflicts with changes from #47338 before merging this PR.

NikolaMilosavljevic · 2025-03-20T21:27:46Z

After picking up latest changes from main, this PR is now hitting a known SB tests issue: dotnet/source-build#4952

I cannot merge on red, so someone would need to help with that - @MichaelSimons @marcpopMSFT

Run installer tests in pipeline

5d3d117

Copilot bot review requested due to automatic review settings March 6, 2025 16:46

NikolaMilosavljevic requested review from a team as code owners March 6, 2025 16:46

dotnet-issue-labeler bot added Area-Infrastructure untriaged Request triage from a team member labels Mar 6, 2025

Copilot AI reviewed Mar 6, 2025

View reviewed changes

ViktorHofer reviewed Mar 6, 2025

View reviewed changes

src/SourceBuild/content/test/tests.proj Outdated Show resolved Hide resolved

MichaelSimons reviewed Mar 6, 2025

View reviewed changes

NikolaMilosavljevic added 4 commits March 6, 2025 18:42

Addressing comments

4c52a0c

Add project override

1569909

Publish test results

defc5d2

Update search folder

08f3e6b

build-analysis bot mentioned this pull request Mar 10, 2025

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

NikolaMilosavljevic added 2 commits March 13, 2025 08:31

Merge branch 'main' into installer.test.pipeline

69eba5d

Merge branch 'main' into installer.test.pipeline

6a94386

Merge branch 'main' into installer.test.pipeline

48efb04

ViktorHofer reviewed Mar 14, 2025

View reviewed changes

ViktorHofer approved these changes Mar 14, 2025

View reviewed changes

Use --ci

5dfa459

MichaelSimons reviewed Mar 14, 2025

View reviewed changes

Few tweaks

d9895e4

MichaelSimons reviewed Mar 14, 2025

View reviewed changes

MichaelSimons approved these changes Mar 19, 2025

View reviewed changes

Merge branch 'main' into installer.test.pipeline

b85afdb

MichaelSimons merged commit 60c0044 into dotnet:main Mar 21, 2025
36 of 39 checks passed

	- ${{ if eq(parameters.runTests, 'True') }}:
	- script: build.cmd
	$(baseArguments)
	$(targetArguments)
	$(buildPassArguments)
	${{ parameters.extraProperties }}
	-test
	-excludeCIBinarylog
	/bl:artifacts/log/Release/Test.binlog
	displayName: Run Tests
	workingDirectory: ${{ variables.sourcesPath }}
	timeoutInMinutes: ${{ variables.runTestsTimeout }}

	- ${{ if eq(variables['_EnableDacSigning'], 'true') }}:
	# TODO: Once we turn off the dotnet/runtime official build, move these templates into the VMR's eng folder.
	- template: ${{ variables['Build.SourcesDirectory'] }}/src/runtime/eng/pipelines/coreclr/templates/remove-diagnostic-certs.yml
	parameters:
	isOfficialBuild: ${{ variables.isOfficialBuild }}
	scriptRoot: '$(Build.SourcesDirectory)/src/runtime'

	- ${{ if ne(parameters.runOnline, 'True' )}}:
	- script: \|
	set -ex
	# Update the owner of the staging directory to the current user
	sudo chown -R $(whoami) $(artifactsStagingDir)
	displayName: Update owner of artifacts staging directory

	- ${{ if eq(parameters.targetOS, 'windows') }}:
	# Node 20.x is a toolset dependency to build aspnetcore
	# Keep in sync with aspnetcore: https://github.com/dotnet/aspnetcore/blob/7d5309210d8f7bae8fa074da495e9d009d67f1b4/.azure/pipelines/ci.yml#L719-L722
	- task: NodeTool@0
	displayName: Install Node 20.x
	inputs:
	versionSpec: 20.x

	- ${{ if eq(variables['_EnableDacSigning'], 'true') }}:
	# TODO: Once we turn off the dotnet/runtime official build, move these templates into the VMR's eng folder.
	- template: ${{ variables['Build.SourcesDirectory'] }}/src/runtime/eng/pipelines/coreclr/templates/install-diagnostic-certs.yml
	parameters:
	isOfficialBuild: ${{ variables.isOfficialBuild }}
	certNames:
	- 'dotnetesrp-diagnostics-aad-ssl-cert'
	- 'dotnet-diagnostics-esrp-pki-onecert'
	vaultName: 'clrdiag-esrp-id'
	azureSubscription: 'diagnostics-esrp-kvcertuser'
	scriptRoot: '$(Build.SourcesDirectory)/src/runtime'

	- script: build.cmd
	$(baseArguments)
	$(targetArguments)
	$(signArguments)
	$(buildPassArguments)
	$(ibcArguments)
	$(_SignDiagnosticFilesArgs)
	${{ parameters.extraProperties }}
	displayName: Build
	workingDirectory: ${{ variables.sourcesPath }}

	- ${{ if eq(variables['_EnableDacSigning'], 'true') }}:
	# TODO: Once we turn off the dotnet/runtime official build, move these templates into the VMR's eng folder.
	- template: ${{ variables['Build.SourcesDirectory'] }}/src/runtime/eng/pipelines/coreclr/templates/remove-diagnostic-certs.yml
	parameters:
	isOfficialBuild: ${{ variables.isOfficialBuild }}
	scriptRoot: '$(Build.SourcesDirectory)/src/runtime'

	- ${{ if eq(parameters.runTests, 'True') }}:
	- script: build.cmd
	$(baseArguments)
	$(targetArguments)
	$(buildPassArguments)
	${{ parameters.extraProperties }}
	-test
	-excludeCIBinarylog
	/bl:artifacts/log/Release/Test.binlog
	displayName: Run Tests
	workingDirectory: ${{ variables.sourcesPath }}
	timeoutInMinutes: ${{ variables.runTestsTimeout }}

	- ${{ else }}:
	- ${{ if eq(parameters.targetOS, 'osx') }}:
	- script: \|
	$(sourcesPath)/eng/common/native/install-dependencies.sh osx
	displayName: Install dependencies
	- ${{ if eq(parameters.buildSourceOnly, 'true') }}:
	- script: \|
	set -ex

	customPrepArgs=""
	prepSdk=true

	if [[ -n '${{ parameters.artifactsRid }}' ]]; then
	customPrepArgs="${customPrepArgs} --artifacts-rid ${{ parameters.artifactsRid }}"
	fi

	if [[ '${{ parameters.withPreviousSDK }}' == 'True' ]]; then
	# Source-built artifacts are from CentOS 9 Stream. We want to download them without
	# downloading portable versions from the internet.
	customPrepArgs="${customPrepArgs} --no-sdk --no-bootstrap"
	prepSdk=false
	elif [[ '${{ length(parameters.reuseBuildArtifactsFrom) }}' -gt '0' ]]; then
	customPrepArgs="${customPrepArgs} --no-sdk --no-artifacts"
	prepSdk=false
	fi

	if [[ "$prepSdk" == "false" ]]; then
	mkdir $(sourcesPath)/.dotnet
	previousSdkPath="$(sourcesPath)/prereqs/packages/archive/dotnet-sdk-*.tar.gz"
	eval tar -ozxf "$previousSdkPath" -C "$(sourcesPath)/.dotnet"
	eval rm -f "$previousSdkPath"

	echo "##vso[task.setvariable variable=additionalBuildArgs]--with-sdk $(sourcesPath)/.dotnet"
	fi

	./prep-source-build.sh $customPrepArgs
	displayName: Prep the Build
	workingDirectory: $(sourcesPath)

	- script: \|
	set -ex
	df -h

	customEnvVars=""
	customPreBuildArgs=""
	customBuildArgs="--ci --clean-while-building --prepareMachine -c ${{ parameters.configuration }} $(officialBuildParameter)"

	if [[ '${{ parameters.runOnline }}' == 'True' ]]; then
	customBuildArgs="$customBuildArgs --online"
	else
	customPreBuildArgs="$customPreBuildArgs sudo unshare -n"
	fi

	if [[ '${{ parameters.enablePoison }}' == 'True' ]]; then
	customBuildArgs="$customBuildArgs --poison"
	fi

	if [[ '${{ parameters.buildFromArchive }}' == 'True' ]]; then
	customBuildArgs="$customBuildArgs --source-repository https://github.com/dotnet/dotnet"
	customBuildArgs="$customBuildArgs --source-version $(git -C "$(vmrPath)" rev-parse HEAD)"
	fi

	if [[ '${{ parameters.buildSourceOnly }}' == 'True' ]]; then
	customBuildArgs="$customBuildArgs --source-only"
	extraBuildProperties="$extraBuildProperties /p:ReportSbrpUsage=true"
	fi

	if [[ '${{ parameters.useMonoRuntime }}' == 'True' ]]; then
	customBuildArgs="$customBuildArgs --use-mono-runtime"
	fi

	if [[ '${{ parameters.sign }}' == 'True' ]] && [[ '${{ parameters.buildSourceOnly }}' != 'True' ]]; then
	customBuildArgs="$customBuildArgs --sign"
	if [[ '$(_SignType)' == 'real' ]] \|\| [[ '$(_SignType)' == 'test' ]]; then
	# Force dry run signing until https://github.com/dotnet/source-build/issues/4793 is resolved - https://github.com/dotnet/source-build/issues/4678
	extraBuildProperties="$extraBuildProperties /p:DotNetSignType=$(_SignType) /p:TeamName=$(_TeamName) /p:ForceDryRunSigning=true"
	else
	extraBuildProperties="$extraBuildProperties /p:ForceDryRunSigning=true"
	fi
	fi

	if [[ -n "${{ parameters.targetRid }}" ]]; then
	customBuildArgs="$customBuildArgs --target-rid ${{ parameters.targetRid }}"
	fi

	if [[ -n "${{ parameters.crossRootFs }}" ]]; then
	customEnvVars="$customEnvVars ROOTFS_DIR=${{ parameters.crossRootFs}}"
	if [[ '${{ parameters.targetArchitecture }}' != 'wasm' ]]; then
	extraBuildProperties="$extraBuildProperties /p:CrossBuild=true"
	fi
	fi

	if [[ ! -z '${{ parameters.targetOS }}' ]]; then
	extraBuildProperties="$extraBuildProperties /p:TargetOS=${{ parameters.targetOS }}"
	fi

	if [[ ! -z '${{ parameters.targetArchitecture }}' ]]; then
	extraBuildProperties="$extraBuildProperties /p:TargetArchitecture=${{ parameters.targetArchitecture }}"
	fi

	if [[ -n "${{ parameters.buildPass }}" ]]; then
	extraBuildProperties="$extraBuildProperties /p:DotNetBuildPass=${{ parameters.buildPass }}"
	fi

	if [[ -n "${{ parameters.extraProperties }}" ]]; then
	extraBuildProperties="$extraBuildProperties ${{ parameters.extraProperties }}"
	fi

	extraBuildProperties="$extraBuildProperties /p:VerticalName=$(Agent.JobName)"
	extraBuildProperties="$extraBuildProperties /p:ArtifactsStagingDir=$(artifactsStagingDir)"

	buildArgs="$(additionalBuildArgs) $customBuildArgs $extraBuildProperties"

	for envVar in $customEnvVars; do
	customEnvVarsWithBashSyntax="$customEnvVarsWithBashSyntax export $envVar;"
	done

	eval $customEnvVarsWithBashSyntax
	$customPreBuildArgs ./build.sh $buildArgs
	displayName: Build
	workingDirectory: $(sourcesPath)

	- ${{ if ne(parameters.runOnline, 'True' )}}:
	- script: \|
	set -ex
	# Update the owner of the staging directory to the current user
	sudo chown -R $(whoami) $(artifactsStagingDir)
	displayName: Update owner of artifacts staging directory

	# Only run tests if enabled
	- ${{ if eq(parameters.runTests, 'True') }}:
	# Setup the NuGet sources used by the tests to use private feeds. This is necessary when testing internal-only product
	# builds where the packages are only available in the private feeds. This allows the tests to restore from those feeds.
	- ${{ if eq(variables['System.TeamProject'], 'internal') }}:
	- task: Bash@3
	displayName: Setup Private Feeds Credentials
	inputs:
	filePath: $(sourcesPath)/src/sdk/eng/common/SetupNugetSources.sh
	arguments: $(sourcesPath)/src/sdk/NuGet.config $Token
	env:
	Token: $(dn-bot-dnceng-artifact-feeds-rw)

	- script: cp "$(sourcesPath)/src/sdk/NuGet.config" "$(sourcesPath)/test/Microsoft.DotNet.SourceBuild.Tests/assets/online.NuGet.Config"
	displayName: Copy Test NuGet Config for Smoke Tests

	- script: \|
	set -ex

	customPreBuildArgs=''
	customBuildArgs=''
	extraBuildProperties=''
	customBuildArgs="--ci --prepareMachine -c ${{ parameters.configuration }} $(officialBuildParameter)"

	if [[ '${{ parameters.runOnline }}' == 'False' ]]; then
	customPreBuildArgs="$customPreBuildArgs sudo"
	fi

	if [[ ! -z '${{ parameters.targetOS }}' ]]; then
	extraBuildProperties="$extraBuildProperties /p:TargetOS=${{ parameters.targetOS }}"
	fi

	if [[ ! -z '${{ parameters.targetArchitecture }}' ]]; then
	extraBuildProperties="$extraBuildProperties /p:TargetArchitecture=${{ parameters.targetArchitecture }}"
	fi

	if [[ '${{ parameters.buildSourceOnly }}' == 'True' ]]; then
	if [[ '${{ parameters.enablePoison }}' == 'True' ]]; then
	customBuildArgs="$customBuildArgs --poison"
	fi
	customBuildArgs="$customBuildArgs --source-only /p:SourceBuildTestsExcludeOmniSharpTests=${{ parameters.excludeOmniSharpTests }}"
	fi

	if [[ -n "${{ parameters.targetRid }}" ]]; then
	customBuildArgs="$customBuildArgs --target-rid ${{ parameters.targetRid }}"
	fi

	extraBuildProperties="$extraBuildProperties /p:VerticalName=$(Agent.JobName)"
	extraBuildProperties="$extraBuildProperties /p:ArtifactsStagingDir=$(artifactsStagingDir)"

	if [[ -n "${{ parameters.extraProperties }}" ]]; then
	extraBuildProperties="$extraBuildProperties ${{ parameters.extraProperties }}"
	fi

	cd $(sourcesPath)
	$customPreBuildArgs ./build.sh --test --excludeCIBinarylog /bl:artifacts/log/Release/Test.binlog $customBuildArgs $extraBuildProperties $(additionalBuildArgs)

	displayName: Run Tests
	timeoutInMinutes: ${{ variables.runTestsTimeout }}

	_excludeLinuxArch = Config.Architecture == Architecture.X64 ?
	_excludeLinuxArch = Architecture.Arm64.ToString().ToLower()

Run installer tests in pipeline #47313

Run installer tests in pipeline #47313

Conversation

NikolaMilosavljevic commented Mar 6, 2025

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikolaMilosavljevic Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikolaMilosavljevic Mar 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikolaMilosavljevic commented Mar 13, 2025

NikolaMilosavljevic commented Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikolaMilosavljevic commented Mar 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelSimons Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NikolaMilosavljevic commented Mar 20, 2025

NikolaMilosavljevic commented Mar 20, 2025

NikolaMilosavljevic Mar 11, 2025 •

edited

Loading

NikolaMilosavljevic Mar 10, 2025 •

edited

Loading

NikolaMilosavljevic commented Mar 14, 2025 •

edited

Loading

MichaelSimons Mar 18, 2025 •

edited

Loading