dotnet tool restore started failing on macOS with NU3037 and NU3028 errors after 11th February

[martincostello]

Describe the bug

Since February 13th (👻), the CI in Polly when running on macOS has stopped working with the error shown below.

The strange thing is this doesn't seem to correlate with any specific code change to Polly itself. We had a successful build on February 11th after updating to the .NET 9.0.200 SDK here, so it doesn't seem to correlate nicely with a change in SDK behaviour.

The runner version doesn't appear to differ either, so that would presumably rule out a change in GitHub Actions:

Current runner version: '2.322.0'
Operating System
  macOS
  14.7.2
  23H311

I don't have access to a physical MacBook but two colleagues can replicate the failure on their machines by cloning the repo and running dotnet tool restore.

The things I don't really understand are:

• Why now, when the countersignature timestamp expired last year?
• Why only on macOS?

I've experimented with a few things in this PR to try and get more information, but with no success: App-vNext/Polly#2496. Check out the commit history and workflow run logs for more detail. For example, setting DOTNET_NUGET_SIGNATURE_VERIFICATION=false hasn't helped.

Seems like either it's failing on macOS when it shouldn't or it should be consistently failing on macOS, Linux and Windows if there is a genuine certificate trust issue.

I've also reported this here against the tool itself in case they've genuinely revoked a certificate or something, but again if that's the case I'd expect it to fail on all OSs: NuGetPackageExplorer/NuGetPackageExplorer#1698

To Reproduce

Clone App-vNext/Poll on a MacBook and run dotnet tool restore from the root of the repository.

Exceptions (if any)

> dotnet tool restore --no-http-cache --verbosity diagnostic
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/cake.tool/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/cake.tool/index.json 75ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/cake.tool/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/cake.tool/index.json 30ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/cake.tool/5.0.0/cake.tool.5.0.0.nupkg
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/cake.tool/5.0.0/cake.tool.5.0.0.nupkg 3ms
Verified that the NuGet package "cake.tool.5.0.0" has a valid signature.
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/index.json 39ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/0.0.1-alpha-1517061409/2.10.2.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/0.0.1-alpha-1517061409/2.10.2.json 31ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.11.0/2.40.0.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.11.0/2.40.0.json 30ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.40.1/2.63.1.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.40.1/2.63.1.json 38ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.64.0/2.78.2.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/docfx/page/2.64.0/2.78.2.json 30ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/docfx/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/docfx/index.json 29ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/docfx/2.78.2/docfx.2.78.2.nupkg
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/docfx/2.78.2/docfx.2.78.2.nupkg 2ms
Verified that the NuGet package "docfx.2.78.2" has a valid signature.
[NuGet Manager] [Info]   GET https://api.nuget.org/v3/registration5-gz-semver2/dotnet-validate/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3/registration5-gz-semver2/dotnet-validate/index.json 38ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/dotnet-validate/index.json
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/dotnet-validate/index.json 29ms
[NuGet Manager] [Info]   GET https://api.nuget.org/v3-flatcontainer/dotnet-validate/0.0.1-preview.304/dotnet-validate.0.0.1-preview.304.nupkg
[NuGet Manager] [Info]   OK https://api.nuget.org/v3-flatcontainer/dotnet-validate/0.0.1-preview.304/dotnet-validate.0.0.1-preview.304.nupkg 3ms
Failed to validate package signing.
Verifying dotnet-validate.0.0.1-preview.304
Signature type: Author
  Subject Name: CN=NuGet Package Explorer (.NET Foundation), O=NuGet Package Explorer (.NET Foundation), L=Redmond, S=Washington, C=US, SERIALNUMBER=603 389 068
  SHA256 hash: 7726640A0DCAB8252F40C73B2FE5C5F38C137E756C0B62521C07DF390288CDAB
  Valid from: 4/25/2021 12:00:00 AM to 7/22/2024 11:59:59 PM
Signature type: Repository
  Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
  SHA256 hash: 5A2901D6ADA3D18260B9C6DFE2[133](https://github.com/App-vNext/Polly/actions/runs/13326053490/job/37219456977?pr=2496#step:5:134)C95D74B9EEF6AE0E5DC334C8454D1477DF4
  Valid from: 2/16/2021 12:00:00 AM to 5/15/2024 11:59:59 PM
warn : NU3018: The repository countersignature found a chain building issue: RevocationStatusUnknown: An incomplete certificate revocation check occurred.
error: NU3037: The repository countersignature validity period has expired.
error: NU3028: The repository countersignature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package signature validation failed.

Further technical details

.NET SDK:
 Version:           9.0.200
 Commit:            90e8b202f2
 Workload version:  9.0.200-manifests.b4a8049f
 MSBuild version:   17.13.8+cbc39bea8
Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  14.7
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /Users/runner/.dotnet/sdk/9.0.200/
.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.
Host:
  Version:      9.0.2
  Architecture: arm64
  Commit:       80aa709f5d
.NET SDKs installed:
  7.0.102 [/Users/runner/.dotnet/sdk]
  7.0.202 [/Users/runner/.dotnet/sdk]
  7.0.[30](https://github.com/App-vNext/Polly/actions/runs/13326053490/job/37219456977?pr=2496#step:5:31)6 [/Users/runner/.dotnet/sdk]
  7.0.410 [/Users/runner/.dotnet/sdk]
  8.0.101 [/Users/runner/.dotnet/sdk]
  8.0.204 [/Users/runner/.dotnet/sdk]
  8.0.303 [/Users/runner/.dotnet/sdk]
  8.0.405 [/Users/runner/.dotnet/sdk]
  8.0.406 [/Users/runner/.dotnet/sdk]
  9.0.102 [/Users/runner/.dotnet/sdk]
  9.0.200 [/Users/runner/.dotnet/sdk]
.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.2 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.4 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.9 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.20 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.1 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.4 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.7 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.12 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.13 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.1 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.2 [/Users/runner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.2 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.4 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.9 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.1 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.4 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.7 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.12 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.13 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.1 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.2 [/Users/runner/.dotnet/shared/Microsoft.NETCore.App]
Other architectures found:
  None
Environment variables:
  DOTNET_ROOT       [/Users/runner/.dotnet]
global.json file:
  /Users/runner/work/Polly/Polly/global.json
Learn more:
  https://aka.ms/dotnet/info
Download .NET:
  https://aka.ms/dotnet/download

[dotnet-policy-service]

Thanks for creating this issue! We believe this issue is related to NuGet tooling, which is maintained by the NuGet team. Thus, we closed this one and encourage you to raise this issue in the NuGet repository instead. Don’t forget to check out NuGet’s contributing guide before submitting an issue!

If you believe this issue was closed out of error, please comment to let us know.

Happy Coding!

[baronfel]

Tool restore isn't owned by NuGet so I've reopened this and tagged it correctly.

[baronfel]

The 'verify signing' error message is coming from the SDK directly:

sdk/src/Cli/dotnet/NugetPackageDownloader/NuGetPackageDownloader.cs

Line 129 in fcba596

await VerifySigning(nupkgPath, repository);

Based on the repository we downloaded the package from, we try to validate the package signature, but I'm wondering if a) we're doing this correctly and b) if we're adhering to the DOTNET_NUGET_SIGNATURE_VERIFICATION environment variable that users have documentation about.

cc @nkolev92 for feedback on the signature validation process, and I guess @marcpopMSFT for thoughts on who we should get to dig into this more on our end?

[baronfel]

A few other notes:

• we have an override to skip verification
• that override is technically exposed for global tool installation, but not for local tool installation
• it feels like, much like we have the unified 'restore action config' to describe installation sources, interactivity, etc, we should also be treating signature verification for tools in a unified way.

[marcpopMSFT]

@joeloff is the signing expert but he's out for the moment. @edvilme @Forgind have done some experience in the tools space.