Upgrade Ubuntu VMs

[richlander]

Follow on from: #61755

@halter73 @wtgodbe

[richlander]

Looks like mostly crypto related issues.

@bartonjs @vcsjones

[vcsjones]

2025-05-09T16:51:48.2734338Z    System.Security.Cryptography.CryptographicException : Error occurred during a cryptographic operation.
2025-05-09T16:51:48.2734366Z   Stack Trace:
2025-05-09T16:51:48.2738127Z      at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.MapOpenSsl30Code(X509VerifyStatusCode code)
2025-05-09T16:51:48.2738755Z    at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.AddElementStatus(X509VerifyStatusCode errorCode, List`1 elementStatus, List`1 overallStatus, Boolean& overallHasNotSignatureValid)
2025-05-09T16:51:48.2741390Z    at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.AddElementStatus(ErrorCollection errorCodes, List`1 elementStatus, List`1 overallStatus, Boolean& overallHasNotSignatureValid)
2025-05-09T16:51:48.2741443Z    at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.BuildChainElements(WorkingChain workingChain, List`1& overallStatus)
2025-05-09T16:51:48.2741468Z    at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.Finish(OidCollection applicationPolicy, OidCollection certificatePolicy)
2025-05-09T16:51:48.2741551Z    at System.Security.Cryptography.X509Certificates.ChainPal.BuildChainCore(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, X509Certificate2Collection customTrustStore, X509ChainTrustMode trustMode, DateTime verificationTime, TimeSpan timeout, Boolean disableAia)
2025-05-09T16:51:48.2741667Z    at System.Security.Cryptography.X509Certificates.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, X509Certificate2Collection customTrustStore, X509ChainTrustMode trustMode, DateTime verificationTime, TimeSpan timeout, Boolean disableAia)
2025-05-09T16:51:48.2757190Z    at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
2025-05-09T16:51:48.2757275Z    at System.Net.Security.SslStreamCertificateContext.Create(X509Certificate2 target, X509Certificate2Collection additionalCertificates, Boolean offline, SslCertificateTrust trust, Boolean noOcspFetch)
2025-05-09T16:51:48.2765116Z    at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware..ctor(ConnectionDelegate next, HttpsConnectionAdapterOptions options, HttpProtocols httpProtocols, ILoggerFactory loggerFactory, KestrelMetrics metrics) in /_/src/Servers/Kestrel/Core/src/Middleware/HttpsConnectionMiddleware.cs:line 108

Looks like dotnet/runtime#114129 but I haven't been able to reproduce it. Without doing so it's a little hard, but maybe we can add some diagnostic code.

Basically OpenSSL 3 is giving us an error we don't know how to handle, but I can't get it to error myself :-D.

@bartonjs What do you think about including the numeric value in the exception? That would at least give us a clue to even understand what part of chain building is failing so we could better understand it.

[bartonjs]

What do you think about including the numeric value in the exception?

We have it in the assert... but yeah, that's not so nice when it happens outside of a debug build.

I'm torn between whether we want to change the message here, or be weird and do a double-throw so it's contained only in an InnerException. It's probably fine to just change the message here.

I was on a thread about this particular one a week or so ago, and no one can repro it outside of Helix, so something weird seems to be happening on these machines. Might need to pull it down from docker.

[richlander]

Might be useful to investigate in the VM itself. We're not using a container in this scenario.

https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[vcsjones]

Might be useful to investigate in the VM itself.

I as a GitHub employee don't have access to most things, including pretty much everything in that link, so that looks difficult.

Even if I could get on a VM, I don't know what I would do there, short of "Try running AspNetCore against a debug build of the runtime" and I don't know where to begin with that, either.

[vcsjones]

Might need to pull it down from docker.

I did, same image @LoopedBard3 mentioned in their report. My concern is it might be network condition, AzSecPack doing something goofy, etc.