[BUG]: 3.239.1 fails to connect to git

[tisonv]

What happened?

After updating from 3.238.0 to 3.239.1, all builds fail.
Rollbacked to 3.238.0 and it works again

Versions

2.239.1 on Windows 2019

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

Azure DevOps Server 2022.1 (AzureDevOpsServer_20240305.2)

Operation system

Windows Server 2019

Version controll system

Git

Relevant log output

git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin
fatal: unable to access 'https://XXXXX/Collection/YYY/_git/YYY/': schannel: next InitializeSecurityContext failed: Unknown error (0x80092013)
##[warning]Git fetch failed with exit code 128, back off 6,964 seconds before retry.

[DmitriiBobreshev]

Hi @tisonv, Thanks for the feedback. We suspect that the problem might be related to git extraheader, agent should remove it after the execution. Could you please check that the Agent have access to it with write permissions and you have nothing in it?
Also could you please install a fresh agent from scratch to check will it works fine?

[tisonv]

Hello @DmitriiBobreshev ! Thanks for the reply
The account running the service is an admin of the collection. I guess it should have all the required permission by default.
The 3.239.1 agent is installed by hand from the zip from github (no automated upgrade from Azure DevOps. Azure DevOps and the agent have very limited internet access)

[tisonv]

I tested the 3.240.1 and the problem is no more.
I suspect this has something to do with 04e4b98 v3.239.1...v3.240.1
There is no detail as why it was reverted though

[tisonv]

@DmitriiBobreshev

We skipped 3.241.1.
We tried to install 3.242.1 and the problem is back.

in the Agent log, I get the following warnings. I don't know if they are linked.

[2024-08-05 14:09:55Z WARN VisualStudioServices] Authentication failed with status code 401.
X-TFS-ProcessId: ...
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: ...
X-TFS-Session: ...
X-VSS-E2EID: ...
X-VSS-SenderDeploymentId: ...
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22fr%22%3ETF400813%3A%20ressource%20non%20disponible%20pour%20l%27acc%C3%A8s%20anonyme.%20L%27authentification%20client%20est%20requise.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20ressource%20non%20disponible%20pour%20l%27acc%C3%A8s%20anonyme.%20L%27authentification%20client%20est%20requise.
WWW-Authenticate: Bearer, Basic realm="https://XXXXX/", Negotiate, NTLM

[2024-08-05 14:09:55Z ERR  VisualStudioServices] GET request to https://XXXXX/_apis/FeatureFlags/DistributedTask.Agent.UseMaskingPerformanceEnhancements is not authorized. Details: TF400813: L'utilisateur n'est pas autorisé à accéder à cette ressource.
[2024-08-05 14:09:55Z WARN FeatureFlagProvider] Unable to retrieve feature flag with following exception: Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: L'utilisateur n'est pas autorisé à accéder à cette ressource.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Agent.Listener.Configuration.FeatureFlagProvider.GetFeatureFlagWithCred(IHostContext context, String featureFlagName, ITraceWriter traceWriter, AgentSettings settings, VssCredentials creds, CancellationToken ctk) in D:\a\_work\1\s\src\Agent.Listener\Configuration\FeatureFlagProvider.cs:line 62

We register our agents with PATas Negotiate.
There are no warning / error during the installation process.

As a test, we gave the account full privileges on the collection from the Azure DevOps console but no success.

[DmitriiBobreshev]

Hi @tisonv, seems like you have problems with certain git versions on windows machines, as I see, now, a default version of git which is provided by the agent is 2.45.2. Since the agent worked fine for you in version 3.240.1 where the default version was 2.39 let's try to run the pipeline under this version.
You can set USE_GIT_2_39_4 knob (as a pipeline or env variable) the agent should download this git version from blobstorage, so if you have firewall rules, please make sure that the url is not blocking. The agent will download this version with a first run and will continue to use it after. The PR with the changes is here.

From the other side, the error from the start of the topic seems related to the ssl certificate, so, could you please also try to configure agent with --sslskipcertvalidation option and with agent.gituseschannel variable. In that case http.sslVerify=false and http.sslbackend arguments will be passed to git.

[tisonv]

I activated the knob and it works ! Thank you !!!

Do you know what changed between 2.39.4 and the next versions that required a knob to be added ?
I can't access the work items describing the problem.
It may help on updating our configuration.

[DmitriiBobreshev]

@tisonv, the WI is about some mingit library dependency and most probably not related to your problem.
What I've found is that your error might be related to ssl certificate .
Could you please tell are you using self-signed certificate?
Is the certificate added to the Windows certificate storage on agents' machines and on the server?

[tisonv]

@DmitriiBobreshev
Historically our Azure DevOps was self signed until a couple years ago when the access was migrated through an appliance that signed queries from http to https with a common certificate.
There aren't anymore self signed certificate on the server (The old one expired in 07/2023).
I searched if there was any reference to this certificate (git config --system --list and the registry) but nothing to be found.

Could a configuration to be inherited through other other means ? The agents are not on the Azure DevOps server and are completely erased at each upgrade.

the WI is about some minigit library dependency and most probably not related to your problem.

Still weird the newest git versions give the problem and not the older ones though.