[BUG]: "Server failed to authenticate the request" error when running VMSS agent provisioning due to SAS token expiration for CustomScriptExtension

[DevOpsAzurance]

What happened?

We have been using VMSS Pools for the past 6 months. When we initially provisioned the integration between ADO and VMSS, the service account created for the integration took over the VMSS and installed two extensions:

 Microsoft.Compute.CustomScriptExtension
 Microsoft.VisualStudio.Services.TeamServicesAgent

After about 6 months of running fine it has started failing with:

[3192+00000001] [08/06/2024 21:20:54.00] [INFO] Downloading files specified in configuration...
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] targetFileName 'Post-Gen-Win.ps1' was parsed for Azure blob uri fileUri_{0}
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] DownloadFiles: fileDownloadPath = C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.17\Downloads\0
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] WebClientDownloader: starting download fileUri = fileUri_{0}
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] WebClientDownloader: Client request ID = 88bcb8b9-b3aa-49a8-9b88-e0aed2dfa3ea
[3192+00000001] [08/06/2024 21:20:54.83] [WARN] WebClient: non retryable error occurred System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.WebClient.DownloadFile(Uri address, String fileName)
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)
[3192+00000001] [08/06/2024 21:20:54.88] [ERROR] DownloadFiles failed: CustomScript failed to download the blob fileUri_{0} because the server returned response code: "The remote server returned an error: (403) Forbidden." Message: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.". Please verify the machine has network connectivity (Service request ID = 3f301104-901e-0005-6046-e883fe000000).
[3192+00000001] [08/06/2024 21:20:54.88] [FATAL] Failed to download all specified files. Existing. Exception: CustomScript failed to download the blob fileUri_{0} because the server returned response code: "The remote server returned an error: (403) Forbidden." Message: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.". Please verify the machine has network connectivity (Service request ID = 3f301104-901e-0005-6046-e883fe000000).
[3192+00000001] [08/06/2024 21:20:54.90] [INFO] {"sequenceNumber":0,"totalDuration":1077,"status":"error","code":1,"filesDownloaded":0,"durationOfFileDownload":820,"durationOfDownloadHashing":0,"statusFileAccessed":2,"operation":"enable","operationDuration":7,"operationResult":"success","operationSizeOfOutput":0,"operationSizeOfError":0,"sha256HashOfCommandToExecute":null,"downloadSummary":null}
[3192+00000001] [08/06/2024 21:20:54.90] [INFO] Event processing is terminating...

Network is not an issue, as it is open outbound, but seems like the SAS token/key that was used when the CSE was installed is expired. Since that extension setting is protected, not a public setting, we have no idea what the blob uri is or the sas for it. We cannot create a new SAS token, since we don't know the storage account, nor have access to it, most likely it is an MS storage account. As such we would not be able to use a managed identity either as, it is not our storage account.

This seems to be an issue with the Microsoft.VisualStudio.Services.TeamServicesAgent extension that it depends on this private blob, versus a public one like the public settings in that extension.

Versions

Windows 2022

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 2022

Version controll system

Git

Relevant log output

[3192+00000001] [08/06/2024 21:20:54.00] [INFO] Downloading files specified in configuration...
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] targetFileName 'Post-Gen-Win.ps1' was parsed for Azure blob uri fileUri_{0}
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] DownloadFiles: fileDownloadPath = C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.17\Downloads\0
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] WebClientDownloader: starting download fileUri = fileUri_{0}
[3192+00000001] [08/06/2024 21:20:54.66] [INFO] WebClientDownloader: Client request ID = 88bcb8b9-b3aa-49a8-9b88-e0aed2dfa3ea
[3192+00000001] [08/06/2024 21:20:54.83] [WARN] WebClient: non retryable error occurred System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.WebClient.DownloadFile(Uri address, String fileName)
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)
[3192+00000001] [08/06/2024 21:20:54.88] [ERROR] DownloadFiles failed: CustomScript failed to download the blob fileUri_{0} because the server returned response code: "The remote server returned an error: (403) Forbidden." Message: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.". Please verify the machine has network connectivity (Service request ID = 3f301104-901e-0005-6046-e883fe000000).
[3192+00000001] [08/06/2024 21:20:54.88] [FATAL] Failed to download all specified files. Existing. Exception: CustomScript failed to download the blob fileUri_{0} because the server returned response code: "The remote server returned an error: (403) Forbidden." Message: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.". Please verify the machine has network connectivity (Service request ID = 3f301104-901e-0005-6046-e883fe000000).
[3192+00000001] [08/06/2024 21:20:54.90] [INFO] {"sequenceNumber":0,"totalDuration":1077,"status":"error","code":1,"filesDownloaded":0,"durationOfFileDownload":820,"durationOfDownloadHashing":0,"statusFileAccessed":2,"operation":"enable","operationDuration":7,"operationResult":"success","operationSizeOfOutput":0,"operationSizeOfError":0,"sha256HashOfCommandToExecute":null,"downloadSummary":null}
[3192+00000001] [08/06/2024 21:20:54.90] [INFO] Event processing is terminating...

[vmapetr]

Hi @DevOpsAzurance thank you for your report!
This repo is specific for the ADO agent and according to the description your issue is related to the VMSS agent extension.
Could you please open an issue in Developer Community to get the right eyes on your issue?

[vmapetr]

I'm closing this one as external - feel free to ask any other questions or let us know if it's still relevant for you.