Update-MgAdminSharepointSetting: Too many retries performed. More than 3 retries

[alpixdotro]

Describe the bug

I am trying to update Sharepoint Tenant settings using either on of the following cmdlets:

Getting the same issue with:
$Uri = "https://graph.microsoft.com/v1.0/admin/sharepoint/settings"
Invoke-MgGraphRequest -Uri $Uri -Method Patch -Body $updateBody

but also with:
Update-MgAdminSharepointSetting -SharingAllowedDomainList $allowedDomains
Update-MgAdminSharepointSetting_UpdateExpanded: Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: GatewayTimeout.▼♦
T�1♫�0♀@ѻx���MH��☺�ؠ
HEZ�P��0������65H+�I¶↕��N�z�f♥☼�g��♠�Z��$� ��,y���?Q�(&k�@��K�♣G�♦Y�►Y∟F-↓����3�%�B��g;��r▼�.��♠!V▲∟v�#:>(�‼�D.�}(♫�m��♥H���) (HTTP request failed with status code: GatewayTimeout.▼♦
T�1♫�0♀@ѻx���4M��☺�Ġ
HEZ�P��(0������:W�9+D8�[�����↔<tY��♠t0���w�♠Y��ɐCCh�l$��BU�/]V�2D��→�F�^t��.♥�!dd▲%���]3�>iY�&b�∟,R2��▲��<��1H�F↕���▼��♥��↨��) (HTTP request failed with status code: GatewayTimeout.▼♦
T�1♫�0♀@ѻx�%7nK��☺\Ǡ
HEZ�P��0������:UH+�↕�˭L�r�f☼�g��♫�R��$��eٻ'�!y$>yNL�"8��|ټ��!☺w↕%�▲E��-��►.◄)�5M eUp���ʂ⌂�����♀Y���႑��h�RC∟�♀�m���♥ʭ�S�) (HTTP request failed with status code: GatewayTimeout.▼♦
T�9♫�0►@ѻL��+�{n�2F◄'�"��◄P����☺�1☼��2D8�k�_���☻�,��☺►0���'q���OW�,�B�'���h↑�x��T!☻9%����x�FꂹI���b��e[(����⌂+�F��C�أ�Abn-aR�Qac�☼����♥gс|�)

I've tried reinstalling Microsoft.Graph modules, attempted in both Poweshell 5+ and 7+ as well.
Same things happens if script is run in Azure Automation runbook.

Microsoft.Graph version 2.25 and 2.24 same issue

Expected behavior

Expected behavior should be long running time ( 1200+ domains ) but then list updated, no error

How to reproduce

1. Connect-MgGraph -TenantId $TenantId -ClientSecretCredential $Global:ClientCredentials
2. $Uri = "https://graph.microsoft.com/v1.0/admin/sharepoint/settings"
3. $SPOSettings = Invoke-MgGraphRequest -Uri $Uri -Method Get
4. $allowedDomains = $SPOSettings.sharingAllowedDomainList
5. $allowedDomains+="contoso.com"
6. $updateBody = @{
   sharingAllowedDomainList = $allowedDomains
   }
7. Invoke-MgGraphRequest -Uri $Uri -Method Patch -Body $updateBody

or

1. Connect-MgGraph -TenantId $TenantId -ClientSecretCredential $Global:ClientCredentials
2. $SPOSettings = Get-MgAdminSharepointSetting
3. $allowedDomains = $SPOSettings.sharingAllowedDomainList
4. $allowedDomains+="contoso.com"
5. Update-MgAdminSharepointSetting -SharingAllowedDomainList $allowedDomains -Debug

or

Either one of the above in Azure Automation runbook Powershell version 7.2

SDK Version

2.25.0,2.24.0,Beta

Latest version known to work for scenario above?

No response

Known Workarounds

no workaround with Microsoft Graph
issue persists with Microsoft.Graph.Beta.Sites

This is not reliable at all, even with Retry-After 240, it errors out, although it adds the domains.

Same error with 600 Retry-After:

Debug output

Click to expand log

```

DEBUG: [CmdletBeginProcessing]: - Update-MgAdminSharepointSetting begin processing with parameterSet 'UpdateExpanded'.
DEBUG: [Authentication]: - AuthType: 'AppOnly', TokenCredentialType: 'ClientSecret', ContextScope: 'Process', AppName: 'ExternalDomainWhitelisting-Automation-26664'.
DEBUG: [Authentication]: - Scopes: [SharePointTenantSettings.ReadWrite.All, Policy.Read.All].

Confirm
Are you sure you want to perform this action?
Performing the operation "Update-MgAdminSharepointSetting_UpdateExpanded" on target "Call remote 'PATCH
/admin/sharepoint/settings' operation".
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): y
DEBUG: [CmdletException]: Received exception with message 'AggregateException - Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: GatewayTimeout.▼♦
T�K♫�0►��̺���☺�{o��c0D-�\►�nЕ��˿��♦��[����� ▲�,�z(�j����A���5i����Y�l ☻♣M�/YV�
♦�l<�L�s↕��/�됽�E��{�AA�ORW�[m�E�f@�FF�c��d��↑72��7�����♥�♂p��) (HTTP request failed with status code: GatewayTimeout.♦�rZ�N�<ZB�g��.2���ϗ.+N↓"�4��♠���zt�g¶�♫��♀6p�^��x�����z� NI�♥��T�♂u♥:�♣ŕ�R�♠K∟�
���☺��♥�jSD�) (HTTP request failed with status code: GatewayTimeout.▼♦
T�A♫� ►@ѻ̚I�)
콁▲��4*���Eӻ����⌂�+H�S��B��@�K����N߬�!���↔@�ؚ���§ �셂�8�♥��♥pה]�) (HTTP request failed with status code: GatewayTimeout.▼♦
T�A♫�0►@ѻ̚IJۡC��@☼P:�!j�♣�pw��������:W�Y¶"\ʭ��r��♠▲�,�z40���'q♥I�ѭ��Ƣqg뢣H-4P���e�I ��ɉY1�c��►�↑A&����◄34�§��<dV
☺S�♣���gjQT�↑�q��þ�▼��♥p�}��) : at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendRetryAsync(HttpResponseMessage response, RetryHandlerOption retryOption, CancellationToken cancellationToken, ActivitySource activitySource)
at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.CompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Graph.PowerShell.Authentication.Handlers.AuthenticationHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Graph.PowerShell.Sites.AdminSharepointUpdateSetting_Call(HttpRequestMessage request, Func3 on2Xx, Func3 onDefault, IEventListener eventListener, ISendAsync sender)
at Microsoft.Graph.PowerShell.Sites.AdminSharepointUpdateSetting_Call(HttpRequestMessage request, Func3 on2Xx, Func3 onDefault, IEventListener eventListener, ISendAsync sender)
at Microsoft.Graph.PowerShell.Sites.AdminSharepointUpdateSetting(IDictionary headers, IMicrosoftGraphSharepointSettings body, Func3 on2Xx, Func3 onDefault, IEventListener eventListener, ISendAsync sender)
at Microsoft.Graph.PowerShell.Cmdlets.UpdateMgAdminSharepointSetting_UpdateExpanded.ProcessRecordAsync()'
Update-MgAdminSharepointSetting_UpdateExpanded: Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: GatewayTimeout.▼♦
T�K♫�0►��̺���☺�{o��c0D-�\►�nЕ��˿��♦��[����� ▲�,�z(�j����A���5i����Y�l ☻♣M�/YV�
♦�l<�L�s↕��/�됽�E��{�AA�ORW�[m�E�f@�FF�c��d��↑72��7�����♥�♂p��) (HTTP request failed with status code: GatewayTimeout.♦�T�1♫�0♀@ѻx���N�8;7�♥D��* §i◄Cջ������♠��� n0�Y!¥�����o6��eI�♥��T����rZ�N�<ZB�g��.2���ϗ.+N↓"�4��♠���zt�g¶�♫��♀6p�^��x�����z� N ♂u♥:�♣ŕ�R�♠K∟�
���☺��♥�jSD�) (HTTP request failed with status code: GatewayTimeout.▼♦
T�A♫� ►@ѻ̚I�)
콁▲��4*���Eӻ����⌂�+H�S��B��@�K����N߬�!���↔@�ؚ���§ /{��:�▬5�-ErQP����y��@♦�&��♥�T☻♫�;�s��.I�♂�4(��Qڂ⌂k���,§3��♥�#r�셂�8�♥��♥pה]�) (HTTP request failed with status code: GatewayTimeout.▼♦
T�A♫�0►@ѻ̚IJۡC��@☼P:�!j�♣�pw��������:W�Y¶"\ʭ��r��♠▲�,�z40���'q♥I�ѭ��Ƣqg뢣H-4P���e�I ��ɉY1�c��►�↑A&����◄34�§��<dV
☺S�♣���gjQT�↑�q��þ�▼��♥p�}��)
DEBUG: [CmdletEndProcessing]: - Update-MgAdminSharepointSetting end processing.

Confirm
Continue with this operation?
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): y
DEBUG: [CmdletException]: Received exception with message 'AggregateException - Too many retries
performed. More than 3 retries encountered while sending the request. : at
Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.CompressionHandler.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at
Microsoft.Graph.PowerShell.Authentication.Handlers.AuthenticationHandler.d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.Beta.PowerShell.Sites.<AdminSharepointUpdateSetting_Call>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Graph.Beta.PowerShell.Sites.<AdminSharepointUpdateSetting_Call>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.Beta.PowerShell.Sites.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at
Microsoft.Graph.Beta.PowerShell.Cmdlets.UpdateMgBetaAdminSharepointSetting_UpdateExpanded.d__148.MoveNext()'

Confirm
Continue with this operation?
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): y

Confirm
Too many retries performed. More than 3 retries encountered while sending the request.
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): y
Update-MgBetaAdminSharepointSetting : Too many retries performed. More than 3 retries encountered while
sending the request.
At line:1 char:1

• Update-MgBetaAdminSharepointSetting -SharingAllowedDomainList $allowe ...

•   + CategoryInfo          : NotSpecified: (:) [Update-MgBetaAd..._UpdateExpanded], AggregateException
    + FullyQualifiedErrorId : Microsoft.Graph.Beta.PowerShell.Cmdlets.UpdateMgBetaAdminSharepointSetting_
   UpdateExpanded
  
  

DEBUG: [CmdletEndProcessing]: - Update-MgBetaAdminSharepointSetting end processing.

Confirm
Continue with this operation?
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): y
PS C:\Users\laszl>

</details>


### Configuration

- OS: Windows 10
- Powershell 5.1
- Powershell 7.4.6

or 

- Azure Automation
- PowerShell 7.2

### Other information

_No response_

[timayabi2020]

Hi @alpixdotro, I wasn't able to reproduce the issue using the provided steps.

.

Please try using this tool https://developer.microsoft.com/en-us/graph/graph-explorer and let me know if you are experiencing the same issue.

[alpixdotro]

Hi @alpixdotro, I wasn't able to reproduce the issue using the provided steps.

.

Please try using this tool https://developer.microsoft.com/en-us/graph/graph-explorer and let me know if you are experiencing the same issue.

Hello @timayabi2020 , thanks for reaching out.

The problem is not with 1 domain adding, or even with 50 domains.
The problem I am facing is with 1263 domains.

Please try again with 1263 random domains.
Here's a script to easiliy generate you 1263 domains:
# Generate 1263 random domains $allowedDomains = 1..1263 | ForEach-Object { "contoso$_.com" }

In production environments, customers have more than 1 domains, and the FLAW in Sharepoint External Domain Allow List is that it does not have ADD or REMOVE, the only way to add 1 domain to the existing 1263 si to save the whole domain list to variable, add 1 and then overwrite old list with new list in SPO Settings.

[alpixdotro]

Also @timayabi2020 , to answer the question, here is test with Graph Explorer:

Error is even worse in Graph Explorer:

Same with BETA endpoint:

[timayabi2020]

@alpixdotro thanks for the feedback. Based on the different environments (SDK, and Graph explorer tool) where the request was tried, I can conclude that this is more of a service issue than an SDK issue. For service/API related issues/questions we are not best placed to give an answer. Kindly raise an issue here https://developer.microsoft.com/en-us/graph/support so that the API owner can respond to it.

[alpixdotro]

Not really helpful:

After spending 20 minutes creating a Q&A request, it gets automatically deleted. Really ? This is the solution to pass the ball to someone else ?

[timayabi2020]

@alpixdotro the link I've shared is the correct channel to raise tickets for errors coming from the API. I've also stated that we have no visibility for the various APIs that different cmdlet calls. Please refer to our troubleshooting guide here which describes the list of parameters required for the graph Api support team to be able to troubleshoot your issue.

[alpixdotro]

Can this request be raised not linked to a tenant ?
Development is done in test tenant and CDX tenant, therefore - Azure request is not applicable only for 21$/month .

[timayabi2020]

CC: @CarolKigoonya, @sebastienlevert