Intermittent failure when downloading artifacts

[MiYanni]

We use the OneOffInsertion.ps1 script to do our insertion into VS process. We have been seeing and issue with it where it fails to download the artifacts. Here are some recent runs with this failure:

• https://devdiv.visualstudio.com/DevDiv/_releaseProgress?releaseId=1193089&_a=release-environment-logs&environmentId=6072101
• https://devdiv.visualstudio.com/DevDiv/_releaseProgress?releaseId=1192212&environmentId=6067436&_a=release-environment-logs
• https://devdiv.visualstudio.com/DevDiv/_releaseProgress?releaseId=1186001&environmentId=6032876&_a=release-environment-logs
• https://devdiv.visualstudio.com/DevDiv/_releaseProgress?releaseId=1184103&environmentId=6022016&_a=release-environment-logs
• https://devdiv.visualstudio.com/DevDiv/_releaseProgress?releaseId=1186001&environmentId=6032876&_a=release-environment-logs

They usually appear in the Attempt 1 of deployment. We manually rerun the deployment when it fails:

Here is the callstack from one of the failures:

2021-10-14T13:01:27.1925369Z Downloading artifacts to C:\Users\AzDevOps\AppData\Local\Temp\1\Project_Systemmain\20211014.1
2021-10-14T13:03:28.0521799Z ##[error] System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
2021-10-14T13:03:28.0534447Z ##[debug]Processed: ##vso[task.logissue type=error] System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
2021-10-14T13:03:28.0536665Z    at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
2021-10-14T13:03:28.0537389Z    at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
2021-10-14T13:03:28.0537894Z    --- End of inner exception stack trace ---
2021-10-14T13:03:28.0538324Z    at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
2021-10-14T13:03:28.0539585Z    at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
2021-10-14T13:03:28.0541512Z --- End of stack trace from previous location where exception was thrown ---
2021-10-14T13:03:28.0542083Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2021-10-14T13:03:28.0542655Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2021-10-14T13:03:28.0543283Z    at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.<ReadAsync>d__4.MoveNext()
2021-10-14T13:03:28.0543848Z --- End of stack trace from previous location where exception was thrown ---
2021-10-14T13:03:28.0544371Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2021-10-14T13:03:28.0545026Z    at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
2021-10-14T13:03:28.0545526Z --- End of stack trace from previous location where exception was thrown ---
2021-10-14T13:03:28.0546048Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2021-10-14T13:03:28.0546661Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2021-10-14T13:03:28.0547284Z    at Roslyn.Insertion.RoslynInsertionTool.<DownloadBuildArtifactsAsync>d__51.MoveNext()
2021-10-14T13:03:28.0547850Z --- End of stack trace from previous location where exception was thrown ---
2021-10-14T13:03:28.0548384Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2021-10-14T13:03:28.0548926Z    at Roslyn.Insertion.RoslynInsertionTool.<GetInsertionArtifactsAsync>d__50.MoveNext()
2021-10-14T13:03:28.0549502Z --- End of stack trace from previous location where exception was thrown ---
2021-10-14T13:03:28.0550129Z    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
2021-10-14T13:03:28.0550704Z    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2021-10-14T13:03:28.0551304Z    at Roslyn.Insertion.RoslynInsertionTool.<PerformInsertionAsync>d__13.MoveNext()

@JoeRobich @RikkiGibson The failure seems to stem from here:

roslyn-tools/src/RoslynInsertionTool/RoslynInsertionTool/RoslynInsertionTool.VisualStudioTeamServices.cs

Line 419 in 7ba6e54

await s.CopyToAsync(ms);

[RikkiGibson]

Perhaps we could adjust some parameters in the client to make it retry more, etc.