Umbraco Cloud Deployment Failure: error in sideband demultiplexer

[steveo101]

Issue description

Umbraco Cloud Deployments have started randomly failing with the following error:

Failed to push to the remote repository.
send-pack: unexpected disconnect while reading sideband packet
Error: error in sideband demultiplexer
To https://scm.umbraco.io/aueast01/xxxx.git
389af10..4162ee2 master -> master
Error encountered while pushing to the remote repository: PushCommand.ExecutePushCommand

The lock file via Kudo contains the following error:

{
  "OperationName": "Handling git receive pack",
  "AcquiredDateTime": "2024-12-12T05:50:38.5214199Z",
  "StackTrace": "   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)\r\n   at System.Environment.get_StackTrace()\r\n   at Kudu.Core.Infrastructure.LockFile.WriteLockInfo(String operationName, Stream lockStream)\r\n   at Kudu.Core.Infrastructure.LockFile.Lock(String operationName)\r\n   at Kudu.Contracts.Infrastructure.LockExtensions.TryLockOperation(IOperationLock lockObj, Action operation, String operationName, TimeSpan timeout)\r\n   at Kudu.Contracts.Infrastructure.LockExtensions.LockOperation(IOperationLock lockObj, Action operation, String operationName, TimeSpan timeout)\r\n   at Kudu.Services.GitServer.ReceivePackHandler.ProcessRequestBase(HttpContextBase context) in C:\\__w\\1\\s\\Kudu.Services\\GitServer\\ReceivePackHandler.cs:line 74\r\n   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()\r\n   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)\r\n   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)\r\n   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)\r\n   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)\r\n   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)\r\n   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)\r\n   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)\r\n   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)\r\n   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)\r\n   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)\r\n   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)",
  "InstanceId": "4c8c45"
}

I have deployments hooked up via CI in Azure DevOps using the scripts posted on Umbraco docs. These scripts have been working well for over 12 months on multiple Umbraco Cloud projects.

Only one of my Umbraco Cloud projects is experiencing this issue which began roughly 2 weeks ago.

I've resorted to removing the lock file and temp deployment log files via kudu to run the deployment multiple times to get it to complete. I've also attempted to push the update directly to the Umbraco Cloud repository from Visual Studio for it to fail as well in the same manner.

Eventually, after multiple deployment attempts, the deployment will succeed.

I have been making minor changes in the appsettings.json file and committing this change to force the deployments.

After advice from Umbraco support, I have attempted to execute the deployments via PowerShell script using the scripts provided from the Umbraco.Cloud.CICDFlow project.

That also comes back indicating the deployment is successful however the lock file remains with the same error shown above.

[steveo101]

I now have this same issue occurring on multiple Umbraco Cloud projects.

[jattwood]

Steve, thanks for starting this post. This issue has driven me crazy for over 6 months on USEast. At one point it got so bad we had to move to dedicated as I simply could not move anything from local to remote dev. After 6 weeks of deployment hell on dedicated with 1 hour+ deployment times, we were forced to move back to shared on a clean repo (rebuild the project from the ground up). This worked for about 2 weeks and then that error started creeping up again. Now I am seeing this at least 1 to 2 times per day. The only "workaround" is to do fake pushes until it takes.

I have done a lot of research on this and everything points to networking issues. More resources needed to be assigned to deploy and SiteExtensions needs to be able to handle this error and carry-on. There is no issue with GIT because the push lands successfully on cloud as indicated by the fact that there is a queued deployment sitting on the dev portal - it's just that the deployment is never triggered.

There are a lot of people experiencing this issue: https://our.umbraco.com/forum/using-umbraco-and-getting-started/114527-umbraco-cloud-manually-trigger-build-and-deploy-without-git-check-in