Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Durable Functions Memory Issues After Isolated #1173

Open
brandonwatts opened this issue Oct 7, 2024 · 4 comments
Open

Durable Functions Memory Issues After Isolated #1173

brandonwatts opened this issue Oct 7, 2024 · 4 comments

Comments

@brandonwatts
Copy link

brandonwatts commented Oct 7, 2024 •
edited
Loading

Hello, I was wondering if someone could help me on some issues I am running into. We have a durable python function app that we have been running via a docker image for a long time now and everything has worked great. Recently after returning to the application we are not able to run it locally anymore. The application will start up and then fail due to memory issues. After some digging I think it may have been this PR that caused the issues. Im not certain but its my best guess. I thought it was just a local issue but my entire team is running into the same issue. We are all on M-Series Mac if that plays any role. Below are some of the issues. The application runs fine locally as in func start. This issue is isolated to just when running in the docker container.

Docker Engine v.27.2.0
azure-functions-durable==1.2.9
azure-functions==1.19.0

dockerfile:

FROM mcr.microsoft.com/azure-functions/python:4-python3.11

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

COPY requirements.txt /
RUN pip install -r /requirements.txt

COPY . /home/site/wwwroot

info: Host.Startup[412]
Host initialized (207ms)
info: Host.Triggers.DurableTask[0]
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at System.Runtime.CompilerServices.CastHelpers.IsInstanceOfAny(Void*, System.Object)
at System.Collections.Generic.List1[[System.ValueTuple2[[Azure.Core.HttpPipelinePosition, Azure.Core, Version=1.38.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Collections.Generic.IEnumerable1<System.ValueTuple2<Azure.Core.HttpPipelinePosition,System.__Canon>>)
at Azure.Core.ClientOptions..ctor(Azure.Core.ClientOptions, Azure.Core.DiagnosticsOptions)
at Azure.Identity.DefaultAzureCredentialOptions..ctor()
at Azure.Identity.DefaultAzureCredentialFactory..ctor(Azure.Identity.DefaultAzureCredentialOptions, Azure.Identity.CredentialPipeline)
at Microsoft.Extensions.Azure.AzureClientsGlobalOptions+<>c.<.ctor>b__11_0(System.IServiceProvider)
 at Microsoft.Azure.WebJobs.Script.StorageClientProvider2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Create(System.String, Microsoft.Extensions.Configuration.IConfiguration, System.__Canon)
at Microsoft.Azure.WebJobs.Script.HostAzureBlobStorageProvider.TryCreateBlobServiceClientFromConnection(System.String, Azure.Storage.Blobs.BlobServiceClient ByRef) at Microsoft.Azure.WebJobs.Script.HostAzureBlobStorageProvider.TryCreateHostingBlobContainerClient(Azure.Storage.Blobs.BlobContainerClient ByRef)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager.CreateBlobContainerClient(System.String)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager.GetContainerClient(System.String)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager+<TryLockAsync>d__9.MoveNext()
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager+<TryLockAsync>d__9, Microsoft.Azure.WebJobs.Host.Storage, Version=5.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<TryLockAsync>d__9 ByRef)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager.TryLockAsync(System.String, System.String, System.String, System.String, System.TimeSpan, System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.SingletonManager+<TryLockAsync>d__19.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.SingletonManager+<TryLockAsync>d__19, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<TryLockAsync>d__19 ByRef)
at Microsoft.Azure.WebJobs.Host.SingletonManager.TryLockAsync(System.String, System.String, Microsoft.Azure.WebJobs.SingletonAttribute, System.Threading.CancellationToken, Boolean)
at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener+<StartAsync>d__13.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener+<StartAsync>d__13, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsync>d__13 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.StartAsync(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+<StartAsync>d__13.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+<StartAsync>d__13, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsync>d__13 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(System.Threading.CancellationToken, Boolean)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+<StartAsync>d__12.MoveNext()
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+<StartAsync>d__12, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsync>d__12 ByRef)
 at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(System.Threading.CancellationToken)
 at Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener+<StartAsync>d__4.MoveNext()
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener+<StartAsync>d__4, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsync>d__4 ByRef)
 at Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener.StartAsync(System.Threading.CancellationToken)
 at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener+<StartAsyncCore>d__8.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener+<StartAsyncCore>d__8, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsyncCore>d__8 ByRef)
 at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener.StartAsyncCore(System.Threading.CancellationToken)
 at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener+<StartAsync>d__5.MoveNext()
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener+<StartAsync>d__5, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsync>d__5 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener.StartAsync(System.Threading.CancellationToken)
 at Microsoft.Azure.WebJobs.JobHost+<StartAsyncCore>d__23.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.JobHost+<StartAsyncCore>d__23, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](<StartAsyncCore>d__23 ByRef)
at Microsoft.Azure.WebJobs.JobHost.StartAsyncCore(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Script.ScriptHost+<StartAsyncCore>d__97.MoveNext()
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.ScriptHost+<StartAsyncCore>d__97, Microsoft.Azure.WebJobs.Script, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](<StartAsyncCore>d__97 ByRef)
at Microsoft.Azure.WebJobs.Script.ScriptHost.StartAsyncCore(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Hosting.JobHostService.StartAsync(System.Threading.CancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host+<<StartAsync>b__15_1>d.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Extensions.Hosting.Internal.Host+<<StartAsync>b__15_1>d, Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<<StartAsync>b__15_1>d ByRef)
 at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(Microsoft.Extensions.Hosting.IHostedService, System.Threading.CancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host+<ForeachService>d__181[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Extensions.Hosting.Internal.Host+<ForeachService>d__181[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<ForeachService>d__181<System.__Canon> ByRef)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable1<System.__Canon>, System.Threading.CancellationToken, Boolean, Boolean, System.Collections.Generic.List1<System.Exception>, System.Func3<System.__Canon,System.Threading.CancellationToken,System.Threading.Tasks.Task>)
 at Microsoft.Extensions.Hosting.Internal.Host+<StartAsync>d__15.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Extensions.Hosting.Internal.Host+<StartAsync>d__15, Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<StartAsync>d__15 ByRef)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(System.Threading.CancellationToken)
  at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<UnsynchronizedStartHostAsync>d__58.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<UnsynchronizedStartHostAsync>d__58, Microsoft.Azure.WebJobs.Script.WebHost, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](<UnsynchronizedStartHostAsync>d__58 ByRef)
    at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(Microsoft.Azure.WebJobs.Script.WebHost.ScriptHostStartupOperation, Int32, JobHostStartupMode)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<StartHostAsync>d__57.MoveNext()
    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<StartHostAsync>d__57, Microsoft.Azure.WebJobs.Script.WebHost, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](<StartHostAsync>d__57 ByRef)
  at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.StartHostAsync(System.Threading.CancellationToken, Int32, JobHostStartupMode, System.Nullable1<System.Guid>)
 at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<StartAsync>d__54.MoveNext()
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<StartAsync>d__54, Microsoft.Azure.WebJobs.Script.WebHost, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](<StartAsync>d__54 ByRef)
  at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.StartAsync(System.Threading.CancellationToken)
  at Microsoft.AspNetCore.Hosting.HostedServiceExecutor+<ExecuteAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.HostedServiceExecutor+<ExecuteAsync>d__5, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<ExecuteAsync>d__5 ByRef)
  at Microsoft.AspNetCore.Hosting.HostedServiceExecutor.ExecuteAsync(System.Func2<Microsoft.Extensions.Hosting.IHostedService,System.Threading.Tasks.Task>, Boolean)
  at Microsoft.AspNetCore.Hosting.WebHost+<StartAsync>d__27.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.WebHost+<StartAsync>d__27, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<StartAsync>d__27 ByRef)
   at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(System.Threading.CancellationToken)
  at Microsoft.AspNetCore.Hosting.WebHostExtensions+<RunAsync>d__5.MoveNext()
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.WebHostExtensions+<RunAsync>d__5, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<RunAsync>d__5 ByRef)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(Microsoft.AspNetCore.Hosting.IWebHost, System.Threading.CancellationToken, System.String)
  at Microsoft.AspNetCore.Hosting.WebHostExtensions+<RunAsync>d__4.MoveNext()
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.WebHostExtensions+<RunAsync>d__4, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<RunAsync>d__4 ByRef)
 at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(Microsoft.AspNetCore.Hosting.IWebHost, System.Threading.CancellationToken)
  at Microsoft.Azure.WebJobs.Script.WebHost.Program.Main(System.String[])
 /opt/startup/start_nonappservice.sh: line 4:     9 Aborted                 /azure-functions-host/Microsoft.Azure.WebJobs.Script.WebHost
contentapi exited with code 134

info: Host.Startup[413]
Host started (601ms)
Host.Startup[0]
Job host started
assertion failed [block != nullptr]: BasicBlock requested for unrecognized address
(BuilderBase.h:550 block_for_offset)
 /opt/startup/start_nonappservice.sh: line 4:     8 Trace/breakpoint trap   /azure-functions-host/Microsoft.Azure.WebJobs.Script.WebHost
@brandonwatts
Copy link
Author

brandonwatts commented Oct 7, 2024 •
edited
Loading

Ok I definitely think something was introduced. I changed my dockerfile to point at the old version and memory issues went away. No error message or issues, everything runs fine. New dockerfile:

FROM mcr.microsoft.com/azure-functions/python:4.35.1.2-python3.11

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

COPY requirements.txt /
RUN pip install -r /requirements.txt

COPY . /home/site/wwwroot

@pragnagopa I think this might be worth looking into. Im willing to share my findings and help out how I can but I think there is an issue here somewhere.

@brandonwatts
Copy link
Author

@unfixa1 Docker does support running x86-64 binaries natively via Rosetta. I also moved to Orbstack and the problem persisted there, just to rule out that it could be Docker Desktop. I will test on an intel system at some point today. Downgrading to 4.35 is the only thing that will not cause the application to crash.

@brandonwatts
Copy link
Author

I was able to run the project on a windows intel-based system no problem. Debugging further into this issue is out of my level of expertise but what I know is that after the 4.36 update I cannot run a durable function app via a docker image on an m-series machine where I was able to before this update. We will move forward with just reverting to the old version for now and hopefully a future update can remedy this issue.

@ollenorennilimaa
Copy link

@brandonwatts We have also been seeing the same problem since the update to 4.36, also running on Mac M-Series. And the workaround is the same, to stay on 4.35 until this is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brandonwatts @ollenorennilimaa and others