Unexplained rise in storage activity causing higher costs

[laveryn]

We have multiple deployments of a Functions App across several tenants. Since the 1st November we have seen a sudden rise in transactions across all storage accounts for these apps, without any clear cause, including on deployments which are idle. This has cause a large increase in costs. The bug has affected different versions of the app, with different runtime versions, in similar ways

The fall in transactions shows where I have stopped the relevant function app.

The issue seems similar to those described in Azure/Azure-Functions#2231 and in Azure/Azure-Functions#2417, although in our case the majority of transactions are queue transactions.

The apps shown above are v4, .Net 6 (LTS) In-Process, running on Windows, with individual Consumption plans. The problem also occurred with a version of the app in v3. The apps do not use the AzureWebJobsDashboard setting.

[SpyroxQ]

@nzthiago

Dear Azure-Functions team, we really need some help with this issue. We have multiple deployments of a Functions App across several tenants. Two of those tenants are productive in a customer environment. Is their any way to escalate this ticket or do we need to give you more information?

Thank you very much.

[Sajith-TalentzQ]

We face similar issue from Nov 1st - and have logged a support request with Microsoft support

In our case , we have this issue with durable functions app deployment (on its storage queues) - Other function app deployments is fine.

Our Durable Function App deployment is in consumption plan - isolated mode.

[simon-tooley-t4a]

We're experiencing the same issue. It appears to be the Queue Peek Message and GetQueueMetadata calls.

[simon-tooley-t4a]

We've performed some more investigations as well as having a support ticket open with Microsoft. It appears that the increase in queue transactions correlates with the number of durable activities and orchestrators within the function app, something that wasn't happening previously.

If there's a total of 10 activities and orchestrators we are seeing 10 times the number of transactions, compared to only one, or compared to the prior Oct 31st Azure update.

[Sajith-TalentzQ]

Our issue is resolved by Microsoft by running a patch on the servers till they permanently fix the issue.

So for all those who are facing this issue - It is not really your problem. Just contact Microsoft and they will resolve it.

[SpyroxQ]

@Sajith-TalentzQ so actually we can´t open a support ticket because our subscription plan is not high enough.
You can write everything down and continue the process until this error message appears.

Do you got any reference or contact to microsoft for us? I can´t see any way to contact Microsoft directly?

Thank you very much for your help, our problem cant be resolved at the moment

[Sajith-TalentzQ]

Mostly its a generic email is - [email protected] which is of no use to you.

What we did is that - we took a min support ticket - something like a 25$ and then logged it.

There is a workaround though - You can downgrade the storage to V1 from V2 - This will reduce the cost by 1/10th

However this will not reduce the spike in transactions - but will only reduce the cost associated with a cheaper storage.

Otherwise the only option is patch update and then you can file for a refund. ( I am told we are also eligible for a refund which we have filed now )

[pragnagopa]

Thanks for reporting the issue. We have identified that this issue is affecting Function Apps with Durable triggers.

Will transfer the issue to https://github.com/Azure/azure-functions-durable-extension

We are actively working on the fix and plan to roll it out starting this week

[pragnagopa]

Addressed in #2967

Pending roll out

[AnatoliB]

The fix is rolled out, so issue is resolved on the Azure Functions service side now.

However, if your app is configured to use runtime-driven scaling (a.k.a. "Runtime Scale Monitoring" in the Function app configuration on the Portal), one additional step is required:

• If your app is using the dotnet isolated model, please upgrade your Microsoft.Azure.Functions.Worker.Extensions.DurableTask to v1.2.2 and redeploy the app.
• Otherwise, please upgrade your Microsoft.Azure.WebJobs.Extensions.DurableTask to v3.0.2 and redeploy the app.

[Saibamen]

@AnatoliB: And what about Azure/Azure-Functions#2417?

And which version of Microsoft.Azure.Functions.Worker.Extensions.DurableTask introduced this bug?

[AnatoliB]

@Saibamen This issue is specifically about excessive GetMessage operations on Azure Queues in the context of Durable apps. The issue you are referring to seems to be about blob operations and a non-Durable app, so this is not related. For the other issue, I would recommend filing an Azure support ticket.

The bug was introduced in:

• Microsoft.Azure.Functions.Worker.Extensions.DurableTask v1.0.4
• Microsoft.Azure.WebJobs.Extensions.DurableTask v2.12.0

Please note that the versions above are relevant only if your app is configured to use runtime-driven scaling (a.k.a. "Runtime Scale Monitoring"). If your app is not using this feature (which is the default configuration), the code in question is on the Functions service side, so the versions of these assemblies in the app content do not matter for these purposes.