ASP NET Core Application gets OOMKilled by kubernetes because seems GC doesn't release swapped objects back to OS memory

[guilherme9718]

Hello folks! I've been studying dotnet GC very hard to accomplish solutions to this problem. But none of the responses on the internet is satisfying.

Context

I have an ASP. NET Core (latest .NET 10 docker image at 31/07/2026) application running in a kubernetes pod with Ubuntu 24.04.4 LTS. The application is running in multiples environemnts, the environment that's important in this scenario is the development environment. Due to resource saving, the vast majority of development applications runs under 256MiB memory limits, configured via kubernetes yaml. But, one of them is consuming more than 256MiB of RAM after one hour and eventually gets Out of Memory Killed by kubernetes/linux.

Studying a little how dotnet GC works (specially Server GC, while GC prioritises the throughput rather memory saving), I tested to define the most restrictive GC configurations to save memory and check if that changes something. I increase memory limit to 512MiB to testing and used the most restrictive parameters below.

DOTNET_gcServer=0 # Workstation GC
DOTNET_GCHeapHardLimitPercent=A # 10% of 512MiB = 52MiB
DOTNET_GCHeapCount=1 # Works only for server GC, but with server GC the same test with 1 heap yield pretty the same results
DOTNET_GCConserveMemory=9 # GC will compact LOH
DOTNET_gcConcurrent=0 # Maybe Non-concurrent GC can save a little memory?

I made a binary search to check which are the minimum heap that my application needs without throwing dotnet OutOfMemoryException, then I found that is about 50MiB. On 52Mib HeapHardLimitPercent GC begins to work very hard on load testing, it collect more gen 1 than gen 0 and almost doesn't let fragmentation exceed 5Mib. As desired, the gen2 collections and pause times are high too. The frustation comes when this don't reflect on working set memory.

To do low and high load testing, I increase pod memory to 512MiB and run dotnet-counters. The results are the heap never gets bigger than 50MiB (check dotnet counters bellow) while the working set is about 400Mib and in linux cgroup current memory are in 309MiB (I don't know why working set and cgroup metrics are different)*. The reason of my discontentment is that the peak memory of the heap plus memory of the executable IL plus fragmentation is about three times less than the cgroup current memory or working set of process. The responsiveness of api is the same if I put this aggressive GC heap limit, all responses are successfull and with good average response time.

*When I put a 256MiB pod memory limit, seems that the cgroup metric is taken in account. If the pod exceed this, it get OOMKilled. I collect with command cat /sys/fs/cgroup/memory.current.

Other thing that a I noticed is that when I do low load testing (10VU) the pod memory (cgroup current memory) increases and stabilizes under 220MiB. Also, when I increase the load testing (50VU) the memory stabilizes at 280MiB, as expected VU increases memory should increase too. However, the memory never gets back to 220Mib or even 270MiB after high load testing. Seems like GC never decommit memory to OS, it is commiting to increase throghput.

This memory saving is very important because the cloud memory cost in Brazil is not ignorable. There are Hundreds of dotnet applications at development environment owned by my company that could consume much less memory.

Other simlar answers on the internet that don't fit in my scenario

GC releases memory but the OS has the work to reclaim it when are memory pressure

Right, if the GC are in fact decommiting memory to OS, why the OS are OOM killing the pod rather than reclaming memory? Other fact is that cloud server doesn't tends to have swap memory, so if GC doesn't decommit virtual memory, that memory is wasted. Is it possible to check if is the OS that is not reclaiming the memory or GC?

GC prioritizes throughput rather than memory consumption

I agreed with that in production scenario. However, my scenario is in development, it is just to test correctness of an application, not performance or throughput. When we want to test performance, we increase resouces to be the same as production. I think GC must to have a config to waste less memory as possible as expense of cpu.

Memory Leak

My application is not leaking because the heap can stays under 50MiB forever without throwing OutOfMemoryException and analisyng memory dumps diffs there are nothing discrepant.

Questions

• What are the misterious commited 200MiB of memory that is not heap, stack, IL bytes loaded and fragmentation?
• Are there any way of reduce this application consumption to stays under 256MiB memory usage and not get OOMKilled, since GCHeapHardLimit doesn't affect the majority of commited memory?
• If is OS fault with cgroups, are there anything that I can do to increase OS reclaming or reduce memory usage?

I'll appreciate any collaboration, thank you all!

[janvorli]

There is also memory for JITted code and native memory that .NET or libraries it uses allocate. For example, we have seen cases when a lot of native memory was consumed by openssl that we use for SSL (HTTPS) and crypto stuff. In some of those cases, it was not because the native openssl code would hold on some memory blocks, but because of fragmentation of the C heap (malloc, new). We've seen even pathological cases where the C heap consumed more than a gigabyte of memory while its in-use memory was tens of megabytes only. The glibc allocator is susceptible to these fragmentation issues.

The GC limits really apply only to the GC allocated memory. The only way other allocated memory (like the native or possible other processes on the pod) influence GC is that GC measures the global memory pressure in the container and tries to be more aggressive when the pressure gets high).

You might also benefit from using server GC with DATAS. It tries to dynamically adapt to the memory needs of the application. See

• https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/datas
• https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#dynamic-adaptation-to-application-sizes-datas
• https://maoni0.medium.com/preparing-for-the-net-10-gc-88718b261ef2

I know I have not directly answered all your questions, but I hope this could help you trying more experiments. If you have a repro that you could share, I'll be happy to take a look.

[guilherme9718]

@janvorli Thanks for your answer!

I didn't realize that this issue could be libraries allocating native memory, even non dotnet libraries. I'm begginer to memory optimization or memory profiling. Is it possible/viable to investigate which each region of working set belongs to? To view that managed heap is these chunks ABC of memory, these others memory chunks CDE is JITed code and these others chunks is probably native memory etc. Also, are there a tool to view native memory fragmentation?

I'm thinking to make an experiment with same libraries but without private code, so you and others could repro.

I read about DATAS, it is very interesting, I presumed that is default enabled in .NET 10