k3s server crashes abruptly when approximately 300 requests are sent in a span of 60 seconds (client QPS is 5 and bursty limit is 10) #10418
Replies: 1 comment 1 reply
-
|
This is a crash DEEP in a core go garbage collection function. I do not believe that this is something we can address here in K3s. The fact that you see this under load, and the resulting stack is in the go runtime garbage collection routine suggests that it is running out of memory and trying to force GC, and that is somehow running into memory corruption while doing so ( ref golang/go#57550 (comment)). How much memory does this node have? How much of that is available to k3s, when your workload is present? Have you tried profiling k3s while updating your application? |
Beta Was this translation helpful? Give feedback.
-
|
There was atleast 2GB of free memory available on the host as the other process are pretty much idle , the overall mem on machine was 16GB, Also each time we see a crash we have a different stack trace |
Beta Was this translation helpful? Give feedback.
-
Environmental Info:
K3s Version:
1.30.1
Node(s) CPU architecture, OS, and Version:
Linux 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
one server one agent
Describe the bug:
K3s server crashes when helm upgrade of a chart is triggered , this basically means that the server crashes when sending around 300 api calls to the server
Steps To Reproduce:
Expected behavior:
k3s server should not crash
Actual behavior:
k3s server crashes
Additional context / logs:
This is the stack trace
Beta Was this translation helpful? Give feedback.
All reactions