High Memory usage after upgrade to version 1.15.6 #28170
Comments
It was showing 2120Mi and after 19h when I checked Memory utilization increased to 2677Mi, something fishy in this version. |
|
And its keep on increasing |
|
Hi @kaleshag , thank you for reporting this! Do you have any logs and server configurations that you could provide us so that we could investigate? |
|
@divyaac I am not finding any logs on the vault and I can only see that suddenly huge spike after just upgrading vault to 1.15.6. |
|
I am just giving the memory information for the pod which shows higher memory |
|
Is there any changes in vault from 1.14.x onwards which causes for more memory utilization? |
|
I just collected some logs from hcdiag and attaching them for your reference, please let me know if you can help something with that. |
|
Hello Team Do we have any update on this issue? We are having a lot of issues with this. |
|
HI @kaleshag; it seems that the change is related to extra metrics logging. Is the high memory utilization consistent? Or just during startup? Also - what version of Vault were you using before? |
|
Hi @divyaac And this cluster have lot of pki tenants Mostly all tenants acme shows |
|
Hi @kaleshag would you be able to provide us with a debug package? https://developer.hashicorp.com/vault/tutorials/monitoring/troubleshooting-vault#vault-debug-tool |
|
@divyaac |
|
@divyaac do we have any update on this request? We are blocked with many clusters because of this issue. |
|
Also wanted to know, even though we have not enabled ACME, why we are seeing ACME engine validations in debug log @divyaac |
|
Hi @kaleshag thank you for sending the debug package over! Let me do some investigation and get back to you. |
As of 1.14.0, ACME is supported on PKI engines. As a result, ACME paths are added to the PKI backend during startup. We are still investigating the high memory usage. |
|
Hi @kaleshag, the issue might be because it is adding all the new ACME paths in the current version for all of the pki mounts. The high memory usage is during the regexes for the new paths, while adding them. Is the cluster unusable? |
|
@divyaac Yes the vault pods are getting restarted with OOM Killed. Is there a way we can remove those acme paths validations because we are not using any domain certificates here. So we can avoid unnecessary load on the cluster. So we want to know how to disable that option to avoid this memory spike on our clusters. |
|
In my old version of vault (1.12.9), I don't see any value for acme But where as in my new version of vault(1.15.6), I can see the acme configuration, but they are in disable state. Is it they are causing the issues? If yes, how can we remove them in our vault version. @divyaac |
|
@divyaac Any update on this issue? |
|
@kaleshag We cannot conditionally add paths at this time, so ACME paths are registered even if not enabled. |
|
@divyaac we already did that and pods are coming up with increased limits. But the utilization is not going down. If you look at my previously uploaded graphs to this case explains you the scenario, where it keeps almost similar utilization like startup or increase it little bit. That's where we have the concern and opened this issue for your support. Is it expected or can we do anything to fix this memory utilization after startup also? |
|
@divyaac
|
|
@1337Seeker |
|
Hi Team |
We have upgraded vault to 1.15.6 and suddenly on high load cluster we are seeing huge spike of Memory utilization, but no difference on low usage clusters.
Same cluster had Memory utilization around 500Mi to 1000Mi, but after upgrade we are seeing huge spike like 2120Mi. We want to know why this version having this bug?
Regards
Kalesha Gagguturu
The text was updated successfully, but these errors were encountered: