ETCD pod OOMKilled

[vikasbudhwat]

We have etcd cluster deployed with resource conf
cpu: 200m memory: 100Mi

Our etcd is used as backend for HA tracker(of premetheus-agent), which means it would just keep key: values for active prometheus agent and its replica.

we have just 4 prometheus agent, each agent has 2 replica. and etcd will just track which replica of each prometheus-agent is active one.

etcd cluster was serving my use case and setup was deployed over 2 weeks back, now suddenly its started crashing with OOMKilled(event force killing pod same issue appeared).

then I increased memory to memory: 150Mi and pods came up .

I am wondering how just to track 4 key values with 100Mi I am getting OOMKilled?

is there anything else etcd does behind the scene which might have taken this space if so, how much memory I should allocate? any guideline around would be helpfull.

[SimFG]

The benchmark doc maybe can help you, detail: https://etcd.io/docs/v3.5/benchmarks/etcd-storage-memory-benchmark/.
The etcd server will keep a lot of history data for low latency. You can try to configure the low storage size limit, see: https://etcd.io/docs/v3.5/dev-guide/limit/.