Kubernetes Node Sizing and Request Limits Usage Best Practices

General recommendations to choose the right size of the nodes and usage of containers' request and limits

• Any single container is part of a solution that can be part of a scope of solutions. It is not possible to estimate the optimization task from a global perspective.
• Requests and limits should be chosen according to the profile of the workload, deeply understanding the internal machinery of the particular service that is running inside the container.
• Pay attention to the resources that the container consumes during the startup and regular consumption.
• Monitor the CPU consumption and service response time to adjust CPU requests and limits during the service lifecycle.
• The right node size is not just the sum of containers' CPU/Memory consumption. The node needs reservation for operating system processes and Kubernetes management processes. Moreover, the node size should include some amount of resources to consume before the auto scaling system runs the new nodes in case of triggering.
• The strict container allocation to the node according to the labels as well as random container allocation are two opposites that work for very limited cases. The first one leads to insufficient loading of nodes' resources, while the second one leads to the eviction of some containers or pending processes.
• It is possible to not use CPU limits with appropriate CPU requests for all containers on the particular nodes. However, this might contradict to Limit Ranges that are applied on the cluster.
• CPU usage metrics are stored in the /sys/fs/cgroup/ folder. Each container has its own subfolder (the container runtime can create additional subfolders). The cpu.stat file has the following metrics: nr_periods, nr_throttled, and throttled_time. These metrics can be used in the CPU request calculation.