Throttling warning causes incorrect behavior

[vbasem]

I was wondering why my wait for was never finishing and while debugging in the container I saw that my API server was returning a warning message on top of the POD infos:

I0624 10:31:01.428761     375 request.go:668] Waited for 1.181153768s due to client-side throttling, not priority and fairness, request: GET:https://<server>/apis/rbac.authorization.k8s.io/v1?timeout=32s
NAME                            READY   STATUS    RESTARTS   AGE
<pod>   1/1     Running   0          173m
<pod>   1/1     Running   0          173m
<pod>   1/1     Running   0          173m

the script seems to be parsing the first line and ends up not detecting the pod status.

I might have some time to prepare a quick fix but for now I will leave an issue report here.

_{Basem Vaseghi [email protected], Daimler TSS GmbH, legal info/Impressum}

[groundnuty]

I guess it's thanks to all those stderr to stdout redirections ;-(
Removing then should make errors visible container logs and also fix the problem.

If you happen to have a spare moment, you're welcome to check if that really solves the problem and I will include that fix in the next release.

[slinkel-vkb]

We were able to solve this issue simply by mounting an empty dir at /.kube

In our case, the wait-for container was not running as root, so kubectl could not create a cache dir in its $HOME (i.e. /)

a detailed explanation can be found at: https://jonnylangefeld.com/blog/the-kubernetes-discovery-cache-blessing-and-curse

tl;dr: kubectl cannot cache global resource definitions if ~/.kube is not present/writable, so it ends up requesting it all the time, so it will get throttled