[Bug] Operator shouldn't need to modify Service Accounts, and/or shouldn't need to add a label to them

[cmi4]

Describe the bug
In some contexts, for security reasons Grafana Operator shouldn't have access to modify ServiceAccount resources, and thus there should be an option for it to skip reconciliation of ServiceAccount resources.

This manifests itself starting with #1661 which adds a label to every resource, including ServiceAccount. As a stopgap, it would be fine to be able to avoid adding that label.

2024-12-03T07:00:10Z	INFO	GrafanaReconciler	running stage	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana","namespace":"grafana"}, "namespace": "grafana", "name": "grafana", "reconcileID": "96ddf732-a581-40d3-849b-739cf6270357", "stage": "service account"}
2024-12-03T07:00:10Z	ERROR	GrafanaReconciler	reconciler error in stage	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana","namespace":"grafana"}, "namespace": "grafana", "name": "grafana", "reconcileID": "96ddf732-a581-40d3-849b-739cf6270357", "stage": "service account", "error": "serviceaccounts \"grafana-sa\" is forbidden: User \"system:serviceaccount:grafana:grafana-operator-controller-manager\" cannot update resource \"serviceaccounts\" in API group \"\" in the namespace \"grafana\""}
github.com/grafana/grafana-operator/v5/controllers.(*GrafanaReconciler).Reconcile
2024-12-03T07:00:10Z	INFO	GrafanaReconciler	stage in progress	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana","namespace":"grafana"}, "namespace": "grafana", "name": "grafana", "reconcileID": "96ddf732-a581-40d3-849b-739cf6270357", "stage": "service account"}

Version
v5.15.1, but starts in v5.13.0

To Reproduce
Steps to reproduce the behavior:

Set up Grafana Operator without API access to serviceaccounts. Run Grafana Operator and observe problem.

Expected behavior
It should be possible to either exclude ServiceAccount resources from reconciliation, or elect not to add the CommonLabels to the service accounts (or to any resources)

Suspect component/Location where the bug might be occurring
GrafanaReconciler, service account stage

[theSuess]

Thanks for reporting this issue. I'll provide some more context here to illustrate why labeling the service account is necessary.

In #1622 we analyzed memory usage of the operator and realized that we excessively cache resources which leads to OOM errors in some situations.
To address this, the best way is by selectively caching resources based on labels. This is why #1661 adds logic to add a new label to all resources we manage.

Ideally, we want to avoid adding special cases for ignoring some resources, but I see where your desire to limit the permissions come from. Can you try adding the "app.kubernetes.io/managed-by": "grafana-operator", label to the service account manually and see if the error still shows up?

Note that this will only be relevant for existing resources. Newly created service accounts shouldn't need any updates as they'll have the label present at creation.

[github-actions]

This issue hasn't been updated for a while, marking as stale, please respond within the next 7 days to remove this label