Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Document Kubernetes standard labels cardinality #10749

Open
sircthulhu opened this issue Sep 29, 2023 · 3 comments
Open

[docs] Document Kubernetes standard labels cardinality #10749

sircthulhu opened this issue Sep 29, 2023 · 3 comments
Assignees
Labels
help wanted We would love help on these issues. Please come help us! type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories

Comments

@sircthulhu
Copy link

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.
I'm researching services to handle logs from my kubernetes pods. Loki documentation warns about high cardinality performance issues. I could not find any information if kubernetes standard pod labels (such as pod_name, image and so on) are not too cardinal for storing in Loki index.
Anything I could found is answer by Loki team on Grafana forum: Is labels like instance or k8s pod_name have too high cardinality for Loki

Describe the solution you'd like
I believe there should be a chapter about kubernetes labels in Loki documentation so that newcomers won't need to google this cruical information.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Could there be another solutions? :)

Additional context
Add any other context or screenshots about the feature request here.

@oleksii-boiko-ua
Copy link
Contributor

oleksii-boiko-ua commented Sep 29, 2023

pod name label would definitely introduce cardinality(depends on number of them of course and how often you deploy things) i would suggest keep unique labels to minimum, we ended up having only namespace, job name, container name as a labels, rest you can now move to unindexed labels(structured metadata)

@JStickler JStickler changed the title Kubernetes standard labels cardinality [docs] Document Kubernetes standard labels cardinality Oct 2, 2023
@JStickler JStickler added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Oct 2, 2023
@JStickler JStickler self-assigned this Oct 2, 2023
@JStickler JStickler added the help wanted We would love help on these issues. Please come help us! label Oct 2, 2023
@Angelin01
Copy link

This comment will be slightly off topic, but I'd like to argue that this is not only an issue not only with documentation, but also with the default Helm Chart values from both Loki and Promtail (which is the default go-to collector for Loki).

I ran a quick analysis against our "default setup" helm install Loki and Promtail, this is the result:

/ # logcli series --org-id 1 --analyze-labels '{}'
Total Streams:  387
Unique Labels:  10

Label Name  Unique Values  Found In Streams
filename    365            387
pod         216            387
container   105            387
job         87             387
app         67             387
namespace   31             387
instance    22             162
component   19             93
node_name   7              387
stream      2              387

There's clearly a lot of high cardinality here. filename, pod, etc.

This, coupled with the rather slow default parameters for Loki, hence this long standing issue, makes for quite a bad experience "out of the box". Our S3 bucket contains chunks that are mere kBs in size:
image

I know at lot of what I am posting is related to Promtail, but at this point I feel like both are very much a coupled experience, most users will simply install both together.

Getting to the point: yes, the documentation could be improved, but I'd much prefer if "saner defaults" were provided instead. People look to default values as a starting point, it's probably the best type of documentation there is: it doesn't rot, and it helps immediately both those that need to customize and those that need to fine tune.

@JStickler
Copy link
Contributor

@sircthulhu This issue is over a year old, but I recently rewrote the labeling topic and added a topic for cardinality. Let me know if they answer your questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted We would love help on these issues. Please come help us! type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

No branches or pull requests

4 participants