Export logs to grafana #42
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Exporting logs to grafana is not trivial. In order to do that we need a datasource that exports logs. Grafana relies on Grafana Loki [1] to do that. Loki is a system that aggregates, index, stores and serves logs. However, to retrieve the logs, Loki relies on Grafana Promtail [2] that is an agent that runs on each node. This PR: * created a Promtail daemonset that scrapes the pod logs. In a future PR, we might want to scrape other system info, container logs, etc. * created a Loki service that aggregates and stores the logs from the Promtail daemonsets. However, the Loki service runs in a single pod, and stores the logs in the local filesystem. In a future PR we might need to tune the storage so we can store more logs, and/or scale up the Loki service. * added a Loki datasource to grafana After this PR, Grafana should be the ultimate place to inspect logs, metrics, and traces.
In a future PR, we should provide the namespace as a config param, so users can run their app with kube in any namespace.
spetrovic77
approved these changes
Aug 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
grafana_logs_new.webm
Exporting logs to grafana is not trivial. In order to do that we need a datasource that exports logs. Grafana relies on Grafana Loki [1] to do that. Loki is a system that aggregates, index, stores and serves logs.
However, to retrieve the logs, Loki relies on Grafana Promtail [2] that is an agent that runs on each node.
This PR:
After this PR, Grafana should be the ultimate place to inspect logs, metrics, and traces.