Skip to content

Commit 545d9ed

Browse files
paulfantompaulfantom
committed
*: rework readme
Signed-off-by: Paweł Krupa (paulfantom) <[email protected]>
1 parent 0e3b5e5 commit 545d9ed

File tree

6 files changed

+299
-333
lines changed

6 files changed

+299
-333
lines changed

‎README.md‎

Lines changed: 21 additions & 331 deletions
Large diffs are not rendered by default.

‎RELEASE.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Kubernetes. We need to make sure that the CI on the main branch is testing the
5555
kube-prometheus configuration against both of these versions by updating the [CI
5656
worklow](.github/workflows/ci.yaml) to include the latest kind version and the
5757
2 latest images versions that are attached to the kind release. Once that is
58-
done, the [compatibility matrix](README.md#kubernetes-compatibility-matrix) in
58+
done, the [compatibility matrix](README.md#compatibility) in
5959
the README should also be updated to reflect the CI changes.
6060

6161
## Create pull request to cut the release
@@ -87,7 +87,7 @@ make generate
8787

8888
### Update the compatibility matrix
8989

90-
Update the [compatibility matrix](README.md#kubernetes-compatibility-matrix) in
90+
Update the [compatibility matrix](README.md#compatibility) in
9191
the README, by adding the new release based on the `main` branch compatibility
9292
and removing the oldest release branch to only keep the latest 5 branches in the
9393
matrix.

‎docs/access-ui.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Access UIs
2+
3+
Prometheus, Grafana, and Alertmanager dashboards can be accessed quickly using `kubectl port-forward` after running the quickstart via the commands below. Kubernetes 1.10 or later is required.
4+
5+
> Note: There are instructions on how to route to these pods behind an ingress controller in the [Exposing Prometheus/Alermanager/Grafana via Ingress](customizations/exposing-prometheus-alertmanager-grafana-ingress.md) section.
6+
7+
## Prometheus
8+
9+
```shell
10+
$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
11+
```
12+
13+
Then access via [http://localhost:9090](http://localhost:9090)
14+
15+
## Grafana
16+
17+
```shell
18+
$ kubectl --namespace monitoring port-forward svc/grafana 3000
19+
```
20+
21+
Then access via [http://localhost:3000](http://localhost:3000) and use the default grafana user:password of `admin:admin`.
22+
23+
## Alert Manager
24+
25+
```shell
26+
$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093
27+
```
28+
29+
Then access via [http://localhost:9093](http://localhost:9093)

‎docs/customizing.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Customizing Kube-Prometheus
2+
3+
This section:
4+
* describes how to customize the kube-prometheus library via compiling the kube-prometheus manifests yourself (as an alternative to the [README.md quickstart section](../README.md#quickstart)).
5+
* still doesn't require you to make a copy of this entire repository, but rather only a copy of a few select files.
6+
7+
## Installing
8+
9+
The content of this project consists of a set of [jsonnet](http://jsonnet.org/) files making up a library to be consumed.
10+
11+
Install this library in your own project with [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler#install) (the jsonnet package manager):
12+
13+
```shell
14+
$ mkdir my-kube-prometheus; cd my-kube-prometheus
15+
$ jb init # Creates the initial/empty `jsonnetfile.json`
16+
# Install the kube-prometheus dependency
17+
$ jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json`
18+
19+
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/example.jsonnet -O example.jsonnet
20+
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
21+
```
22+
23+
> `jb` can be installed with `go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest`
24+
25+
> An e.g. of how to install a given version of this library: `jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main`
26+
27+
In order to update the kube-prometheus dependency, simply use the jsonnet-bundler update functionality:
28+
29+
```shell
30+
$ jb update
31+
```
32+
33+
## Generating
34+
35+
e.g. of how to compile the manifests: `./build.sh example.jsonnet`
36+
37+
> before compiling, install `gojsontoyaml` tool with `go install github.com/brancz/gojsontoyaml@latest` and `jsonnet` with `go install github.com/google/go-jsonnet/cmd/jsonnet@latest`
38+
39+
Here's [example.jsonnet](../example.jsonnet):
40+
41+
> Note: some of the following components must be configured beforehand. See [configuration](#configuring) and [customization-examples](customizations).
42+
43+
```jsonnet mdox-exec="cat example.jsonnet"
44+
local kp =
45+
(import 'kube-prometheus/main.libsonnet') +
46+
// Uncomment the following imports to enable its patches
47+
// (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
48+
// (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
49+
// (import 'kube-prometheus/addons/node-ports.libsonnet') +
50+
// (import 'kube-prometheus/addons/static-etcd.libsonnet') +
51+
// (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
52+
// (import 'kube-prometheus/addons/external-metrics.libsonnet') +
53+
// (import 'kube-prometheus/addons/pyrra.libsonnet') +
54+
{
55+
values+:: {
56+
common+: {
57+
namespace: 'monitoring',
58+
},
59+
},
60+
};
61+
62+
{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
63+
{
64+
['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
65+
for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
66+
} +
67+
// { 'setup/pyrra-slo-CustomResourceDefinition': kp.pyrra.crd } +
68+
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
69+
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
70+
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
71+
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
72+
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
73+
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
74+
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
75+
// { ['pyrra-' + name]: kp.pyrra[name] for name in std.objectFields(kp.pyrra) if name != 'crd' } +
76+
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
77+
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
78+
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
79+
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
80+
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }
81+
```
82+
83+
And here's the [build.sh](../build.sh) script (which uses `vendor/` to render all manifests in a json structure of `{filename: manifest-content}`):
84+
85+
```sh mdox-exec="cat ./build.sh"
86+
#!/usr/bin/env bash
87+
88+
# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.
89+
90+
set -e
91+
set -x
92+
# only exit with zero if all commands of the pipeline exit successfully
93+
set -o pipefail
94+
95+
# Make sure to use project tooling
96+
PATH="$(pwd)/tmp/bin:${PATH}"
97+
98+
# Make sure to start with a clean 'manifests' dir
99+
rm -rf manifests
100+
mkdir -p manifests/setup
101+
102+
# Calling gojsontoyaml is optional, but we would like to generate yaml, not json
103+
jsonnet -J vendor -m manifests "${1-example.jsonnet}" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}
104+
105+
# Make sure to remove json files
106+
find manifests -type f ! -name '*.yaml' -delete
107+
rm -f kustomization
108+
109+
```
110+
111+
> Note you need `jsonnet` (`go install github.com/google/go-jsonnet/cmd/jsonnet@latest`) and `gojsontoyaml` (`go install github.com/brancz/gojsontoyaml@latest`) installed to run `build.sh`. If you just want json output, not yaml, then you can skip the pipe and everything afterwards.
112+
113+
This script runs the jsonnet code, then reads each key of the generated json and uses that as the file name, and writes the value of that key to that file, and converts each json manifest to yaml.
114+
115+
## Configuring
116+
117+
Jsonnet has the concept of hidden fields. These are fields, that are not going to be rendered in a result. This is used to configure the kube-prometheus components in jsonnet. In the example jsonnet code of the above [Generating section](#generating), you can see an example of this, where the `namespace` is being configured to be `monitoring`. In order to not override the whole object, use the `+::` construct of jsonnet, to merge objects, this way you can override individual settings, but retain all other settings and defaults.
118+
119+
The available fields and their default values can be seen in [main.libsonnet](../jsonnet/kube-prometheus/main.libsonnet). Note that many of the fields get their default values from variables, and for example the version numbers are imported from [versions.json](../jsonnet/kube-prometheus/versions.json).
120+
121+
Configuration is mainly done in the `values` map. You can see this being used in the `example.jsonnet` to set the namespace to `monitoring`. This is done in the `common` field, which all other components take their default value from. See for example how Alertmanager is configured in `main.libsonnet`:
122+
123+
```
124+
alertmanager: {
125+
name: 'main',
126+
// Use the namespace specified under values.common by default.
127+
namespace: $.values.common.namespace,
128+
version: $.values.common.versions.alertmanager,
129+
image: $.values.common.images.alertmanager,
130+
mixin+: { ruleLabels: $.values.common.ruleLabels },
131+
},
132+
```
133+
134+
The grafana definition is located in a different project (https://github.com/brancz/kubernetes-grafana ), but needed configuration can be customized from the same top level `values` field. For example to allow anonymous access to grafana, add the following `values` section:
135+
136+
```
137+
grafana+:: {
138+
config: { // http://docs.grafana.org/installation/configuration/
139+
sections: {
140+
"auth.anonymous": {enabled: true},
141+
},
142+
},
143+
},
144+
```
145+
146+
## Apply the kube-prometheus stack
147+
148+
The previous generation step has created a bunch of manifest files in the manifest/ folder.
149+
Now simply use `kubectl` to install Prometheus and Grafana as per your configuration:
150+
151+
```shell
152+
# Update the namespace and CRDs, and then wait for them to be available before creating the remaining resources
153+
$ kubectl apply --server-side -f manifests/setup
154+
$ kubectl apply -f manifests/
155+
```
156+
157+
> Note that due to some CRD size we are using kubeclt server-side apply feature which is generally available since
158+
> kubernetes 1.22. If you are using previous kubernetes versions this feature may not be available and you would need to
159+
> use `kubectl create` instead.
160+
161+
Alternatively, the resources in both folders can be applied with a single command
162+
`kubectl apply --server-side -Rf manifests`, but it may be necessary to run the command multiple times for all components to
163+
be created successfully.
164+
165+
Check the monitoring namespace (or the namespace you have specific in `namespace: `) and make sure the pods are running. Prometheus and Grafana should be up and running soon.
166+
167+
## Minikube Example
168+
169+
To use an easy to reproduce example, see [minikube.jsonnet](../examples/minikube.jsonnet), which uses the minikube setup as demonstrated in [Prerequisites](../README.md#prerequisites). Because we would like easy access to our Prometheus, Alertmanager and Grafana UIs, `minikube.jsonnet` exposes the services as NodePort type services.

‎docs/troubleshooting.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Troubleshooting
2+
3+
See the general [guidelines](community-support.md) for getting support from the community.
4+
5+
## Error retrieving kubelet metrics
6+
7+
Should the Prometheus `/targets` page show kubelet targets, but not able to successfully scrape the metrics, then most likely it is a problem with the authentication and authorization setup of the kubelets.
8+
9+
As described in the [README.md Prerequisites](../README.md#prerequisites) section, in order to retrieve metrics from the kubelet token authentication and authorization must be enabled. Some Kubernetes setup tools do not enable this by default.
10+
11+
- If you are using Google's GKE product, see [cAdvisor support](GKE-cadvisor-support.md).
12+
- If you are using AWS EKS, see [AWS EKS CNI support](EKS-cni-support.md).
13+
- If you are using Weave Net, see [Weave Net support](weave-net-support.md).
14+
15+
### Authentication problem
16+
17+
The Prometheus `/targets` page will show the kubelet job with the error `403 Unauthorized`, when token authentication is not enabled. Ensure, that the `--authentication-token-webhook=true` flag is enabled on all kubelet configurations.
18+
19+
### Authorization problem
20+
21+
The Prometheus `/targets` page will show the kubelet job with the error `401 Unauthorized`, when token authorization is not enabled. Ensure that the `--authorization-mode=Webhook` flag is enabled on all kubelet configurations.
22+
23+
## kube-state-metrics resource usage
24+
25+
In some environments, kube-state-metrics may need additional
26+
resources. One driver for more resource needs, is a high number of
27+
namespaces. There may be others.
28+
29+
kube-state-metrics resource allocation is managed by
30+
[addon-resizer](https://github.com/kubernetes/autoscaler/tree/master/addon-resizer/nanny)
31+
You can control it's parameters by setting variables in the
32+
config. They default to:
33+
34+
```jsonnet
35+
kubeStateMetrics+:: {
36+
baseCPU: '100m',
37+
cpuPerNode: '2m',
38+
baseMemory: '150Mi',
39+
memoryPerNode: '30Mi',
40+
}
41+
```
42+
43+
## Error retrieving kube-proxy metrics
44+
45+
By default, kubeadm will configure kube-proxy to listen on 127.0.0.1 for metrics. Because of this prometheus would not be able to scrape these metrics. This would have to be changed to 0.0.0.0 in one of the following two places:
46+
47+
1. Before cluster initialization, the config file passed to kubeadm init should have KubeProxyConfiguration manifest with the field metricsBindAddress set to 0.0.0.0:10249
48+
2. If the k8s cluster is already up and running, we'll have to modify the configmap kube-proxy in the namespace kube-system and set the metricsBindAddress field. After this kube-proxy daemonset would have to be restarted with
49+
`kubectl -n kube-system rollout restart daemonset kube-proxy`

‎docs/update.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Update kube-prometheus
2+
3+
You may wish to fetch changes made on this project so they are available to you.
4+
5+
## Update jb
6+
7+
`jb` may have been updated so it's a good idea to get the latest version of this binary:
8+
9+
```shell
10+
$ go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
11+
```
12+
13+
## Update kube-prometheus
14+
15+
The command below will sync with upstream project:
16+
17+
```shell
18+
$ jb update
19+
```
20+
21+
## Compile the manifests and apply
22+
23+
Once updated, just follow the instructions under [Generating](customizing.md#generating) and [Apply the kube-prometheus stack](customizing.md#apply-the-kube-prometheus-stack) from [customizing.md doc](customizing.md) to apply the changes to your cluster.
24+
25+
## Migration from previous versions
26+
27+
If you are migrating from `release-0.7` branch or earlier please read [what changed and how to migrate in our guide](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-guide.md).
28+
29+
Refer to [migration document](migration-example) for more information about migration from 0.3 and 0.8 versions of kube-prometheus.

0 commit comments

Comments
 (0)