If you followed one of the
comprehensive install guides or you
performed a custom installation and included the monitoring.yaml
file in your
installation, all of the observability features are already installed and you
can skip down to the
Create Elasticsearch Indices section.
If you have not yet installed any observability plugins, continue to the next sections to do so now.
-
Run the following command to install Prometheus and Grafana:
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.3.0/monitoring-metrics-prometheus.yaml
-
Ensure that the
grafana-*
,kibana-logging-*
,kube-state-metrics-*
,node-exporter-*
andprometheus-system-*
pods all report aRunning
status:kubectl get pods --namespace knative-monitoring --watch
For example:
NAME READY STATUS RESTARTS AGE grafana-798cf569ff-v4q74 1/1 Running 0 2d kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d kube-state-metrics-75bd4f5b8b-8t2h2 4/4 Running 0 2d node-exporter-cr6bh 2/2 Running 0 2d node-exporter-mf6k7 2/2 Running 0 2d node-exporter-rhzr7 2/2 Running 0 2d prometheus-system-0 1/1 Running 0 2d prometheus-system-1 1/1 Running 0 2d
Tip: Hit CTRL+C to exit watch mode.
Accessing Metrics for more information about metrics in Knative.
Knative offers three different setups for collecting logs. Choose one to install:
-
Run the following command to install an ELK stack:
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.3.0/monitoring-logs-elasticsearch.yaml
-
Ensure that the
elasticsearch-logging-*
,fluentd-ds-*
, andkibana-logging-*
pods all report aRunning
status:kubectl get pods --namespace knative-monitoring --watch
For example:
NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 2d elasticsearch-logging-1 1/1 Running 0 2d fluentd-ds-5kc85 1/1 Running 0 2d fluentd-ds-vhrcq 1/1 Running 0 2d fluentd-ds-xghk9 1/1 Running 0 2d kibana-logging-7d474fbb45-6qb8x 1/1 Running 0 2d
Tip: Hit CTRL+C to exit watch mode.
-
Verify that each of your nodes have the
beta.kubernetes.io/fluentd-ds-ready=true
label:kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
-
If you receive the
No Resources Found
response:-
Run the following command to ensure that the Fluentd DaemonSet runs on all your nodes:
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true"
-
Run the following command to ensure that the
fluentd-ds
daemonset is ready on at least one node:kubectl get daemonset fluentd-ds --namespace knative-monitoring --watch
Tip: Hit CTRL+C to exit watch mode.
-
-
When the installation is complete and all the resources are running, you can continue to the next section and begin creating your Elasticsearch indices.
To visualize logs with Kibana, you need to set which Elasticsearch indices to explore.
-
To open the Kibana UI (the visualization tool for Elasticsearch), you must start a local proxy by running the following command:
kubectl proxy
This command starts a local proxy of Kibana on port 8001. For security reasons, the Kibana UI is exposed only within the cluster.
-
Navigate to the Kibana UI. It might take a couple of minutes for the proxy to work.
-
Within the "Configure an index pattern" page, enter
logstash-*
toIndex pattern
and select@timestamp
fromTime Filter field name
and click onCreate
button.
See Accessing Logs for more information about logs in Knative.
To configure and setup monitoring:
-
Clone the Knative Serving repository:
git clone https://github.com/knative/serving knative-serving cd knative-serving git checkout v0.3.0
-
Choose a container image that meets the Fluentd image requirements. For example, you can use a public image. Or you can create a custom one and upload the image to a container registry which your cluster has read access to.
You must configure and build your own Fluentd image if either of the following are true:
- Your Knative Serving component is not hosted on a Google Cloud Platform (GCP) based cluster.
- You want to send logs to another GCP project.
-
Follow the instructions in "Setting up a logging plugin" to configure the stackdriver components settings.
-
Install Knative Stackdriver components by running the following command from the root directory of knative/serving repository:
kubectl apply --recursive --filename config/monitoring/100-namespace.yaml \ --filename third_party/config/monitoring/logging/stackdriver
-
Ensure that the
fluentd-ds-*
pods all report aRunning
status:kubectl get pods --namespace knative-monitoring --watch
For example:
NAME READY STATUS RESTARTS AGE fluentd-ds-5kc85 1/1 Running 0 2d fluentd-ds-vhrcq 1/1 Running 0 2d fluentd-ds-xghk9 1/1 Running 0 2d
Tip: Hit CTRL+C to exit watch mode.
-
Verify that each of your nodes have the
beta.kubernetes.io/fluentd-ds-ready=true
label:kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
-
If you receive the
No Resources Found
response:-
Run the following command to ensure that the Fluentd DaemonSet runs on all your nodes:
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true"
-
Run the following command to ensure that the
fluentd-ds
daemonset is ready on at least one node:kubectl get daemonset fluentd-ds --namespace knative-monitoring
-
See Accessing Logs for more information about logs in Knative.
-
If Elasticsearch is not installed or if you don't want to persist end to end traces, run:
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.3.0/monitoring-tracing-zipkin-in-mem.yaml
-
If Elasticsearch is installed and you want to persist end to end traces, first run:
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.3.0/monitoring-tracing-zipkin.yaml
Next, create an Elasticsearch index for end to end traces:
- Open Kibana UI as described in Create Elasticsearch Indices section.
- Select
Create Index Pattern
button on top left of the page. Enterzipkin*
toIndex pattern
and selecttimestamp_millis
fromTime Filter field name
and click onCreate
button.
Visit Accessing Traces for more information on end to end traces.
- Learn more about accessing logs, metrics, and traces:
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License.