[Kubernetes provider]Update kubernetes provider README #5990
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,20 +1,32 @@ | ||||||
| ## Autodiscover provider in Elastic Agent | ||||||
|
|
||||||
|
|
||||||
| https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| Kubernetes provider can be configured both in **standalone** and **fleet managed** elastic agent. | ||||||
|
|
||||||
| a. In **standalone** elastic agent, user needs to edit the [elastic-agent-standalone-kubernetes.yaml](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L28) | ||||||
| b. In **fleet managed** elastic agent, kubernetes provider is enabled by default with default values. User needs to follow the steps described in the [documentation](https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html) to configure it. | ||||||
|
|
||||||
|
There was a problem hiding this comment. I feel like a blank line is missing from here (between a. and b.) There was a problem hiding this comment. Does that cause it to render different? I don't think so. There was a problem hiding this comment. it actually does, now a. and b. are in the same paragraph which makes it kind hard to distinguish the two. However if you add an empty line between a. and b. they appear properly separated 🙂 There was a problem hiding this comment. Then absolutely fix before merge. |
||||||
| Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). | ||||||
|
|
||||||
| Hints based autodiscover is only supported in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html)). | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ### Conditions based autodiscover | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| Example: | ||||||
|
There was a problem hiding this comment. with the new write up below Example: feels redundant |
||||||
|
|
||||||
| As an example we will use the redis module. | ||||||
|
|
||||||
| To automatically identify a Redis Pod and monitor it with the Redis integration, add the following input configuration inside the [Elastic Agent Standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml): | ||||||
|
|
||||||
|
There was a problem hiding this comment. qq: for the Conditions-based autodiscover to work correctly is any kubernetes provider required to be enabled? If yes please provide the necessary settings of the provider in this example There was a problem hiding this comment. The default settings of the kubernetes provider should be enough for conditions to work. |
||||||
| ``` | ||||||
| # Add extra input blocks here, based on conditions | ||||||
| # so as to automatically identify targeted Pods and start monitoring them | ||||||
|
|
@@ -43,28 +55,50 @@ In agent.yml(configmap) an extra input block needs to be added. | |||||
| condition: ${kubernetes.pod.labels.app} == 'redis' | ||||||
| ``` | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| What makes this input block dynamic are the variables under hosts block and the condition. | ||||||
|
|
||||||
| `${kubernetes.pod.ip}` and `${kubernetes.pod.labels.app} == 'redis'` | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| #### High level description | ||||||
|
|
||||||
| The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables([list of variables](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html#_provider_for_pod_resources)) for each k8s resource with unique value for each one of them. | ||||||
|
|
||||||
| Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blogs of the configurations. | ||||||
|
|
||||||
|
There was a problem hiding this comment.
Suggested change
Also, maybe add a link to the controller module here? |
||||||
| This means that for every mapping that the condition matches (`kubernetes.pod.labels.app == redis`), a new input will be created in which the condition will be removed(not needed anymore) and the `kubernetes.pod.ip` variable will be substituted from the value in the same mapping. | ||||||
|
|
||||||
| The updated complete inputs block will be then forwarded to agent to spawn/update metricbeat and filebeat instances. | ||||||
|
|
||||||
|
|
||||||
| ### Hints based autodiscover | ||||||
|
|
||||||
| Standalone elastic agent supports autodiscover based on hints collected from the [Kubernetes Provider](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html). The hints mechanism looks for hints in kubernetes pod annotations that have the prefix `co.elastic.hints`. As soon as the container starts, elastic agent checks it for hints and launches the proper configuration for the container. Hints tell elastic agent how to monitor the container by using the proper integration. | ||||||
| The full list of supported hints can be found [here](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_required_hints). | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| Example: | ||||||
|
|
||||||
| As an example we will use again redis module. | ||||||
| Add the following annotations to a redis pod. Elastic agent will then initiate a new input with redis module to properly monitor the redis pod. | ||||||
| ``` | ||||||
| apiVersion: v1 | ||||||
| kind: Pod | ||||||
| metadata: | ||||||
| name: redis | ||||||
| annotations: | ||||||
| co.elastic.hints/package: redis | ||||||
| co.elastic.hints/data_streams: info | ||||||
| co.elastic.hints/info.period: 5m | ||||||
| ``` | ||||||
|
|
||||||
| In order to enable hints based autodiscover, user needs to uncomment the following lines in the [Elastic Agent Standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) | ||||||
| 1. Add the [init container]((https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L697-L709)) that downloads the templates of various packages. | ||||||
|
There was a problem hiding this comment. point to the lines in the k8s yaml (https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L31) or even better, if you can, capture which agent providers configuration is needed and then point to the lines |
||||||
| 2. Add the [volumeMount](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L783-L785) and [volume](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L824-L826). | ||||||
|
|
||||||
| The init container will start before the elastic agent pod and will donwload all the templates of packages [supported](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). | ||||||
| Elastic agent will then collect from the pods running(through the watchers mechanism) all the hints annotations and will try to match them with the correct package. | ||||||
| In the redis example, it will use the [redis template](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/redis.yml) and substitute the template variables with the values specified in the annotations. Default values will be used for variables not provided. | ||||||
| A new input will be then created for redis and reds data will start being collected. | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't this provider enabled by default in general? I believe this is the case for all providers.