[EEM] Add built in definitions for hosts and containers #193157

miltonhultgren · 2024-09-17T12:29:56Z

Summary

This PR adds built in definitions for hosts and containers based on ECS data.

How to test

Check out this branch of Kibana
Start ES and Kibana, verify that start up works and that the two definitions are installed (by calling GET /internal/entities/definition or checking that the transforms are installed).
Ingest some data for each definition to work with, verify that you get data in entities-host-* and entities-container-* and that the data matches the expected shape (metadata and metrics[1])

[1]

obltmachine · 2024-09-17T12:30:17Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

miltonhultgren · 2024-09-17T12:31:26Z

@roshan-elastic @smith @cauemarcondes Here is the PR to add the two new definitions.

One question I have though, do you still want to include the metrics in version 1.0.0 if you don't plan to use them yet or should I drop them and you can add them later when you are ready to consume that data?

roshan-elastic · 2024-09-17T12:33:51Z

Thanks @miltonhultgren

One question I have though, do you still want to include the metrics in version 1.0.0 if you don't plan to use them yet or should I drop them and you can add them later when you are ready to consume that data?

@smith - I don't mind them being in the indices as long as it doesn't add a burden to our team that we don't want? We won't be showing them in Inventory but I'm happy for them to be there (we get feedback from users if they click through to Discover).

WDYT?

miltonhultgren · 2024-09-17T12:36:18Z

Do consider that adding metrics does increase the compute and storage cost of the definition (which long term I imagine we'll need to justify, of course, but perhaps it is wise to defer it until later).
Is it likely that users will see it in Discover and use it for some problem solution?

roshan-elastic · 2024-09-17T12:46:01Z

Do consider that adding metrics does increase the compute and storage cost of the definition (which long term I imagine we'll need to justify, of course, but perhaps it is wise to defer it until later).

I'll defer to @chrisdistasio on this aspect. If he's OK with it, I am (from a PM perspective).

Is it likely that users will see it in Discover and use it for some problem solution?

We don't know really. Sometimes putting something in the hands of users and seeing what they do is the best way to find out! It's basically cheap discovery.

smith

I have some suggestions here but in general this looks ok to merge.

As long as we're assuming Elastic-specific ECS data these match the data models of existing Elastic apps.

Are we planning on having additional entity definitions later on for otel-native schemas of these entity types (Host for example?)

smith · 2024-09-17T22:43:59Z

x-pack/plugins/entity_manager/server/lib/entities/built_in/containers.ts

+    type: 'container',
+    indexPatterns: ['filebeat-*', 'logs-*', 'metrics-*', 'metricbeat-*'],
+    identityFields: ['container.id'],
+    displayNameTemplate: '{{container.id}}',


This should be container.name.

Please see https://github.com/elastic/elastic-entity-model/issues/168

x-pack/plugins/entity_manager/server/lib/entities/built_in/containers.ts

miltonhultgren · 2024-09-18T09:43:19Z

Are we planning on having additional entity definitions later on for otel-native schemas of these entity types (Host for example?)

I don't know, I would defer to @chrisdistasio to speak to the plans for the EEM team.

To share my 2 cents though:
If the ECO team wants to extract host entities from SemConv data then you should feel free to submit a PR that adds such a built in (as done in this simple PR) or find a way for the OTEL agent to install such a definition (similar to what we'll do for Kubernetes via the Integrations).

I don't believe it is productive for the EEM team to create and maintain specific definitions.
I believe we should focus our time on making sure your team has the tools to extract the data in the way you need to.
Similar to the split between Beats core and Beats modules.

…ner-builtin-definition

miltonhultgren · 2024-09-18T10:38:58Z

@roshan-elastic @chrisdistasio FYI I think it's okay to leave them in for now (similar for the service built in definition), we need to decide on if we will descope the current metrics features and will treat that as a follow up once we decide on where we want to go.

kibana-ci · 2024-09-18T11:12:37Z

💚 Build Succeeded

Buildkite Build
Commit: fa5115a

Metrics [docs]

✅ unchanged

History

💔 Build #234814 failed 16108c4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

klacabane · 2024-09-18T13:40:44Z

x-pack/plugins/entity_manager/server/lib/entities/built_in/containers_from_ecs_data.ts

+          {
+            name: 'A',
+            aggregation: 'doc_count',
+            filter: 'log.level: * OR error.log.level: *',


should we align with the services filter ? https://github.com/elastic/kibana/blob/main/x-pack/plugins/entity_manager/server/lib/entities/built_in/services.ts#L124

klacabane · 2024-09-18T13:42:19Z

x-pack/plugins/entity_manager/server/lib/entities/built_in/hosts_from_ecs_data.ts

+  indexPatterns: ['filebeat-*', 'logs-*', 'metrics-*', 'metricbeat-*'],
+  identityFields: ['host.name'],
+  displayNameTemplate: '{{host.name}}',
+  history: {


should we also align these settings with the services builtin https://github.com/elastic/kibana/blob/main/x-pack/plugins/entity_manager/server/lib/entities/built_in/services.ts#L36-L44

kibanamachine · 2024-09-18T14:08:07Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [AI Assistant] Add scopes to AI assistant (#192611) - Adds @mswjs/http-middleware to renovate.json (#193257) - [EDR Workflows][Osquery] Adapt to policy_ids array instead of policy_id (#193150) - [Fleet] use @kbn/config-schema in Fleet API (Part 1) (#192883) - [Fleet] Handle missing permissions when creating standalone agent API keys (#193218) - [Automatic import] refactor merge (#193270) - [Automatic Import] Modify codegen to autogenerate openAPI spec code (#193243) - [Security Solution][Detection Engine] Reduce CPU usage when creating alerts from source documents (#192287) - Update dependency tough-cookie to v5 (main) (#193107) - Updating Devcontainer Dockerfile so the vscode user can run bootstrap (#193224)

Manual backport

To create the backport manually run:

node scripts/backport --pr 193157

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2024-09-30T18:20:55Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 193157 locally

kibanamachine · 2024-10-01T18:49:20Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 193157 locally

kibanamachine · 2024-10-02T18:49:26Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 193157 locally

kibanamachine · 2024-10-03T19:48:41Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 193157 locally

kibanamachine · 2024-10-04T09:45:28Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11177739213

kibanamachine · 2024-10-04T09:50:20Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…c#193157) This PR adds built in definitions for `hosts` and `containers` based on ECS data. 1. Check out this branch of Kibana 2. Start ES and Kibana, verify that start up works and that the two definitions are installed (by calling `GET /internal/entities/definition` or checking that the transforms are installed). 3. Ingest some data for each definition to work with, verify that you get data in `entities-host-*` and `entities-container-*` and that the data matches the expected shape (metadata and metrics[1]) [[1]](elastic#193157 (comment))

… (#194921) # Backport This will backport the following commits from `main` to `8.x`: - [[EEM] Add built in definitions for hosts and containers (#193157)](#193157)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Milton Hultgren <[email protected]>

[EEM] Add built in definitions for hosts and containers

16108c4

miltonhultgren added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Feature:EEM Elastic Entity Model labels Sep 17, 2024

miltonhultgren requested a review from a team as a code owner September 17, 2024 12:29

smith approved these changes Sep 17, 2024

View reviewed changes

miltonhultgren added 2 commits September 18, 2024 11:48

Rename files, add one more metadata field

51dcfd2

Merge branch 'main' of github.com:elastic/kibana into eem-host-contai…

fa5115a

…ner-builtin-definition

klacabane reviewed Sep 18, 2024

View reviewed changes

klacabane approved these changes Sep 18, 2024

View reviewed changes

miltonhultgren merged commit 26a50f7 into elastic:main Sep 18, 2024
19 checks passed

kibanamachine added the v9.0.0 label Sep 18, 2024

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 30, 2024

klacabane removed the backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) label Oct 4, 2024

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 4, 2024

kibanamachine added the backport:skip This commit does not require backporting label Oct 4, 2024

klacabane added backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) and removed backport:skip This commit does not require backporting labels Oct 4, 2024

kibanamachine mentioned this pull request Oct 4, 2024

[8.x] [EEM] Add built in definitions for hosts and containers (#193157) #194921

Merged

kibanamachine added the v8.16.0 label Oct 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EEM] Add built in definitions for hosts and containers #193157

[EEM] Add built in definitions for hosts and containers #193157

miltonhultgren commented Sep 17, 2024 •

edited by kibanamachine

Loading

obltmachine commented Sep 17, 2024

miltonhultgren commented Sep 17, 2024

roshan-elastic commented Sep 17, 2024

miltonhultgren commented Sep 17, 2024

roshan-elastic commented Sep 17, 2024

smith left a comment

smith Sep 17, 2024

miltonhultgren Sep 18, 2024

miltonhultgren commented Sep 18, 2024 •

edited

Loading

miltonhultgren commented Sep 18, 2024

kibana-ci commented Sep 18, 2024

klacabane Sep 18, 2024

klacabane Sep 18, 2024

kibanamachine commented Sep 18, 2024

kibanamachine commented Sep 30, 2024

kibanamachine commented Oct 1, 2024

kibanamachine commented Oct 2, 2024

kibanamachine commented Oct 3, 2024

kibanamachine commented Oct 4, 2024

kibanamachine commented Oct 4, 2024

[EEM] Add built in definitions for hosts and containers #193157

[EEM] Add built in definitions for hosts and containers #193157

Conversation

miltonhultgren commented Sep 17, 2024 • edited by kibanamachine Loading

Summary

How to test

obltmachine commented Sep 17, 2024

🤖 GitHub comments

miltonhultgren commented Sep 17, 2024

roshan-elastic commented Sep 17, 2024

miltonhultgren commented Sep 17, 2024

roshan-elastic commented Sep 17, 2024

smith left a comment

Choose a reason for hiding this comment

smith Sep 17, 2024

Choose a reason for hiding this comment

miltonhultgren Sep 18, 2024

Choose a reason for hiding this comment

miltonhultgren commented Sep 18, 2024 • edited Loading

miltonhultgren commented Sep 18, 2024

kibana-ci commented Sep 18, 2024

💚 Build Succeeded

Metrics [docs]

History

klacabane Sep 18, 2024

Choose a reason for hiding this comment

klacabane Sep 18, 2024

Choose a reason for hiding this comment

kibanamachine commented Sep 18, 2024

💔 All backports failed

Manual backport

Questions ?

kibanamachine commented Sep 30, 2024

kibanamachine commented Oct 1, 2024

kibanamachine commented Oct 2, 2024

kibanamachine commented Oct 3, 2024

kibanamachine commented Oct 4, 2024

kibanamachine commented Oct 4, 2024

💚 All backports created successfully

Questions ?

miltonhultgren commented Sep 17, 2024 •

edited by kibanamachine

Loading

miltonhultgren commented Sep 18, 2024 •

edited

Loading