Change ElasticsearchWriter to write perfdata as an array of objects

[martialblog]

Hi,

Let me preface this PR with: I'm not a C++ developer. So please excuse my C++, tried my best, feedback is welcome.

Prior to this change the ElasticsearchWriter created a new field for every metric in check command. This leads to a vast number of fields in the indices, aka a field mapping explosion.

Now, it uses an array of objects. This avoids the explosion, Makes the index fields more predictable when searching and Makes the perfdata independently searchable.

How the new output looks like:

          "check_result.latency": 0.0007982254028320313,
          "check_result.output": "LOAD OK - total load average: 1.84, 1.77, 2.27",
          "check_result.perfdata": [
            {
              "crit": 10,
              "label": "load1",
              "min": 0,
              "value": 1.84,
              "warn": 5
            },
            {
              "crit": 6,
              "label": "load5",
              "min": 0,
              "value": 1.77,
              "warn": 4
            },
            {
              "crit": 4,
              "label": "load15",
              "min": 0,
              "value": 2.27,
              "warn": 3
            }
          ],
          "check_result.schedule_end": "2025-07-24T10:05:52.877+0000",
          "check_result.schedule_start": "2025-07-24T10:05:52.870+0000",

The resulting mapping in the index:

            "perfdata": {
              "properties": {
                "crit": {
                  "type": "long"
                },
                "label": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "max": {
                  "type": "long"
                },
                "min": {
                  "type": "long"
                },
                "unit": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "value": {
                  "type": "long"
                },
                "warn": {
                  "type": "long"
                }
              }
            },

Users can decide themselves if they want a custom mapping in the index that will cause the perfdata field to be a nested type.

Fixes #10511

[martialblog]

Tested it successfully with:

• elasticsearch:8.18.4
• elasticsearch:9.0.3
• opensearch:2.19.3
• opensearch:3.1.0

[oxzi]

First, thanks for your PR.

How is this change reflected in the ElasticSearch? Will this break compatibility due to the format change, especially for custom dashboards?

[martialblog]

This will change the format of what is written to Elasticsearch from this:

    "check_result.perfdata.load1.crit": 10,
    "check_result.perfdata.load1.min": 0,
    "check_result.perfdata.load1.value": 0.24,
    "check_result.perfdata.load1.warn": 5,

To this:

  "check_result.perfdata": [
 {
  "crit": 10,
  "label": "load1",
  "min": 0,
  "value": 0.24,
  "warn": 5
 }
]

Since existing Elasticsearch indices use a dynamic field mapping, no changes will have to be made in Elasticsearch (unless the users have a fixed mapping, but I doubt that, because of the current design of the fields that wouldn't be practical).

But yes, custom dashboards will break.

However, as described in the issue, the current designed was flawed from the beginning. Creating new fields for the metrics is not how this should have been done in the first place. In the SQL world this would be equal to creating a new column for each new metric, which would make no sense.

[martialblog]

This issue here also makes some valid points: #6805

ES can only handle 1000 different fields as default. You can increase the limit, but this will only shift the issue to a later moment. It is also very hard to handle the different perfdata labels in Grafana for example, because I can not just get all perfdata labels for a specific host. With template variables I could query all values with perfdata for a specific host. This is not possible with the current field-solution.

[oxzi]

As discussed offline, I generally support this PR, even if it results in breakages. However, I would like to test it before approving it. Since I don't currently have a testing ES setup, this may take a while.

[martialblog]

A quick and simple Elasticsearch/OpenSearch setup can be done with:

#OS 
podman run --name os2 -p 9201:9200 -m 1g --rm -ti -e "plugins.security.disabled=true" -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Foobar_123456" docker.io/opensearchproject/opensearch:2
podman run --name os3 -p 9201:9200 -m 1g --rm -ti -e "plugins.security.disabled=true" -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Foobar_123456" docker.io/opensearchproject/opensearch:3

# ES
podman run --name es9 -p 9200:9200 -m 1g --rm -it -e "xpack.security.enabled=false" -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:9.0.3
podman run --name es8 -p 9200:9200 -m 1g --rm -it -e "xpack.security.enabled=false" -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:8.18.4

Here some example calls:

# Get all indices
curl -X GET "http://localhost:9200/_cat/indices"

# Get the mapping of an index
curl -X GET "http://localhost:9200/icinga2/_mapping"

# Get the last docs in an index
curl -X GET "http://localhost:9200/icinga2/_search"