How to Create Indexed Attributes with opentelemetry-spring-boot-starter Exporting to ELK

[ghost]

I am using opentelemetry-spring-boot-starter 2.7.0 in my application, along with the OpenTelemetry Collector to centralize metrics and traces, exporting this data to the ELK Stack. In the Elastic APM documentation, when using the Elastic agent, there is a functionality called transaction.addTag that allows adding tags/attributes which are automatically indexed in Elasticsearch (documentation: Elastic APM Java Agent - addTag).

My question is: What is the equivalent of this functionality when using only opentelemetry-spring-boot-starter and OpenTelemetry? I tried the following code:

Span.current().setAttribute("order.id", order.getId());

However, I noticed that this only creates non-indexed attributes in Elasticsearch. How can I ensure that these attributes are indexed correctly?

[laurit]

cc @jackshirazi

[SylvainJuge]

Hi @Watlaass,

I am not very familiar with the OpenTelemetry collector exporting to ELK, however I can describe how it works with the APM server:

With the APM agent, the fields that are added with addTag in the agent API are stored in labels.*, which are indexed by default.

When using an OpenTelemetry API or instrumentation agent, a custom attribute order.id should be stored in labels.order_id or numeric_labels.order_id depending if it's a string or a numeric value.

So while the naming isn't 100% preserved, you should be able to search by order ID.

Can you link on the collector/extensions version and configuration of the otel collector that you used to write to Elasticsearch here ?

[Watlas]

Hi @SylvainJuge

Yes, the field is indeed sent as labels.order_id. However, the issue is that this attribute is not being indexed in ELK, which prevents global searches in Kibana or creating dashboards based on this attribute. As a result, the field remains just as metadata within the specific transaction, without the ability to be used in broader queries or visualizations.

As requested, I’m sharing the relevant configuration below. Here are the Docker Compose file and the specific part of the ELK configuration:

  collector:
    image: otel/opentelemetry-collector-contrib:106.1
    networks:
      - elk
    command:
      - --config=/etc/otelcol-contrib/otel-collector.yml
      - --feature-gates=receiver.prometheusreceiver.UseCreatedMetric,receiver.prometheusreceiver.EnableNativeHistograms
    volumes:
      - ./collector/otel-collector.yml:/etc/otelcol-contrib/otel-collector.yml
    restart: always
    ports:
      - "4317:4317"
      - "4318:4318"
      - "8889:8889"
      - "8888:8888"
      - "1888:1888" # pprof extension
      - "13133:13133" # health_check extension
      - "55679:55679" # zpages extension
    depends_on:
      - loki
      - jaeger
      - zipkin
      - tempo
      - fleet-server

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    enable_open_metrics: true

  prometheusremotewrite:
    endpoint: 'http://prometheus:9090/api/v1/write'
    tls:
      insecure: true

  otlp/jaeger:
    endpoint: "http://jaeger:4317"
    tls:
      insecure: true

  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    format: proto

  otlp/tempo:
    endpoint: "http://tempo:4317"
    tls:
      insecure: true

  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

  otlp/elastic:
    endpoint: "http://apm-server:8200"
    tls:
      insecure: true

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [ health_check, pprof, zpages ]
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ otlp/jaeger, zipkin, otlp/tempo, otlp/elastic ]
    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ otlp/elastic, prometheusremotewrite, prometheus ]
    logs:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ otlp/elastic, loki ]

[SylvainJuge]

So if I understand this correctly you are exporting to apm-server via the OTLP protocol, not using the otel collector Elasticsearch exporter with the ECS mapping, so what I wrote previously about APM server mapping should apply here.

The labels should be indexed as keywords, can you check the index template in Kibana by going in Stack Management > Index Management > Index Templates tab, then open the traces-apm managed template.

Then on the right panel, on the 4th and last tab named Preview you should see the JSON of the index template, it should contain this for labels.* and similar for numerical_labels.*:

{
  "labels": {
    "path_match": "labels.*",
    "match_mapping_type": "string",
    "mapping": {
      "type": "keyword"
    }
  }
},