opentelemetry-demo failed to index metrics

[carsonip]

APM Server version (apm-server version): 8.15

Description of the problem including expected versus actual behavior:

Sending opentelemetry demo data to apm-server causes an error in apm-server.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

1. run open-telemtery/opentelemetry-demo, add exporter to export otlphttp to apm-server 8.15
2. check apm-server logs
3. see apm-server error logs

Provide logs (if relevant):

{"log.level":"error","@timestamp":"2024-09-10T13:16:56.049+0100","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"[email protected]/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2024.09.10-000001' (document_parsing_exception): [1:824] failed to parse field [redis.commands] of type [double] in document with id 'Grbd25EB293cGPB_h0Uh'","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

[simitt]

It would be good to get this fixed soon but I am not aware of this issue blocking anything, so don't see it critical for this iteration.

[raultorrecilla]

thanks @simitt, moving it forward then to it-108

[ericywl]

Update: Managed to reproduce this issue in 8.15.5 and 8.17.2, will look into the root cause.

8.15.5

2025-02-24 16:49:53 {"log.level":"error","@timestamp":"2025-02-24T08:49:53.200Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"[email protected]/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2025.02.24-000001' (document_parsing_exception): [1:779] failed to parse field [redis.commands] of type [double] in document with id 'obwmN5UBXPKckzJvBMQr'","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

8.17.2

2025-02-24 16:43:53 {"log.level":"error","@timestamp":"2025-02-24T08:43:53.375Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"[email protected]/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2025.02.24-000003' (document_parsing_exception): ","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

[ericywl]

@carsonip I have found the issue, it seems to be due to some incompatibility with redisreceiver and Elasticsearch.

redisreceiver has these 2 metrics:

• redis.commands (Gauge)
• redis.commands.processed (Sum)

When ES receives redis.commands.processed, it constructs mapping as:

{
    "redis": {
        "commands": {
            "processed": 123
        }
    }
}

but this is incompatible with redis.commands which is a double, hence the error. There are no more errors when either redis.commands or redis.commands.processed is disabled.

Side note: One other perculiar thing is that redis.version is a resource label and shows up in ES as labels.redis_version instead of redis.version because apm-data converts it to redis_version, causing ES to not recognize it as under redis.