Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opentelemetry-demo failed to index metrics #14029

Open
carsonip opened this issue Sep 10, 2024 · 4 comments
Open

opentelemetry-demo failed to index metrics #14029

carsonip opened this issue Sep 10, 2024 · 4 comments

Comments

@carsonip
Copy link
Member

APM Server version (apm-server version): 8.15

Description of the problem including expected versus actual behavior:

Sending opentelemetry demo data to apm-server causes an error in apm-server.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

  1. run open-telemtery/opentelemetry-demo, add exporter to export otlphttp to apm-server 8.15
  2. check apm-server logs
  3. see apm-server error logs

Provide logs (if relevant):

{"log.level":"error","@timestamp":"2024-09-10T13:16:56.049+0100","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"[email protected]/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2024.09.10-000001' (document_parsing_exception): [1:824] failed to parse field [redis.commands] of type [double] in document with id 'Grbd25EB293cGPB_h0Uh'","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}
@simitt
Copy link
Contributor

simitt commented Jan 13, 2025

It would be good to get this fixed soon but I am not aware of this issue blocking anything, so don't see it critical for this iteration.

@raultorrecilla
Copy link

raultorrecilla commented Jan 13, 2025

thanks @simitt, moving it forward then to it-108

@ericywl
Copy link
Contributor

ericywl commented Feb 24, 2025

Update: Managed to reproduce this issue in 8.15.5 and 8.17.2, will look into the root cause.

8.15.5

2025-02-24 16:49:53 {"log.level":"error","@timestamp":"2025-02-24T08:49:53.200Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"[email protected]/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2025.02.24-000001' (document_parsing_exception): [1:779] failed to parse field [redis.commands] of type [double] in document with id 'obwmN5UBXPKckzJvBMQr'","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

8.17.2

2025-02-24 16:43:53 {"log.level":"error","@timestamp":"2025-02-24T08:43:53.375Z","log.origin":{"function":"github.com/elastic/go-docappender/v2.(*Appender).flush","file.name":"[email protected]/appender.go","file.line":443},"message":"failed to index documents in '.ds-metrics-apm.app.unknown-default-2025.02.24-000003' (document_parsing_exception): ","service.name":"apm-server","documents":1,"ecs.version":"1.6.0"}

@ericywl
Copy link
Contributor

ericywl commented Feb 24, 2025

@carsonip I have found the issue, it seems to be due to some incompatibility with redisreceiver and Elasticsearch.

redisreceiver has these 2 metrics:

  • redis.commands (Gauge)
  • redis.commands.processed (Sum)

When ES receives redis.commands.processed, it constructs mapping as:

{
    "redis": {
        "commands": {
            "processed": 123
        }
    }
}

but this is incompatible with redis.commands which is a double, hence the error. There are no more errors when either redis.commands or redis.commands.processed is disabled.

Side note: One other perculiar thing is that redis.version is a resource label and shows up in ES as labels.redis_version instead of redis.version because apm-data converts it to redis_version, causing ES to not recognize it as under redis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants