[BUG] OTLP arrays are converted to strings

[ghelyar]

Agent Environment
docker image datadog/agent:7.55.3 with DD_LOGS_ENABLED=true and DD_OTLP_CONFIG_LOGS_ENABLED=true

Describe what happened:
Sending arrays to the Datadog agent with OTLP over gRPC causes the array to be displayed as a string in the Datadog UI, and the individual items are not indexed.

If the log is exported from the Datadog UI, it contains "arr": "[\"ab\",\"cd\"]"

Describe what you expected:

I expected these to be preserved as an array instead of flattened to a string, as it is with JSON over TCP

Sending arrays to the Datadog agent on the JSON lines over TCP format causes arrays to be correctly displayed in the Datadog UI.

This is just the following JSON sent over TCP to the agent with a line break afterwards.

{"service":"test","message":"sent from tcp","arr":["ab","cd"]}

This allows filtering on queries like @arr:ab

Steps to reproduce the issue:
This is a C# example but it's just using the protobuf generated from https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/logs/v1/logs_service.proto

The same thing happens with https://github.com/open-telemetry/opentelemetry-dotnet but I just wanted to rule that library out and do it directly.

This should repro in any language as long as it sends an array over OTLP (only tested with gRPC).

using var channel = GrpcChannel.ForAddress("http://localhost:4317");
var client = new LogsService.LogsServiceClient(channel);

var timestamp = (ulong)DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1_000_000;
var logRecord = new LogRecord() {
    TimeUnixNano = timestamp,
    ObservedTimeUnixNano = timestamp,
    SeverityNumber = SeverityNumber.Info,
    SeverityText = "Information",
    Body = new AnyValue { StringValue = "sent from otlp" },
    Attributes = {
        new KeyValue {
            Key = "arr",
            Value = new AnyValue {
                ArrayValue = new ArrayValue {
                    Values = {
                        new AnyValue { StringValue = "ab" },
                        new AnyValue { StringValue = "cd" }
                    }
                }
            }
        }
    }
};

client.Export(new ExportLogsServiceRequest {
    ResourceLogs = {
        new ResourceLogs {
            Resource = new Resource { Attributes = { new KeyValue { Key = "service.name", Value = new AnyValue { StringValue = "test" } } } },
            ScopeLogs = { new ScopeLogs { LogRecords = { logRecord } } }
        }
    }
});

[jackgopack4]

Thanks for reporting this. I instrumented a basic Golang application with both an otlp http logger and an otlp grpc logger and sent the same log, with attributes ["ab", "cd"] to a locally running Datadog Agent (docker container with tag latest) and I am getting the string arr value for both.

However, the specific log exporter I am using uses protobuf to transmit both gRPC and HTTP logs, so I think it's more related to JSON format than gRPC/HTTP.

[ghelyar]

Yes the bug happens when the service exports using OTLP protobuf over gRPC (I did not try HTTP).

The service is sending the array in the protobuf but the Datadog web UI shows it as a string so it's being converted to a string at some point after the agent receives the protobuf, but I don't know exactly where.

The JSON lines over TCP example I gave is what I was expecting it to do, where it is preserved as an array with separate child items in the Datadog UI instead of being converted to a string.

We're trying to migrate from using JSON over TCP to OTLP over gRPC.

[jackgopack4]

I noticed you linked the sending custom logs to Agent via TCP documentation; so you aren't sending them currently to the OTLP endpoint in Agent, just directly to the custom logs endpoint in Agent?

[ghelyar]

We are currently using custom logs with JSON over TCP, and trying to migrate to OpenTelemetry using OTLP over gRPC.

This bug report is only for OTLP in the Datadog agent.

I only provided the JSON/TCP example to demonstrate the expected behaviour, but there is no bug with custom logs, the bug is only when OTLP is used.

The bug with OTLP is currently blocking us from migrating to OpenTelemetry because it is breaking the data that we are sending to Datadog.

[jackgopack4]

Thank you. I understand. I've opened an internal ticket to investigate and fix this behavior.

It might still be helpful if you can provide more details on the config.yaml you are using for the custom logs, as well as the command you are using to transmit this log via TCP to the agent? I am having trouble reproducing it the same way that it shows on your end in the custom log pipeline; it shows up as the "message" as opposed to the attributes.

If I can reproduce the way you are currently doing it via custom logs, it may provide insight on how I can fix the issue with the OTLP intake pipeline.

[ghelyar]

logs_enabled: true in the main config, or DD_LOGS_ENABLED=true environment variable on the agent docker container.

test.d/conf.yaml

  - type: tcp
    port: 10518
    service: test
    source: custom

Then you just send utf-8 encoded JSON with a \n line break after as described above to the agent on TCP 10518.

If it appears as the message it generally means the JSON is invalid and could not be parsed by Datadog.

[jackgopack4]

thanks, was just missing the line break. I'll let you know when I have any updates or further questions

[jackgopack4]

just wanted to let you know I've got a fix pending, waiting on a tweak to our backend api to get released so that I can push this change.