Index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)

[montag]

Describe the bug
Promscale stops writing trace data with an 'Error sending samples to remote storage' error for each sample ingested.

To Reproduce
Attempt to store otel trace data with large SQL or stacktrace tag values.

Expected behavior
Expect traces to be stored and not dropped.

Screenshots
If applicable, add screenshots to help explain your problem.

Configuration (as applicable)

• Promscale Connector: 0.7.0
• TimescaleDB: Timescaledb-ha:pg14.5-ts2.8.1
• Prometheus: 2.39.1
• Jaeger: 1.38.1
• OpenTelemetry: collector 0,61.9
  Version
• Distribution/OS: linux (docker)
• Promscale: 0.15.0
• TimescaleDB: Timescaledb-ha:pg14.5-ts2.8.1

Additional context
Traces are collected without error for a short period of time then the error starts and traces are no longer stored.

Example log entries:

version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:31.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:36.182Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:36.611Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:41.218Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:41.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:46.253Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:46.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:51.288Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:51.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:56.303Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:56.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=info ts=2022-10-13T06:55:57.589Z caller=throughput.go:90 msg=“ingestor throughput” metric-metadata/sec=496 level=warn ts=2022-10-13T06:56:01.337Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:01.587Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:56:06.372Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:06.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:56:11.403Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:11.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.05

[ramonguiu]

@montag could you explain how you are generating and sending the trace data to Promscale? I assume you are using the OpenTelemetry collector? What's the configuration you are using? Are you also sending metrics?

From this log message the issue seems to be with metric ingestion and not traces. Maybe you have something that is automatically generating metrics from trace data?

[sumerman]

FTR: this is the same as #1091, which we fixed for tracing tables. The fix should be similar should we decide to go for it.

[montag]

Here's my otel collector config

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp:
    endpoint: promscale:9202
    tls:
      insecure: true

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

And my app's ENV vars

OTEL_EXPORTER_OTLP_ENDPOINT: otel_collector:4317
OTEL_EXPORTER_OTLP_INSECURE: true
OTEL_EXPORTER_OTLP_TRACES_INSECURE: true
OTEL_SERVICE_NAME: ******
OTEL_PYTHON_LOG_CORRELATION: true      

[montag]

I'm not using otel for app metrics but some of the instrumentation libraries might be sending some. I'm using these libs:

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator

[montag]

I worked around this issue by removing metrics from my otel collector config but at some point, I will need to be able to use them again.