How to increase data storage time (for Prometheus, loki, Tempo) #151

Walli-DO · 2024-11-15T08:04:17Z

Walli-DO
Nov 15, 2024

Hi, I used this image and it successfully deployed and works on my server. However, I forgot that by default the data is stored for only 15 days, I would like to increase the data storage time for Prometheus, Loki and Tempo

services:

  otel-lgtm:
    image: grafana/otel-lgtm:0.7.5
    container_name: otel-lgtm
    volumes:
      # Loki data (logs)
      - ./loki-data/loki:/loki
      # Prometheus data (metrics)
      - ./prometheus-software-data/data/prometheus:/data/prometheus
      # Tempo data (traces)
      - ./tempo-data/tmp/tempo/wal:/tmp/tempo/wal
      - ./tempo-data/tmp/tempo/blocks:/tmp/tempo/blocks
      - ./tempo-data/tmp/tempo/generator/wal:/tmp/tempo/generator/wal
      - ./tempo-data/tmp/tempo/generator/traces:/tmp/tempo/generator/traces
      # Grafana data (dashboards and settings)
      - ./grafana-data/data/grafana:/data/grafana
      - ./grafana-data/otel-lgtm/run-grafana.sh:/otel-lgtm/run-grafana.sh:ro
    environment:
      - GF_PATHS_DATA=/data/grafana
    ports:
      - "3000:3000" # Port for Grafana
      # - "4317:4317" # Port for OTLP gRPC
      # - "4318:4318" # Port for OTLP HTTP
    restart: unless-stopped
    networks:
      - monitoring-net

Before this, I worked with Prometheus separately, and it has a setting that is responsible for the data storage time (--storage.tsdb.retention.time=180d). Can something similar be applied in this case?

services:

  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yaml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=180d"
    volumes:
      - ./prometheus-data/prometheus:/prometheus
      - ./prometheus-data/etc/prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
    restart: unless-stopped
    networks:
      - monitoring-net

And one more question, what is the best way to check that the changes have successfully taken effect and the data is now stored for 180 days?

Answered by Walli-DO

Nov 27, 2024

Yes, ideally, we should create our own build, but at the moment there is no time for that, so we are improving the existing solution.

I completely agree that the data should be stored outside the container. In the future, we plan to move to our own build, so I have distributed the data across different folders.

Here is the approximate solution that satisfies me at the moment.

1.) Docker Compose yml file

Create the docker-compose.yml file:

vim docker-compose.yml

Content of the file docker-compose.yml:

# version: '3.8'

services:

  otel-lgtm:
    image: grafana/otel-lgtm:0.7.5
    container_name: otel-lgtm
    volumes:
      # Loki data (logs)
      - ./loki-data/loki:/loki
      - ./loki-…

View full answer

zeitlinger · 2024-11-21T11:28:16Z

zeitlinger
Nov 21, 2024
Maintainer

Can something similar be applied in this case?

I think so - just add it next to

docker-otel-lgtm/docker/prometheus.yaml

Line 25 in 2988e1f

out_of_order_time_window: 10m

what is the best way to check that the changes have successfully taken effect and the data is now stored for 180 days?

@bboreham can you help?

0 replies

bboreham · 2024-11-21T12:19:04Z

bboreham
Nov 21, 2024

I don't see any provision in the image to specify additional command-line flags or Prometheus config.

The key point on the front page is "intended for development, demo, and testing environments". It's not for serious work.
For instance if you want to retain data you should have it on a volume, not inside the container.

I can suggest either you learn more about these things and make your own image, or look at some Prometheus packages that are intended for production use.

1 reply

zeitlinger Nov 22, 2024
Maintainer

This also used to be my standard reply.
It turns out users appreciate to start with a complete package that works - despite this known limitation.

AFAIK, there is no other solution that packages the LGTM stack (with promethus).

Walli-DO · 2024-11-27T10:47:47Z

Walli-DO
Nov 27, 2024
Author

Yes, ideally, we should create our own build, but at the moment there is no time for that, so we are improving the existing solution.

I completely agree that the data should be stored outside the container. In the future, we plan to move to our own build, so I have distributed the data across different folders.

Here is the approximate solution that satisfies me at the moment.

1.) Docker Compose yml file

Create the docker-compose.yml file:

vim docker-compose.yml

Content of the file docker-compose.yml:

# version: '3.8'

services:

  otel-lgtm:
    image: grafana/otel-lgtm:0.7.5
    container_name: otel-lgtm
    volumes:
      # Loki data (logs)
      - ./loki-data/loki:/loki
      - ./loki-data/otel-lgtm/loki-config.yaml:/otel-lgtm/loki-config.yaml:ro # Configuration
      # Prometheus data (metrics)
      - ./prometheus-software-data/data/prometheus:/data/prometheus
      - ./prometheus-software-data/otel-lgtm/run-prometheus.sh:/otel-lgtm/run-prometheus.sh:ro # Configuration
      # Tempo data (traces)
      - ./tempo-data/tmp/tempo/wal:/tmp/tempo/wal
      - ./tempo-data/tmp/tempo/blocks:/tmp/tempo/blocks
      - ./tempo-data/tmp/tempo/generator/wal:/tmp/tempo/generator/wal
      - ./tempo-data/tmp/tempo/generator/traces:/tmp/tempo/generator/traces
      - ./tempo-data/otel-lgtm/tempo-config.yaml:/otel-lgtm/tempo-config.yaml:ro # Configuration
      # Grafana data (dashboards)
      - ./grafana-data/data/grafana:/data/grafana
      - ./grafana-data/otel-lgtm/run-grafana.sh:/otel-lgtm/run-grafana.sh:ro # Configuration
    environment:
      - GF_PATHS_DATA=/data/grafana
    ports:
      - "3000:3000" # Port for Grafana
      # - "9090:9090" # Port for Prometheus
      # - "3100:3100" # Port for Loki
      # - "3200:3200" # Port for Tempo
      # - "4317:4317" # Port for OTLP gRPC
      # - "4318:4318" # Port for OTLP HTTP
    restart: unless-stopped
    networks:
      - monitoring-net

networks:
  monitoring-net:
    name: monitoring-net
    external: true

2.) Loki

Create a folder for Loki data:

mkdir -p ./loki-data/loki

Create the loki-config.yaml file:

mkdir -p ./loki-data/otel-lgtm
vim ./loki-data/otel-lgtm/loki-config.yaml

Content of the file loki-config.yaml:

The parameter responsible for log retention period limits_config.retention_period: 4320h (180 days in hours):

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  retention_period: 4320h # 180 days in hours

3.) Prometheus

Create a folder for Prometheus data:

mkdir -p ./prometheus-data/data/prometheus

Create the run-prometheus.sh script:

mkdir -p ./prometheus-data/otel-lgtm
vim ./prometheus-data/otel-lgtm/run-prometheus.sh

Content of the file run-prometheus.sh:

The parameter responsible for log retention period --storage.tsdb.retention.time=180d (180 days):

#!/bin/bash

source ./logging.sh

run_with_logging "Prometheus ${PROMETHEUS_VERSION}" "${ENABLE_LOGS_PROMETHEUS:-false}" ./prometheus/prometheus \
      --web.enable-remote-write-receiver \
      --enable-feature=otlp-write-receiver \
      --enable-feature=exemplar-storage \
      --enable-feature=native-histograms \
      --storage.tsdb.path=/data/prometheus \
      --storage.tsdb.retention.time=180d \
      --config.file=./prometheus.yaml

Make the Prometheus startup script executable:

chmod +x ./prometheus-data/otel-lgtm/run-prometheus.sh

4.) Tempo

Create folders for Tempo data:

mkdir -p ./tempo-data/tmp/tempo/wal
mkdir -p ./tempo-data/tmp/tempo/blocks
mkdir -p ./tempo-data/tmp/tempo/generator/wal
mkdir -p ./tempo-data/tmp/tempo/generator/traces

Create the tempo-config.yaml file:

mkdir -p ./tempo-data/otel-lgtm
vim ./tempo-data/otel-lgtm/tempo-config.yaml

Content of the file tempo-config.yaml:

The parameter responsible for log retention period compactor.compaction.block_retention: 4320h (180 days in hours):

server:
  http_listen_port: 3200
  grpc_listen_port: 9096

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "localhost:4417"
        http:
          endpoint: "localhost:4418"

storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal
    local:
      path: /tmp/tempo/blocks

metrics_generator:
  processor:
    local_blocks:
      filter_server_spans: false
  traces_storage:
    path: /tmp/tempo/generator/traces
  storage:
    path: /tmp/tempo/generator/wal
    # TODO: support otlp at metrics_generator
    remote_write:
      - url: http://localhost:9090/api/v1/write
        send_exemplars: true

overrides:
  metrics_generator_processors: [service-graphs, local-blocks]

compactor:
  compaction:
    block_retention: 4320h # 180 days in hours

5.) Grafana

Create a folder for Grafana data:

mkdir -p ./grafana-data/data/grafana

Create the Grafana script:

mkdir -p ./grafana-data/otel-lgtm
vim ./grafana-data/otel-lgtm/run-grafana.sh

Content of the file run-grafana.sh:

#!/bin/bash

source ./logging.sh

export GF_AUTH_ANONYMOUS_ENABLED=false
export GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
export GF_AUTH_DISABLE_LOGIN_FORM=false

cd ./grafana
run_with_logging "Grafana ${GRAFANA_VERSION}" "${ENABLE_LOGS_GRAFANA:-false}>" ./bin/grafana server

Make the Grafana startup script executable:

chmod +x ./grafana-data/otel-lgtm/run-grafana.sh

0 replies

zeitlinger · 2024-11-27T11:11:55Z

zeitlinger
Nov 27, 2024
Maintainer

great - I'll leave that here for others to copy

0 replies

Walli-DO · 2024-11-27T11:32:53Z

Walli-DO
Nov 27, 2024
Author

OK, thank you

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to increase data storage time (for Prometheus, loki, Tempo) #151

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

How to increase data storage time (for Prometheus, loki, Tempo) #151

Walli-DO Nov 15, 2024

1.) Docker Compose yml file

Replies: 5 comments · 1 reply

zeitlinger Nov 21, 2024 Maintainer

bboreham Nov 21, 2024

zeitlinger Nov 22, 2024 Maintainer

Walli-DO Nov 27, 2024 Author

1.) Docker Compose yml file

2.) Loki

3.) Prometheus

4.) Tempo

5.) Grafana

zeitlinger Nov 27, 2024 Maintainer

Walli-DO Nov 27, 2024 Author

Walli-DO
Nov 15, 2024

Replies: 5 comments 1 reply

zeitlinger
Nov 21, 2024
Maintainer

bboreham
Nov 21, 2024

zeitlinger Nov 22, 2024
Maintainer

Walli-DO
Nov 27, 2024
Author

zeitlinger
Nov 27, 2024
Maintainer

Walli-DO
Nov 27, 2024
Author