fluent-bit not updating pod metadata when pod restarts/updates

[danfinn]

Bug Report

Describe the bug
A kubernetes statefulset had been upgraded which along with it changed the values for the labels of the pods. One of the pods in the statefulset was still shipping logs with the old pod metadata (pod labels, container tag, pod IP) which made it appear in grafana that this pod had not been updated but when you checked the details of the pod on the kubernetes cluster it showed it had been updated and was running the newer version.

It took us a while to track to why our grafana logs from this pod were still appearing as coming from the old version but we eventually tracked it down to fluent-bit. Restarting the fluent-bit pod that is running on the same node as this statefulset pod fixed the issue and corrected the problems with the pod metadata in the logs.

Here is a log viewed from grafana that was shipped via fluent-bit:

{
  "@timestamp": "2026-06-18T15:11:43.021Z",
  "_p": "F",
  "kubernetes.container_hash": "dev.azurecr.io/pw-app-windows@sha256:90bb1ca8e481270412503dd5dbc0b03add41b6cc3ea6428221fdba4d5835d5fb",
  "kubernetes.container_image": "dev.azurecr.io/pw-app-windows:25.0.5-stable.8032",
  "kubernetes.container_name": "pw-app-app-pw",
  "kubernetes.docker_id": "5c86a7792a8bc90d5e8a60d40ab5ca2008469e3c47872dc7481b89d0615da72f",
  "kubernetes.host": "akswp2200006p",
  "kubernetes.labels.app_name": "app-pw",
  "kubernetes.labels.apps_kubernetes_io/pod-index": "1",
  "kubernetes.labels.controller-revision-hash": "app-pw-596b649d4f",
  "kubernetes.labels.namespace": "pw-customer-1001383261",
  "kubernetes.labels.region": "southcentralus",
  "kubernetes.labels.statefulset_kubernetes_io/pod-name": "app-pw-1",
  "kubernetes.labels.version": "25.0.5-stable.8032",
  "kubernetes.namespace_name": "pw-customer-1001383261",
  "kubernetes.pod_id": "a7f564c4-2395-43db-96dd-06117f45e69a",
  "kubernetes.pod_ip": "10.27.249.145",
  "kubernetes.pod_name": "app-pw-1",
  "kubernetes_namespace.labels.azure-key-vault-env-injection": "enabled",
  "kubernetes_namespace.labels.deployment-name": "xorailpw11",
  "kubernetes_namespace.labels.kubernetes_io/metadata_name": "pw-customer-1001383261",
  "kubernetes_namespace.labels.plan-id": "66e88ef473e06f4318bdc3a5",
  "kubernetes_namespace.labels.purpose": "pw-customer-1001383261",
  "kubernetes_namespace.labels.ultimate-id": "1001383261",
  "kubernetes_namespace.name": "pw-customer-1001383261",
  "message": "2026-06-18 15:11:43.021000000 +0000 log.pwapp.bentleylogs: {\"message\":\"2026-06-18 15:11:43,021 ERROR [0x00009f78] pwise.socket - aaSockSec_ReadFromSocket returns -10101\"}",
  "stream": "stdout"
}

and here is a describe on the pod as it's currently running from the k8s clusters:

Name:             app-pw-1
Namespace:        pw-customer-1001383261
Priority:         0
Service Account:  default
Node:             akswp2200006p/10.27.249.104
Start Time:       Wed, 03 Jun 2026 01:32:32 -0400
Labels:           app=app-pw
                  apps.kubernetes.io/pod-index=1
                  controller-revision-hash=app-pw-675b4f4ffb
                  namespace=pw-customer-1001383261
                  region=southcentralus
                  statefulset.kubernetes.io/pod-name=app-pw-1
                  version=26.0.0-stable.8017
Annotations:      checksum/pw-app-configmap: 0ccb05a3780d6998a15a509f7c33f14b26dd97bedc5df2c816890d800fa8e805
                  checksum/pw-app-dmskrnl-configmap: a0d371a59ea2117cdb2b98e261d3bdc53e5b9a260a46680240e616bcf69abc64
                  kubectl.kubernetes.io/restartedAt: 2025-09-03T13:16:21-06:00
                  prometheus.io/path: /metrics
                  prometheus.io/port: 9182
                  prometheus.io/scrape: true
                  secret.reloader.stakater.com/reload: pw-app-tls-cert
Status:           Running
IP:               10.27.249.116
IPs:
  IP:           10.27.249.116
Controlled By:  StatefulSet/app-pw
Containers:
  pw-app-app-pw:
    Container ID:  containerd://aab627760cba61d8a37ea2c8e557dcefb87a92e9d6e8df3cbc18628131359426
    Image:         devbentleyhosted.azurecr.io/pw-app-windows:26.0.0-stable.8017
    Image ID:      devbentleyhosted.azurecr.io/pw-app-windows@sha256:e991398d0a6e889cad022fbd7118b9b3ffcb27f7b00f283f99502726177c9b7e

For example, you can see in the grafana logs that the label for version in the logs coming into grafana are from the "old" (25.0.5-stable.8032) version before this statefulset was updated (to 26.0.0-stable.8017).

Restarting the fluent-bit running on the k8s node that this pod is on fixes the issue and the logs come into our obsv with the correct and up to date labels.

We are running fluent-bit on windows kubernetes nodes in Azure AKS. We build our fluent-bit windows daemonset from this image:
FROM fluent/fluent-bit:windows-2022-4.1.0

Expected behavior
fluent-bit should update pod metadata when a pod is restarted

Screenshots

Your Environment
mentioned above

Additional context
We use this data to track application versions across deployments in different k8s clusters. Currently we cannot trust this data because of this issue with fluent-bit.

[danfinn]

in case it's relevant, here is our fluent-bit configmap:

  custom_parsers.conf: |
    [PARSER]
      Name docker_no_time
      Format json
      Time_Keep On
      Time_Key time
      Time_Format %Y-%m-%dT%H:%M:%S.%L
    [PARSER]
      Name pw_custom_fields
      Format regex
      Regex pwise(?:.+?ODBC ERROR: (?<x_odbc_error_code>[A-Z0-9]{5}))?(?:.+?Invalid object name '(?<x_invalid_object>\w+[^_seq])')?(?:.+?Transaction has been open for (?<x_pwise_db_transaction>\d+\.\d+) seconds)?
      Types x_pwise_db_transaction:float
    [PARSER]
      Name pw_tmstp
      Format regex
      Regex (?<pw_time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{9})\s+(?<log>.*)
      Time_Key pw_time
      Time_Format %Y-%m-%d %H:%M:%S.%L
      Time_Keep Off
      Preserve_Key On
      Reserve_Data On
  fluent-bit.conf: |
    [SERVICE]
      Daemon Off
      Flush 5
      Log_Level info
      Parsers_File C:\fluent-bit\etc\parsers.conf
      Parsers_File C:\fluent-bit\etc\conf\custom_parsers.conf
      HTTP_Server On
      HTTP_Listen 0.0.0.0
      HTTP_Port 2020
      Health_Check On
      storage.path C:\fluent-bit\buffer-storage\
      storage.sync normal
      storage.max_chunks_up 128

    [INPUT]
      Name tail
      Path C:\var\log\containers\*.log
      multiline.parser docker, cri
      DB C:\fluent-bit\buffer-storage\fluent-buffer.db
      Tag kube.*
      Mem_Buf_Limit 32MB
      Skip_Long_Lines On
      read_from_head true
      Refresh_Interval 5
      storage.type filesystem
      Exclude_Path C:\var\log\containers\fluent-bit-*.log,C:\var\log\containers\metricbeat*.log,C:\var\log\containers\*calico-system*.log,C:\var\log\containers\microsoft-defender-*.log,C:\var\log\containers\*gatekeeper-system*.log,C:\var\log\containers\*kube-system*.log,C:\var\log\containers\*kube-public*.log,C:\var\log\containers\*kube-node-lease*.log,C:\var\log\containers\*tigera-operator*.log,C:\var\log\containers\*prometheus*.log,C:\var\log\containers\fluent-daemon*.log

    [INPUT]
      Name tail
      Tag kubelet.err
      Path C:\k\kubelet.err.log
      DB C:\fluent-bit\tail_kubelet.db
      storage.type filesystem

    [INPUT]
      Name              forward
      Listen            0.0.0.0
      Port              2025
      Tag               idx.*
      Buffer_Chunk_Size 100M
      Buffer_Max_Size   500M

    [INPUT]
      Name                 fluentbit_metrics
      Tag                  internal_metrics
      Scrape_interval      2

    [OUTPUT]
      name prometheus_exporter
      match internal_metrics
      host 0.0.0.0
      port 2021
      add_label cluster ${CLUSTER_PREFIX}
      add_label fluent_node_os windows
      add_label fluent_node ${NODE_NAME}

    [FILTER]
      Name modify
      Match idx.*
      Add log_source ics_idx

    # Custom parser to extract timestamps from PW dmskrnl logs
    [FILTER]
      Name parser
      Match kube.*
      Key_Name log
      Parser pw_tmstp
      Reserve_Data true
      Preserve_Key true

    [FILTER]
      Name parser
      Match kube.*
      Key_Name log
      Parser pw_custom_fields
      Reserve_Data true
      Preserve_Key true

    [FILTER]
      Name kubernetes
      Match kube.*
      Kube_URL https://kubernetes.default.svc.cluster.local:443
      Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
      tls.verify Off
      Merge_Log On
      Keep_Log Off
      Use_Kubelet true
      Kubelet_Host ${NODE_IP}
      Merge_Log_Key event
      K8S-Logging.Parser Off
      K8S-Logging.Exclude Off
      Buffer_Size 1MB
      Annotations Off
      Labels On
      Namespace_Labels On

    [FILTER]
      Name modify
      Match kube.*
      Remove kubernetes.docker_id
      Remove kubernetes.container_name
      Remove kubernetes.pod_id
      Remove time
      Remove kubernetes.container_hash

    [FILTER]
      Name nest
      Match kube.*
      Operation lift
      Nested_under event
      Add_prefix event_

    [FILTER]
      Name nest
      Match kube.*
      Operation lift
      Nested_under event_log
      Add_prefix event_log_

    [FILTER]
      Name modify
      Match kube.*
      Rename event_Level level
      Rename event_level level
      Rename event_log_Level level
      Rename event_message message
      Hard_rename event_@timestamp @timestamp
      Rename event_Level Level
      Rename log message
      Rename event_msg message
      Remove event_ts

    [OUTPUT]
      Name es
      Match *
      Host ${FLUENT_ELASTICSEARCH_HOST}
      Port 9200
      HTTP_User ${FLUENT_ELASTICSEARCH_USER}
      HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
      Logstash_Format On
      Logstash_Prefix ${FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX}
      Time_Key @timestamp
      Trace_Error On
      Trace_Output On
      tls On
      tls.verify Off
      Suppress_Type_Name On
      Pipeline ${FLUENT_ELASTICSEARCH_INGEST_PIPELINE}
      Retry_Limit false
      Buffer_Size 32MB
      Replace_Dots On
      storage.total_limit_size 1GB

[patrick-stephens]

Fluent Bit does not poll the K8S API for updates to the metadata to keep the impact of those queries minimal and just caches the data for a configurable amount of time.

As it is a stateful set I suspect your pod names are fixed so not invalidating the cache and just providing the previous set of labels for the pod with the matching name.

You can configure the TTL for the Kubernetes filter using kube_meta_cache_ttl which defaults to keeping it until capacity is reached: https://docs.fluentbit.io/manual/data-pipeline/filters/kubernetes
This will still not immediately pick up the change in labels but you can tune it to an appropriate value for your use case. Be aware of the impact on the K8S API though if you are querying repeatedly.

Restarting the Fluent Bit pods obviously also clears the cache completely hence why it then updates too.

Something like reloader or a similar tool can also be used to auto-restart the Fluent Bit pods when other things change like the stateful set but it's probably easier just to tune kube_meta_cache_ttl to some acceptable value with the caveat that your log entries may report old labels for that period of time (but will then update).

[patrick-stephens]

Potentially there is a feature request here to get the K8S filter to be able to track pod eviction to remove cache entries to then force them to be re-queried. However this would also have to ensure all existing data in the pipeline is flushed before that cache entry is removed otherwise you will have no labels or incorrectly add new labels to old log entries.

[danfinn]

@patrick-stephens do you think Cache_Use_Docker_Id would help in this situation? I'm testing that in our dev environment now

[patrick-stephens]

It may do, assuming you have a Docker ID (or equivalent) from whatever container runtime you are using. This will be part of the filename and then the regex specified to extract it from the filename which can be overridden by regex_parser but check the details here: https://docs.fluentbit.io/manual/data-pipeline/filters/kubernetes#workflow-of-tail-and-kubernetes-filter

Useful to know if it works and what your platform is as well for anyone else who runs across it.