Not able to send logs to multiple outputs(Splunk and Filesystem)

[Tamannaverma1912]

Hi, So we are trying to have multiple outputs, One of which will be splunk and other one will be file system. I can see my logs being sent to file-system after some time but not to splunk. I don't know why this is happening and what am I going wrong in here? Can anyone help with this?
Here is my configmap
`apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: laser-test1
labels:
log-app: fluentd
data:
fluent.conf: |

log_level trace

<label @FLUENT_LOG>
  <match fluent.**>
    @type null
  </match>
</label>

<source>
  @type tail
  path "/var/log/containers/*#{ENV['POD_NAME']}*_#{ENV['POD_NAMESPACE']}_#{ENV['CONTAINER_NAME']}-*.log"
  pos_file /var/log/fluentd-containers.log.pos
  read_from_head true
  exclude_path ["/var/log/containers/*_kube-system_*.log", "/var/log/containers/*_gpu-monitoring_*.log", "/var/log/containers/*_monitoring_*.log", "/var/log/containers/*_cert-manager_*.log","/var/log/containers/*_fluentd*.log"]
  tag kube.*
  <parse>
    @type regexp
    expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
    </parse>
</source>

<filter kube.var.log.containers.**>
@type record_transformer
enable_ruby true
<record>
  tag ${tag}
  namespace_name  ${tag.match(/.*_([^_]+)_([^_]+)$/)[1]}
  pod_name ${tag.match(/.*containers\.([^_]+)_([^_]+)_[^_]+$/)[1]}
  container_name ${tag.match(/.*_([^_]+)-[0-9a-f]+\.log$/)[1]}
  </record>
  remove_keys stream,logtag
</filter>

<match kube.**>
@type copy
<store>
  @type file
  #path /Users/jgunasekaran/fluent-bit
  path "#{ENV['LOG_PATH']}/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S"
  append true
  <buffer namespace_name,pod_name,container_name,time>
    @type file
    flush_interval 10s
    chunk_limit_size 265m
    total_limit_size 1g 
    timekey  "#{ENV['timekey']}"
    timekey_wait 30s
    timekey_use_utc true
    overflow_action block
  </buffer>
  <format>
    @type remove_metadata
  </format>
</store>
<store>
  @type splunk_hec
  hec_port 8088
  hec_token "#{ENV['FLUENTD_HEC_TOKEN']}"
  hec_host "splunk-us-hec-internal.adobelaas.com"
  sourcetype sandbox-stage01   #Parameterise this later
</store>
</match>` and Here's my test Deployment```

 apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd-splunk-test
  namespace: laser-test1
  labels:
    log-app: fluentd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccountName: default
      imagePullSecrets:
      - name: adobe-registry-key
      containers:
        - name: nginx
          image: nginx:1.21.1
          ports:
            - containerPort: 80
          command: ["sh", "-c", "while true; do  date; echo  'This is multiple output testing'; sleep 0.05; done"]
          volumeMounts:
            - name: fluentd-forward-logs
              mountPath: /sensei-fs
        - name: fluentd
          image: docker-matrix-experiments-snapshot.dr-uw2.adobeitc.com/senf-infra/multiple-output-logs:1.0.2
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 123436
            runAsGroup: 100
          ports:
          - containerPort: 24224
            name: forward
            protocol: TCP
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: CONTAINER_NAME
            value: nginx
          - name: LOG_PATH
            value: "/sensei-fs/users/tamannaverm/fluentd"
          - name: timekey
            value: "10m"
          - name: ENABLE_METRICS
            value: "On"
          - name: SPLUNK_HOST
            value: "splunk-us-hec-internal.adobelaas.com"
          - name: SPLUNK_PORT
            value: "8088"
          - name: SPLUNK_INDEX
            value: "runai_service_prod"
          - name: SPLUNK_TLS
            value: "On"
          - name: FLUENTD_HEC_TOKEN
            valueFrom:
              secretKeyRef:
                name: splunk-token
                key: hec_token
          volumeMounts:
          - name: varlog
            mountPath: /var/log
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
            readOnly: true
          - name: mnt
            mountPath: /mnt
            readOnly: true
          - name: fluentd-forward-logs
            mountPath: /sensei-fs
          - name: fluentd-forward-config
            mountPath: /fluentd/etc/
            # command: ["sleep", "inf"]
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: mnt
        hostPath:
          path: /mnt
      - name: fluentd-forward-logs
        persistentVolumeClaim:
          claimName: laser-test1-sensei-fs-pvc
          readOnly: false
      - name: fluentd-forward-config
        configMap:
          name: fluentd-config

and the image I am using here contains custom plugin we had written and splunk support. I tried config map with separate matches one for each splunk and file-system, It was working fine and Here are my fluentd container logs

2024-06-27 12:29:40 +0000 [info]: fluent/log.rb:362:info: starting fluentd-1.17.0 pid=1 ruby="3.2.4"
2024-06-27 12:29:40 +0000 [info]: fluent/log.rb:362:info: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2024-06-27 12:29:40 +0000 [info]: #0 fluent/log.rb:362:info: init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-06-27 12:29:40 +0000 [info]: fluent/log.rb:362:info: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered output plugin 'null'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered metrics plugin 'local'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered buffer plugin 'memory'
2024-06-27 12:29:40 +0000 [info]: fluent/log.rb:362:info: adding filter pattern="kube.var.log.containers.**" type="record_transformer"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered filter plugin 'record_transformer'
2024-06-27 12:29:40 +0000 [info]: fluent/log.rb:362:info: adding match pattern="kube.**" type="copy"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered output plugin 'copy'
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: adding store type="file"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered output plugin 'file'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered buffer plugin 'file'
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: 'flush_interval' is ignored because default 'flush_mode' is not 'interval': 'lazy'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered formatter plugin 'remove_metadata'
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: adding store type="splunk_hec"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered output plugin 'splunk_hec'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered formatter plugin 'json'
2024-06-27 12:29:40 +0000 [info]: fluent/log.rb:362:info: adding source type="tail"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered parser plugin 'regexp'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered parser plugin 'multiline'
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: registered input plugin 'tail'
2024-06-27 12:29:40 +0000 [info]: #0 fluent/log.rb:362:info: starting fluentd worker pid=20 ppid=1 worker=0
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: buffer started instance=2460 stage_size=0 queue_size=0
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: flush_thread actually running
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: enqueue_thread actually running
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: enqueueing all chunks in buffer instance=2460
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde2656b6f699b4c128ffcd7563da0.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde26575fe3ffafc4c0a830fab9ffe.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde267bb5ab4b73cc5ff39b68c9c93.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde267e54c4c7c2d0847597cead3ce.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde267e93e476067b92e268a597ed8.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde26842c84a3658f16191706f621c.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde268526c3442c47c1e2181e9dc66.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde26853a73e679be146cadb41f2d6.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: restoring buffer file: path = /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde2691b3eb6fa64dbe51e654970ba.log
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: buffer started instance=2400 stage_size=709399 queue_size=8004
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: flush_thread actually running
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: enqueue_thread actually running
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: enqueueing all chunks in buffer instance=2400
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Unparsable line in pos_file: /var/log/containers/fluentd-splunk-test-5d4f45dc9b-k9mpd_laser-test1_nginx-5ad899861ea6ce4616eb23bbdde6863a79d6c1a8d12a17686750d4488ad409bc.log000000000001ccd2a	00000000304003d5

2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Unparsable line in pos_file: 000000000021e93600000000002feca90000000000301d18000000000010acde0000000000271fd600000000006d166400000000002c92fa
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: Compacted entries: []
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: Remove missing entries. existing_targets=["/var/log/containers/fluentd-splunk-test-5d4f45dc9b-4w8m7_laser-test1_nginx-142ed6e6976bc991d0e23498736cd3c108b3d68d1140635aa2f21052e4ab39ab.log"] entries_after_removing=[]
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: tailing paths: target = /var/log/containers/fluentd-splunk-test-5d4f45dc9b-4w8m7_laser-test1_nginx-142ed6e6976bc991d0e23498736cd3c108b3d68d1140635aa2f21052e4ab39ab.log | existing = 
2024-06-27 12:29:40 +0000 [info]: #0 fluent/log.rb:362:info: following tail of /var/log/containers/fluentd-splunk-test-5d4f45dc9b-4w8m7_laser-test1_nginx-142ed6e6976bc991d0e23498736cd3c108b3d68d1140635aa2f21052e4ab39ab.log
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: writing events into buffer instance=2400 metadata_size=1
2024-06-27 12:29:40 +0000 [debug]: #0 fluent/log.rb:341:debug: Created new chunk chunk_id="61bde48f1bb3820addf281d8a9e009c7" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1719490800, tag=nil, variables={:namespace_name=>"laser-test1", :pod_name=>"fluentd-splunk-test-5d4f45dc9b-4w8m7", :container_name=>"nginx"}, seq=0>
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: chunk /sensei-fs/users/tamannaverm/fluentd/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/buffer.b61bde48f1bb3820addf281d8a9e009c7.log size_added: 1830 new_size: 1830
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: dequeueing a chunk instance=2400
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: chunk dequeued instance=2400 metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1719490800, tag=nil, variables={:namespace_name=>"tenant-ea37ae57a28d44f485b7ca993efe3e6e-firefly-project-uv", :pod_name=>"testpod3", :container_name=>"nginx"}, seq=0>
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: trying flush for a chunk chunk="61bde268526c3442c47c1e2181e9dc66"
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: adding write count instance=2380
2024-06-27 12:29:40 +0000 [trace]: #0 fluent/log.rb:319:trace: executing sync write chunk="61bde268526c3442c47c1e2181e9dc66"
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending
2024-06-27 12:29:40 +0000 [warn]: #0 fluent/log.rb:383:warn: Event after formatting was blank, not sending

[Tamannaverma1912]

Can anyone help here?