[Q&A] filter_grep exclude not filtering out?

[meditator3]

What is a problem?

using:

 <filter containers.**>
        @type grep
          <exclude>
            # previous pattern /^info$/i
            key level
            pattern \s*\\"info\\"
          </exclude>
      </filter>

which is supposedly supposed to filter out all info logs.
I"ve tried a bunch of others. also didn't work

also I'm unsure how fluentD is reading my messages, those that are about to be filtered out?
i have on my a parsing to json, so i assume, all my raw logs are parsed as json? so it should be able to ID key-level ?
this is example of a part of a raw log file i took from coralogix(which takes our fluentD to more readable)-

\"status\":200,\"event\":\"Response log\",\"level\":\"info\"}","stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"

this is my <source> in case its needed:

Describe the configuration of Fluentd###

 coralogix.conf: |-
      <system>
        log_level "#{ENV['LOG_LEVEL']}"
      </system>
    
      <source>
        @type tail
        @id in_tail_container_logs
        path /var/log/containers/*.log
        exclude_path ["/var/log/containers/*_argocd_*.log", "/var/log/containers/*_default_*.log", "/var/log/containers/*_kube-node-lease_*.log", "/var/log/containers/*_kube-system_*.log"]
        path_key filename
        pos_file /var/log/fluentd-containers.log.pos
        tag raw.containers.*
        read_from_head false
        <parse>
        @type multi_format
          <pattern>
            format json
            time_key time
            time_format %Y-%m-%dT%H:%M:%S.%NZ
            keep_time_key true
          </pattern>
          <pattern>
            format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
            time_format %Y-%m-%dT%H:%M:%S.%N%:z
            keep_time_key true
          </pattern>
        </parse>
      </source>

   
      # This Segment is using the detect-exceptions-plugin
      # It will scan the "log" field for well known sctructures of Exception messages
      # I any are found it will assemble them and will aatempt to resolve the multiline.
      # this segment will also remove the "raw" prefix fro mthe tag
      <match raw.containers.**>
        @id raw.containers
        @type detect_exceptions
        remove_tag_prefix raw
        message log
        stream stream
        multiline_flush_interval 5
        max_bytes 500000
        max_lines 1000
      </match>

      # This Segment takes the raw logs and enriches them with the kubernetes metadata
      # Other Parts in this config are relaying on it.
      <filter containers.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        skip_labels false
        skip_container_metadata false
        skip_namespace_metadata true
        skip_master_url true
      </filter>
      # this filters info messages because of overloaded quota limit on coralogix(0.03 daily)
 ....here is the filter_grep i showed before..

Describe the logs of Fluentd

No response

Environment

- Fluentd version:
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version:

[daipom]

<filter containers.**>

raw.containers.**??

[meditator3]

same result.. it used to be that..

[daipom]

I see!
There are 2 points!

• The setting order is important. Basically, You need to write filter setting before match setting.
  • Because the events are processed in the order in which tags are matched.
• You need to use the record_accessor syntax to specify a nested field.
  • https://docs.fluentd.org/filter/grep#less-than-regexp-greater-than-directive
  • https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax

Example:

<source>
 @type sample
 tag raw.containers
 sample {"log":{"msg":"normal message","level":"info"},"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
</source>

<source>
 @type sample
 tag raw.containers
 sample {"log":{"msg":"warning message","level":"warn"},"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
</source>

<filter raw.containers.**>
 @type grep
 <exclude>
   key $.log.level
   pattern /info/
 </exclude>
</filter>

<match raw.containers.**>
 @type stdout
</match>

This excludes info events, and output only warn events.

2025-05-22 20:31:53.062383764 +0900 raw.containers: {"log":{"msg":"warning message","level":"warn"},"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
2025-05-22 20:31:54.064916350 +0900 raw.containers: {"log":{"msg":"warning message","level":"warn"},"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
2025-05-22 20:31:55.066716407 +0900 raw.containers: {"log":{"msg":"warning message","level":"warn"},"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
...

[meditator3]

thank you, I will try to implement the key pattern you gave, it seems more accurate indeed.

but i don't understand the sample you gave? its hardcoded date...
also that sample doesn't follow the sample i gave, i think?

{"log":"{\"name\":\"core\",\"request_id\":\"30eefd49-26b8-4e1f-9d54-43fabbbc1e47\",\"method\":\"PUT\",\"path\":\"\/api\/v1\/skills-progress\/45c26729-3838-480a-9aec-9e1d465f1c48\/steps\/cf84e53d-9ae2-4f7f-9022-d33208992848\/widgets-data\/3a5b8580-54f1-4d8d-960b-71bb68b183ed\/content\",\"client_ip\":\"10.10.24.65:34949\",\"user_agent\":\"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/136.0.0.0 Safari\/537.36\",\"jwt_validated\":\"True\",\"user_id\":\"cf91adaf-9d50-4712-8203-f0b0f6e093d6\",\"user_name\":\"\\u05e8\\u05d5\\u05ea\\u05d9 \\u05e1\\u05e8\\u05d1\\u05e8\\u05e0\\u05d9\\u05e7\",\"org_id\":\"0f5659ba-5ff9-4a5e-9055-5128c007a0ce\",\"jwt_provided\":true,\"status\":200,\"event\":\"Response log\",\"level\":\"info\"}","stream":"stdout","time":"2025-05-21T08:32:49.702332001Z","docker":{"container_id":"9a69c4182ac4fc319a581ba6d853eb86f1c06553e62a73fcd2b1550f98b7b5ea"}} 

it seems my raw log key has a value of a string, which within we have keys:patterns as a dictionary/object.

how can fluentD parser the string to a readable key value? instead of the string i just showed in my sample?

EDIT:
modifying only filter exclude for $.log.level - didn't work

[daipom]

but i don't understand the sample you gave? its hardcoded date...
also that sample doesn't follow the sample i gave, i think?

Yes. That sample is just an example.

it seems my raw log key has a value of a string,

I see!
Now I understand the problem.
The value of the log key is a String, not a Hash.

{"log":"{\"name\":\"core\",...,\"level\":\"info\"}","stream":"stdout",...}

So Fluentd can not access the value of the level key.
To do this, Fluentd needs to parse the internal string: "{\"name\":\"core\",...,\"level\":\"info\"}", recursively.
There would be some way to do that...

[daipom]

You can parse an internal String value in JSON, you can as follows.

<filter raw.containers.**>
  @type parser
  key_name log
  reserve_time
  reserve_data
  remove_key_name_field
  <parse>
    @type json
  </parse>
</filter>

By this, the JSON will be flattened, and you can filter the events by an internal key.
Example:

<source>
  @type sample
  tag raw.containers
  sample {"log":"{\"msg\":\"normal message\",\"level\":\"info\"}","stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
</source>

<source>
  @type sample
  tag raw.containers
  sample {"log":"{\"msg\":\"warning message\",\"level\":\"warn\"}","stream":"stdout","time":"2025-05-21T08:32:49.702332001Z"}
</source>

<filter raw.containers.**>
  @type parser
  key_name log
  reserve_time
  reserve_data
  remove_key_name_field
  <parse>
    @type json
  </parse>
</filter>

<filter raw.containers.**>
  @type grep
  <exclude>
    key level
    pattern /info/
  </exclude>
</filter>

<match raw.containers.**>
  @type stdout
</match>

2025-05-26 16:18:08.013017434 +0900 raw.containers: {"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z","msg":"warning message","level":"warn"}
2025-05-26 16:18:09.014621117 +0900 raw.containers: {"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z","msg":"warning message","level":"warn"}
...

The time value is not applied to the event time in the example. It is because of in_sample. If you use other input plugins and parse the time correctly, the event time will be applied correctly.

More specific example:

<source>
  @type tail
  path /tmp/test.log
  tag raw.containers.*
  read_from_head
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    keep_time_key
  </parse>
</source>

<filter raw.containers.**>
  @type parser
  key_name log
  reserve_time
  reserve_data
  remove_key_name_field
  <parse>
    @type json
  </parse>
</filter>

<filter raw.containers.**>
  @type grep
  <exclude>
    key level
    pattern /info/
  </exclude>
</filter>

<match raw.containers.**>
  @type stdout
</match>

/tmp/test.log:

{"log":"{\"name\":\"core\",\"request_id\":\"30eefd49-26b8-4e1f-9d54-43fabbbc1e47\",\"method\":\"PUT\",\"path\":\"\/api\/v1\/skills-progress\/45c26729-3838-480a-9aec-9e1d465f1c48\/steps\/cf84e53d-9ae2-4f7f-9022-d33208992848\/widgets-data\/3a5b8580-54f1-4d8d-960b-71bb68b183ed\/content\",\"client_ip\":\"10.10.24.65:34949\",\"user_agent\":\"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/136.0.0.0 Safari\/537.36\",\"jwt_validated\":\"True\",\"user_id\":\"cf91adaf-9d50-4712-8203-f0b0f6e093d6\",\"user_name\":\"\\u05e8\\u05d5\\u05ea\\u05d9 \\u05e1\\u05e8\\u05d1\\u05e8\\u05e0\\u05d9\\u05e7\",\"org_id\":\"0f5659ba-5ff9-4a5e-9055-5128c007a0ce\",\"jwt_provided\":true,\"status\":200,\"event\":\"Response log\",\"level\":\"info\"}","stream":"stdout","time":"2025-05-21T08:32:49.702332001Z","docker":{"container_id":"9a69c4182ac4fc319a581ba6d853eb86f1c06553e62a73fcd2b1550f98b7b5ea"}}

Result:

• The event is excluded correctly, and there is no output.
• If you comment out the filter_grep setting, you can see the following ouput.

2025-05-21 08:32:49.702332001 +0900 raw.containers.test.fluentd.case.gh-4973.input.log: {"stream":"stdout","time":"2025-05-21T08:32:49.702332001Z","docker":{"container_id":"9a69c4182ac4fc319a581ba6d853eb86f1c06553e62a73fcd2b1550f98b7b5ea"},"name":"core","request_id":"30eefd49-26b8-4e1f-9d54-43fabbbc1e47","method":"PUT","path":"/api/v1/skills-progress/45c26729-3838-480a-9aec-9e1d465f1c48/steps/cf84e53d-9ae2-4f7f-9022-d33208992848/widgets-data/3a5b8580-54f1-4d8d-960b-71bb68b183ed/content","client_ip":"10.10.24.65:34949","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36","jwt_validated":"True","user_id":"cf91adaf-9d50-4712-8203-f0b0f6e093d6","user_name":"רותי סרברניק","org_id":"0f5659ba-5ff9-4a5e-9055-5128c007a0ce","jwt_provided":true,"status":200,"event":"Response log","level":"info"}

[meditator3]

thank you, yes i found out about it also(parsing the log key to a json), unfortunately from claude, and not through documentation.

I used :
reserve_original_record true
to maintain my time stamps

but it seems you are able to test your samples quite fast? my rig is that it goes to coralogix for me to see the logs.

is there a tool somewhere that i can test my configurations over a sample like you did, or your setup simply allows to view specific samples very fast?

[daipom]

If you have a Ruby environment, you can test the behavior very easily in your local as follows.

$ git clone [email protected]:fluent/fluentd.git
$ cd fluentd
$ bundle
$ bundle exec fluentd -c test.conf

(Ref: https://docs.fluentd.org/installation/install-from-source)

If you need to add some plugins that are not included by default, you can add the dependency as follows:

$ bundle add fluent-plugin-multi-format-parser

Also, it would not be difficult to install and run the package in your local or a virtual environment.

(DEB: https://docs.fluentd.org/installation/install-fluent-package/install-by-deb-fluent-package)
(RPM: https://docs.fluentd.org/installation/install-fluent-package/install-by-rpm-fluent-package)