filelog receiver reads old logs even when start_at is set to end

[obs-gh-enricogiorio]

Hi @djaglowski I am still an issue where filelog starts reading from beginning of files despite a fix that was mentioned in this other issue.
That issue mentions that when the max concurrent files is reached, the receiver enters "batch" mode, and starts reading from the beginning of the files, regardless of what "start_at" is set to.
I made sure start_at is set to end and I have checked the receiver's logs, nothing mentions that any batching happened.
Also given the amount of log files I have in the system, I can pretty much ensure that the default limit of 1024 concurrent files has ever been reached.

For instance, on 16/10/24 13:22:24.363 CEST, the receiver read a log line with timestamp 14/05/24 14:40:41.042 CEST, which is >5 months old.

From this screenshot you can see:

BUNDLE_TIMESTAMP is when the log line was parsed (technically it's when it reached our system, but it's likely just a few seconds earlier)
timestamp is the log's timestamp
delta is the discrepancy between the two. The logs you see are 4 months old
The purple bits above are when the data falls over time. You can see that this did not happen constantly but only twice, with many instances of very old logs being read all together

This is how I configured the receiver:

  filelog:
    exclude:  '[]'
    include:  '["/var/log/pods/*/*/*.log", "/var/log/kube-apiserver-audit.log"]'
    include_file_name: false
    include_file_path: true
    exclude_older_than: 24h
    operators:
    - id: container-parser
      max_log_size: 102400
      type: container
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval:  30s
      max_elapsed_time: 5m
    start_at: "end"
    storage: file_storage

[djaglowski]

For instance, on 16/10/24 13:22:24.363 CEST, the receiver read a log line with timestamp 14/05/24 14:40:41.042 CEST, which is >5 months old.

Just to be clear, neither start_at or exclude_older_than settings are intended to filter logs based on their timestamps. These settings are both concerned with aspects of the files themselves.

The start_at setting only has an effect when the receiver first starts running. From then on, any file which appears to be a new file will be ingested from the beginning of the file. Given that, is there any chance that these files containing old logs were moved, or could new logs have been written to the files?

[VihasMakwana]

Perhaps related to #32727. Not sure though

[VihasMakwana]

Maybe, what's happening is:

• You recently updated a file which was 4 months old.
• By this time, the collector was already running, and setting start_at wouldn’t impact the later set of matched files
• The receiver considers that the 4 months old file new (because it was updated) and read everything from beginning.