Skip to content

Add ignore_missing_path option to Tail input for optional log file globs #11964

Description

@nee13559

Is your feature request related to a problem? Please describe.

I want to use one shared Fluent Bit Tail input configuration across multiple containerized workloads, but not every workload has the same log file paths.

For example, one workload may write:

/var/log/app/*.log

while another may also write optional component logs under:

/var/log/components/*/*.log

If the shared Tail config includes both paths, workloads that do not create the optional component log directory can emit noisy missing-path or unmatched-glob errors, even though the absence of those files is expected.

Describe the solution you'd like

Add an opt-in Tail input option such as:

ignore_missing_path: true

Example:

pipeline:
  inputs:
    - name: tail
      path: /var/log/app/*.log,/var/log/components/*/*.log
      tag: app_logs
      ignore_missing_path: true

When enabled, Fluent Bit should not emit an error if a configured path or glob does not currently match any files. The Tail input should still refresh normally and begin tailing matching files if they are created later, according to existing behavior such as refresh_interval.

The default should preserve current behavior, for example:

ignore_missing_path: false

Describe alternatives you've considered

I have considered maintaining separate Fluent Bit configs per workload type, but that creates duplication and makes config drift more likely.

I have also considered creating placeholder directories/files so optional globs always match something, but that feels like a workaround and adds artificial files only to satisfy the Tail input.

Another option is generating the Tail config dynamically at container startup, but that adds startup scripting and makes the effective runtime config differ from the checked-in config.

Additional context

This would be useful for containerized and orchestrated environments where a single logging sidecar or shared Fluent Bit image is reused across heterogeneous workloads.

Some log paths are mandatory, while others are optional depending on enabled features, plugins, or workload type. Having an explicit ignore_missing_path option would make optional log paths easier to model without duplicating configs or adding placeholder files.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions