output stops under heavy input

[sorran]

Bug Report

Describe the bug

output collapses under heavy input. When a tail → multiline → parser × N pipeline is driven past its processing capacity, proc_records stops incrementing entirely.

^when running with a load ramp going from 2 -> 4 -> 0mb/s inputs all output stops after 3.5mb/s.

To Reproduce

• Steps to reproduce the problem:
  Run a performance benchmark that saturates the input of a tail log.

I'll send configs / logs / repos if someone wants it. Callouts are:

tail input:
a lot of reg exp field extraction and a multiline parser
threaded: true
storage.type: filesystem
storage.pause_on_chunks_overlimit: off #don't block the input drop at the output

output:
workers: 1
storage.total_limit_size: 128MB #no back pressure active, we just drop what the output can't absorb
retry_limit: false

service:
storage.max_chunks_up: 8

Expected behavior
Some level of continued output.

Screenshots
A ramp test of fluent-bit performance under load

Your Environment

fluent-bit 5.1.2 (RPM fluent-bit-5.1.2-1.x86_64)

• AWS EC2 c7a.large (2 vCPU / 4 GB RAM, AMD EPYC Genoa)
• Rocky Linux 9 / x86_64
• systemd-managed service

Additional context
We use fluent-bit to forward logs to an analysis platform - when there's a production outage we get high logs / errors and fluent-bit stops sending which makes fluent-bit useless for the scenario its aimed at.