Fluent-bit stops processing logs after kinesis_firehose discards large records

[Sagar-AWS]

Describe the question/issue

In the current configuration, Fluent-bit is processing logs to Kinesis_firehose and in some situations during heavy load conditions, kinesis_firehose discards the logs because the file size is larger than the max size allowed by Firehose. When the logs are rejected by Firehose, fluent-bit is not processing any other logs further and completely stops sending logs to Kinesis_firehose.

Error from firehose:
[2024/12/06 19:50:28] [ warn] [output:kinesis_firehose:kinesis_firehose.2] [size=1222935] Discarding record which is larger than max size allowed by Firehose, AppEcsLogs-Ops-sb-ph

Configuration

` "logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "null"
}
},

"firelensConfiguration": {
"options": {
"config-file-type": "file",
"config-file-value": "/mnt/fluentbit/fbconf/ecs/app/std/ecs-extra.conf"
},
"type": "fluentbit"
},
"healthCheck": null,
"hostname": null,
"image": "1232456745646.dkr.ecr.us-east-1.amazonaws.com/fluent-bit:v2.31.12.3",
"interactive": null,
"links": null,
"linuxParameters": null,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/app",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
},

        "memory": null,
        "memoryReservation": 512,
        "mountPoints": [
            {
                "containerPath": "/mnt/fluentbit",
                "readOnly": null,
                "sourceVolume": "Fluent-bit"
            }
        ],`

-->

Which AWS for Fluent Bit Versions have you tried?*
fluent-bit:v2.31.12.3

Cluster Details

Yes, log_router side car container has been deployed

Are there any related/similar aws/aws-for-fluent-bit or fluent/fluent-bit GitHub issues?

From past 1 year we are seeing that these images has fluent-bit version 1.9 only, as per the Fluent-bit offcial site they have released 3.0 version few months back. So when can we expect latest version of fluent-bit image, If not how can we build image with latest version of fluentbit.
We are thinking that latest version of fluentbit might solve our issues