Summary
The aws_s3 input (SQS-driven mode) currently has only one, hard-coded behaviour when an SQS notification points at an S3 object that cannot be downloaded because it no longer exists (NoSuchKey / 404): it logs a warning, acks the notification (DeleteMessage) and discards it. The notification can therefore never reach an SQS dead-letter queue.
This is the right default for pipelines that enable delete_objects — a redelivered notification for an object this input already processed and deleted is expected and harmless. But it is wrong for pipelines that need a missing object to be treated as a delivery failure.
Use case
We run an S3 → Kafka pipeline with a contractual at-least-once delivery requirement and an SQS redrive policy (DLQ, maxReceiveCount). One acceptance criterion is:
If the S3 object referenced by a notification cannot be found, the notification must be dead-lettered with an ERROR log (not silently dropped), so it can be inspected and replayed.
Today there is no way to express this: a missing object is always silently dropped at WARN level, so the DLQ never sees it and the failure is invisible to alerting on DLQ depth.
Proposal
Add an advanced enum field sqs.on_missing_object with two values:
drop (default, current/historical behaviour): warn, ack, discard.nack: log an error and return the notification to the queue (the existing nack path), so an SQS redrive policy dead-letters it after maxReceiveCount.
To prevent a foot-gun, combining on_missing_object: nack with delete_objects: true is rejected at config-parse time: a redelivered notification for an object this input legitimately deleted would otherwise be dead-lettered.
The default value preserves existing behaviour exactly, so this is fully backward compatible.
I have an implementation ready and will open a PR referencing this issue.
Summary
The
aws_s3input (SQS-driven mode) currently has only one, hard-coded behaviour when an SQS notification points at an S3 object that cannot be downloaded because it no longer exists (NoSuchKey/ 404): it logs a warning, acks the notification (DeleteMessage) and discards it. The notification can therefore never reach an SQS dead-letter queue.This is the right default for pipelines that enable
delete_objects— a redelivered notification for an object this input already processed and deleted is expected and harmless. But it is wrong for pipelines that need a missing object to be treated as a delivery failure.Use case
We run an S3 → Kafka pipeline with a contractual at-least-once delivery requirement and an SQS redrive policy (DLQ,
maxReceiveCount). One acceptance criterion is:Today there is no way to express this: a missing object is always silently dropped at WARN level, so the DLQ never sees it and the failure is invisible to alerting on DLQ depth.
Proposal
Add an advanced enum field
sqs.on_missing_objectwith two values:drop(default, current/historical behaviour): warn, ack, discard.nack: log an error and return the notification to the queue (the existing nack path), so an SQS redrive policy dead-letters it aftermaxReceiveCount.To prevent a foot-gun, combining
on_missing_object: nackwithdelete_objects: trueis rejected at config-parse time: a redelivered notification for an object this input legitimately deleted would otherwise be dead-lettered.The default value preserves existing behaviour exactly, so this is fully backward compatible.
I have an implementation ready and will open a PR referencing this issue.