Skip to content

aws_sqs: enforce 256 KB message and batch size limits#4048

Merged
twmb merged 1 commit intomainfrom
756
Mar 16, 2026
Merged

aws_sqs: enforce 256 KB message and batch size limits#4048
twmb merged 1 commit intomainfrom
756

Conversation

@twmb
Copy link
Copy Markdown
Contributor

@twmb twmb commented Mar 2, 2026

Reject individual messages exceeding the SQS 256 KB limit before sending, and split batches by cumulative byte size so that each SendMessageBatch call stays within the 256 KB total payload limit.

Closes #756

@claude
Copy link
Copy Markdown

claude bot commented Mar 2, 2026

Claude finished @twmb's task in 9m 6s —— View job


Review: aws_sqs: enforce 256 KB message and batch size limits

  • Gather context and read project guidelines
  • Validate commit policy
  • Review code changes
  • Post summary

Commits
LGTM

Review
Solid implementation. The sqsEntrySize function correctly accounts for what SQS counts toward the 256 KB limit (body + attribute names + attribute string values + data types), and the comment explicitly documents why only StringValue is counted. The batch-splitting loop's n > 0 guard is safe because oversized individual messages are already rejected before the loop. The byte-size splitting composes correctly with the count-based splitting in writeChunk. Tests cover the key cases: entry size calculation, oversized single message rejection, and byte-aware batch splitting.

LGTM

@josephwoodward
Copy link
Copy Markdown
Contributor

Do we still need to be concerned about @Jeffail's comment in the original issue?

@mmatczuk
Copy link
Copy Markdown
Contributor

mmatczuk commented Mar 4, 2026

@twmb please rebase

Reject individual messages exceeding the SQS 256 KB limit before
sending, and split batches by cumulative byte size so that each
SendMessageBatch call stays within the 256 KB total payload limit.

Closes #756

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@twmb
Copy link
Copy Markdown
Contributor Author

twmb commented Mar 16, 2026

@josephwoodward

"""
Regarding Jeffail's concern: he worried about "odd side effects if we change the default batching config to something non-zeroed" but said "Even if we can't do this for whatever reason we can enforce it at the edge so at the very least we won't receive errors."

This PR does exactly the safe approach he suggested — it does not change any default batching config. It enforces limits at the edge in WriteBatch:

  1. Rejects individual messages > 256 KB before they reach SQS
  2. Splits batches by cumulative byte size so each SendMessageBatch call stays within the 256 KB total payload limit

The batch policy (NewBatchPolicyField) is completely untouched — users still configure their own batching as before. The byte-size splitting is purely a safety net at send time. So Jeffail's concern doesn't apply here.
"""

@claude
Copy link
Copy Markdown

claude bot commented Mar 16, 2026

Commits
LGTM

Review
Adds SQS 256 KB message and batch size enforcement with byte-aware chunking before writeChunk. Individual oversized messages are rejected early, and multi-message batches are split to stay within the 256 KB total payload limit. Good test coverage for size calculation, rejection, and splitting.

LGTM

@twmb twmb merged commit 7f85ad8 into main Mar 16, 2026
7 checks passed
@twmb twmb deleted the 756 branch March 16, 2026 22:45
twmb added a commit that referenced this pull request Mar 30, 2026
twmb added a commit that referenced this pull request Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AWS SQS output batching should be aware of the 256KB message size limits

3 participants