fix: reject cross-segment ExclusiveStartKey in parallel Scan - #203
Open
LeeroyHannigan wants to merge 1 commit into
Open
fix: reject cross-segment ExclusiveStartKey in parallel Scan#203LeeroyHannigan wants to merge 1 commit into
LeeroyHannigan wants to merge 1 commit into
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn and
yesyayen
as code owners
July 7, 2026 16:08
jcshepherd
previously approved these changes
Jul 7, 2026
jcshepherd
left a comment
Collaborator
There was a problem hiding this comment.
It's a little bizarre that we're executing a SQL query to do a mathematical calculation. I get why but in a perfect world I wonder if we'd be taking a dependency on a Postgres internal hash function for assigning segments.
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Jul 8, 2026
A parallel Scan (TotalSegments/Segment) accepted an ExclusiveStartKey that belongs to a different segment, silently returning a truncated or empty page. Validate the start key against the same segment-assignment function used for the scan and reject a mismatch with a ValidationException naming the correct Segment, so a key returned as a LastEvaluatedKey for one segment is only valid when the same segment is re-scanned. Verified against DynamoDB. Adds a scan integration test. Signed-off-by: Lee Hannigan <lhnng@amazon.com>
LeeroyHannigan
force-pushed
the
fix/parallel-scan-exclusive-start-key
branch
from
July 21, 2026 11:40
0064bb1 to
43200aa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reject a cross-segment
ExclusiveStartKeyin a parallelScan.A parallel
Scan(TotalSegments/Segment) accepted anExclusiveStartKeybelonging to a different segment, silently returning a truncated or empty page. This validates the start key against the same segment-assignment function usedfor the scan and rejects a mismatch with a
ValidationExceptionnaming the correctSegment, so a key returned as aLastEvaluatedKeyfor one segment is only valid when the same segment is re-scanned.Why
DynamoDB validates that an
ExclusiveStartKeysupplied to a parallel scan belongs to the requested segment, and rejects it otherwise; the engine instead accepted any key and combined it with the segment predicate, silently yielding a truncated or empty page rather than an error. This is a correctness/parity gap for parallel-scan pagination. The behavior was verified against the real DynamoDB service.Closes #
Testing done
ExclusiveStartKey, and rejected with aValidationExceptionwhen supplied for a different segment.ExclusiveStartKeypaginates within the segment; a cross-segment key returnsValidationException: The provided starting key is invalid: Invalid ExclusiveStartKey. Please use ExclusiveStartKey with correct Segment. TotalSegments: 4 Segment: N.cargo test -p extenddb-storage-postgresunit pass; scan integration test passes;cargo clippy --workspace --all-targetsandcargo fmt --all -- --checkclean.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a -- no change to wire protocol,
Storagetrait, auth model,on-disk format, or public CLI surface. Request-validation correctness only.
Breaking changes
None. A request that was previously mis-accepted (a cross-segment
ExclusiveStartKey) now returns the DynamoDB-correctValidationException; valid same-segment pagination is unchanged.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.