Skip to content

fix: reject cross-segment ExclusiveStartKey in parallel Scan - #203

Open
LeeroyHannigan wants to merge 1 commit into
mainfrom
fix/parallel-scan-exclusive-start-key
Open

fix: reject cross-segment ExclusiveStartKey in parallel Scan#203
LeeroyHannigan wants to merge 1 commit into
mainfrom
fix/parallel-scan-exclusive-start-key

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

What

Reject a cross-segment ExclusiveStartKey in a parallel Scan.

A parallel Scan (TotalSegments/Segment) accepted an ExclusiveStartKey belonging to a different segment, silently returning a truncated or empty page. This validates the start key against the same segment-assignment function used
for the scan and rejects a mismatch with a ValidationException naming the correct Segment, so a key returned as a LastEvaluatedKey for one segment is only valid when the same segment is re-scanned.

Why

DynamoDB validates that an ExclusiveStartKey supplied to a parallel scan belongs to the requested segment, and rejects it otherwise; the engine instead accepted any key and combined it with the segment predicate, silently yielding a truncated or empty page rather than an error. This is a correctness/parity gap for parallel-scan pagination. The behavior was verified against the real DynamoDB service.

Closes #

Testing done

  • Scan integration test: a key that belongs to a segment is accepted as that segment's ExclusiveStartKey, and rejected with a ValidationException when supplied for a different segment.
  • Verified verbatim against the real DynamoDB service (us-east-1): a same-segment ExclusiveStartKey paginates within the segment; a cross-segment key returns ValidationException: The provided starting key is invalid: Invalid ExclusiveStartKey. Please use ExclusiveStartKey with correct Segment. TotalSegments: 4 Segment: N.
  • cargo test -p extenddb-storage-postgres unit pass; scan integration test passes; cargo clippy --workspace --all-targets and cargo fmt --all -- --check clean.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a -- no change to wire protocol, Storage trait, auth model,
on-disk format, or public CLI surface. Request-validation correctness only.

Breaking changes

None. A request that was previously mis-accepted (a cross-segment ExclusiveStartKey) now returns the DynamoDB-correct ValidationException; valid same-segment pagination is unchanged.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

jcshepherd
jcshepherd previously approved these changes Jul 7, 2026

@jcshepherd jcshepherd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little bizarre that we're executing a SQL query to do a mathematical calculation. I get why but in a perfect world I wonder if we'd be taking a dependency on a Postgres internal hash function for assigning segments.

@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queue Jul 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Jul 8, 2026
A parallel Scan (TotalSegments/Segment) accepted an ExclusiveStartKey
that belongs to a different segment, silently returning a truncated or
empty page. Validate the start key against the same segment-assignment
function used for the scan and reject a mismatch with a ValidationException
naming the correct Segment, so a key returned as a LastEvaluatedKey for one
segment is only valid when the same segment is re-scanned. Verified against
DynamoDB.

Adds a scan integration test.

Signed-off-by: Lee Hannigan <lhnng@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants