[v25.3.x] Implement cross-segment prefetching for small segments in cloud storage reads#29795
Open
vbotbuildovich wants to merge 4 commits intoredpanda-data:v25.3.xfrom
Open
Conversation
Add `try_get_segment_units()` which attempts to acquire segment units without blocking. This is useful for optional segment hydrations from prefetching or otherwise. (cherry picked from commit e15fbda)
(cherry picked from commit 9dd78fa)
While the existing chunk prefetching works well for larger segments it doesn't help when there is many small segments due to compaction or otherwise. Prior to this commit we'd end up having to read and wait for each small segment from cloud storage which greatly reduced throughput. Now when reading small segments the reader can prefetch additional small segments in parallel(up to `cloud_storage_prefetch_segments_max`) to increase throughput. (cherry picked from commit 5f84706)
When multiple callers (e.g., segment prefetch and reader) concurrently request hydration for the same segment and the index download fails, a race condition could cause some callers to receive an unhandled exception instead of retrying with legacy mode. The issue occurred because: 1. Multiple callers add promises to the shared _wait_list 2. Index download fails, all waiters receive the exception 3. First caller to handle the exception sets _fallback_mode and retries 4. Subsequent callers see _fallback_mode is already set, causing the condition (ex.path == _index_path && !_fallback_mode) to be false 5. These callers rethrow the exception instead of retrying This commit changes the logic of do_hydrate to always retry if the failed download was the index. (cherry picked from commit 9fd4a11)
Collaborator
Author
Retry command for Build#81615please wait until all jobs are finished before running the slash command |
Collaborator
Author
CI test resultstest results on build#81615
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PR #29496