Skip to content

dd: optimize O_DIRECT buffer alignment to reduce syscall overhead#9104

Open
naoNao89 wants to merge 1 commit into
uutils:mainfrom
naoNao89:feature/o-direct-buffer-alignment
Open

dd: optimize O_DIRECT buffer alignment to reduce syscall overhead#9104
naoNao89 wants to merge 1 commit into
uutils:mainfrom
naoNao89:feature/o-direct-buffer-alignment

Conversation

@naoNao89

@naoNao89 naoNao89 commented Nov 1, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes #6078

page-aligned buffers + smarter O_DIRECT handling. Theory says 5x fewer syscalls. 🗿

Checklist:

  • dd's first benchmark suite to verify
  • merge bench to this branch for check report pref

@naoNao89 naoNao89 force-pushed the feature/o-direct-buffer-alignment branch 3 times, most recently from 9f131bd to fecebff Compare November 1, 2025 06:05
@codspeed-hq

codspeed-hq Bot commented Nov 1, 2025 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 319 untouched benchmarks
⏩ 46 skipped benchmarks1

Comparing naoNao89:feature/o-direct-buffer-alignment (43a187e) with main (55549fa)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@naoNao89 naoNao89 force-pushed the feature/o-direct-buffer-alignment branch from fecebff to 2560240 Compare November 1, 2025 06:13
@github-actions

github-actions Bot commented Nov 1, 2025

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@sylvestre

sylvestre commented Nov 1, 2025

Copy link
Copy Markdown
Contributor

@naoNao89 i do value your contribution and terrific work. But please, also focus on finishing the existing work instead of starting new work

#9088 => red CI
#9094 => red CI
#9026 => WIP
#8949 => still draft
etc :)

@naoNao89

naoNao89 commented Nov 1, 2025

Copy link
Copy Markdown
Contributor Author

I need more dopamine when stuck on a bug, so new PRs might be good :))

@cakebaker cakebaker changed the title optimize O_DIRECT buffer alignment to reduce syscall overhead dd: optimize O_DIRECT buffer alignment to reduce syscall overhead Nov 1, 2025
@github-actions

github-actions Bot commented Nov 2, 2025

Copy link
Copy Markdown

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

@naoNao89 naoNao89 mentioned this pull request Nov 4, 2025
Merged
@naoNao89 naoNao89 closed this by deleting the head repository Nov 6, 2025
@naoNao89 naoNao89 reopened this Nov 7, 2025
@github-actions

github-actions Bot commented Nov 7, 2025

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

cre4ture
cre4ture reviewed Nov 17, 2025
View reviewed changes

@cre4ture cre4ture left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

Comment thread src/uu/dd/src/dd.rs Outdated
Comment thread src/uu/dd/src/dd.rs Outdated
/// This function allocates a `Vec<u8>` with proper alignment to support O_DIRECT
/// without triggering EINVAL errors.
#[cfg(any(target_os = "linux", target_os = "android"))]
fn allocate_aligned_buffer(size: usize) -> Vec<u8> {

@cre4ture cre4ture Nov 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sylvestre is this something we could move to a more central location? Or is this the only place where we need aligned memory allocations?

@sylvestre sylvestre Dec 31, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which programs will use it ? thanks

naoNao89 added a commit to naoNao89/coreutils that referenced this pull request Dec 31, 2025
@naoNao89
Fix PR uutils#9104 review comments and clippy errors
0d984d0 
- Remove dead code: non-Linux stub for handle_o_direct_write
  The stub was unreachable since write_with_o_direct_handling already
  has a non-Linux stub that doesn't call this helper function.

- Fix clippy::ptr-as-ptr lint error
  Replace unsafe `as *mut u8` cast with safer `.cast::<u8>()` method
  in allocate_aligned_buffer function.

Addresses review comments and CI/CD failures in PR uutils#9104.
@github-actions

github-actions Bot commented Dec 31, 2025

Copy link
Copy Markdown

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

@oech3

oech3 commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

conflicting

naoNao89 added a commit to naoNao89/coreutils that referenced this pull request May 26, 2026
@naoNao89
Fix PR uutils#9104 review comments and clippy errors
9fec9d9 
- Remove dead code: non-Linux stub for handle_o_direct_write
  The stub was unreachable since write_with_o_direct_handling already
  has a non-Linux stub that doesn't call this helper function.

- Fix clippy::ptr-as-ptr lint error
  Replace unsafe `as *mut u8` cast with safer `.cast::<u8>()` method
  in allocate_aligned_buffer function.

Addresses review comments and CI/CD failures in PR uutils#9104.
@naoNao89 naoNao89 force-pushed the feature/o-direct-buffer-alignment branch 4 times, most recently from 09798a1 to 61dc228 Compare May 26, 2026 11:13
@naoNao89 naoNao89 marked this pull request as draft May 26, 2026 11:16
@naoNao89 naoNao89 marked this pull request as ready for review May 26, 2026 11:16
@naoNao89 naoNao89 mentioned this pull request May 26, 2026
Closed
@sylvestre sylvestre force-pushed the feature/o-direct-buffer-alignment branch from 61dc228 to a9d8373 Compare May 26, 2026 11:27
@naoNao89 naoNao89 force-pushed the feature/o-direct-buffer-alignment branch from a9d8373 to de996e2 Compare May 26, 2026 15:28
@github-actions

github-actions Bot commented May 26, 2026 •
edited
Loading

Copy link
Copy Markdown

GNU testsuite comparison:

GNU test failed: tests/seq/seq-io-errors. tests/seq/seq-io-errors is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/date/date-locale-hour (fails in this run but passes in the 'main' branch)
Note: The gnu test tests/basenc/bounded-memory is now being skipped but was previously passing.
Note: The gnu test tests/cut/cut-huge-range is now being skipped but was previously passing.
Note: The gnu test tests/printf/printf-surprise is now being skipped but was previously passing.

@oech3

oech3 commented May 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

@naoNao89 Did you see #12143 which has better CodSpeed result?

@oech3

oech3 commented May 28, 2026

Copy link
Copy Markdown
Contributor

By the way, CodSpeed does excludes time for syscalls. So you cannot use CodSpeed at here.

@naoNao89 naoNao89 force-pushed the feature/o-direct-buffer-alignment branch 2 times, most recently from 5352334 to 93ecdcd Compare May 28, 2026 12:54
@naoNao89 naoNao89 force-pushed the feature/o-direct-buffer-alignment branch from 93ecdcd to 43a187e Compare May 28, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sylvestre sylvestre sylvestre left review comments
+1 more reviewer
@cre4ture cre4ture cre4ture left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

dd fails on gnu test tests/dd/direct.sh - dynamic removal of O_DIRECT missing

4 participants

@naoNao89 @sylvestre @oech3 @cre4ture