Skip to content

dd: improve allocation strategy for huge bs#11577

Open
oech3 wants to merge 1 commit intouutils:mainfrom
oech3:dd0malloc
Open

dd: improve allocation strategy for huge bs#11577
oech3 wants to merge 1 commit intouutils:mainfrom
oech3:dd0malloc

Conversation

@oech3
Copy link
Copy Markdown
Contributor

@oech3 oech3 commented Apr 1, 2026

closes #11544

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Apr 1, 2026

Merging this PR will improve performance by ×7.5

⚡ 2 improved benchmarks
✅ 301 untouched benchmarks
⏩ 46 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation dd_copy_64k_blocks 365.8 µs 298.7 µs +22.44%
Simulation dd_copy_1m_blocks 1,226.9 µs 164.1 µs ×7.5

Comparing oech3:dd0malloc (b2e307c) with main (7ab8a17)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@oech3 oech3 force-pushed the dd0malloc branch 2 times, most recently from 749b867 to eb5c742 Compare April 1, 2026 08:01
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

GNU testsuite comparison:

Skip an intermittent issue tests/pr/bounded-memory (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tty/tty-eof (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/dd/no-allocate is now being skipped but was previously passing.
Congrats! The gnu test tests/printf/printf-surprise is now passing!

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/tail/tail-n0f is now being skipped but was previously passing.
Congrats! The gnu test tests/printf/printf-surprise is now passing!

@oech3 oech3 marked this pull request as ready for review April 1, 2026 09:16
@oech3 oech3 force-pushed the dd0malloc branch 2 times, most recently from 6cba7fd to b2e307c Compare April 1, 2026 12:13
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)

@xtqqczze
Copy link
Copy Markdown
Contributor

xtqqczze commented Apr 1, 2026

We need Read::read_buf to be stabilised to avoid unsafe.

// std cannot fix https://github.com/uutils/coreutils/issues/11544 without unsafe yet
// SAFETY: write-only to reserved area of buf and becomes unaccessible at self.read(slice)? soon
let slice = unsafe {
std::slice::from_raw_parts_mut(chunk.as_mut_ptr().cast::<u8>(), chunk.len())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unsafe code is the equivalent of chunk.assume_init_mut(), which causes undefined behavior when the content is not fully initialized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dd takes a really long time to allocate memory with large buffers (tested with 1G)

2 participants