Skip to content

Add profiler to bootstrap command #143525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

Shourya742
Copy link
Contributor

This PR adds command profiling to the bootstrap command. It tracks the total execution time and records cache hits for each command. It also provides the ability to export execution result to a JSON file. Integrating this with Chrome tracing could further enhance observability.

r? @Kobzol

@rustbot rustbot added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Jul 6, 2025
Copy link
Member

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I don't think that we'll need JSON output for now, I think that the main output should be some lightly formatted text that will be human interpretable.

@rust-log-analyzer

This comment has been minimized.

@Shourya742 Shourya742 force-pushed the 2025-07-06-add-profiler branch from 2730ce9 to aeec9c4 Compare July 8, 2025 18:27
Copy link
Member

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna try it tomorrow.

@rust-log-analyzer

This comment has been minimized.

@Shourya742 Shourya742 force-pushed the 2025-07-06-add-profiler branch from 424e09b to dba3b7f Compare July 9, 2025 16:39
@rustbot rustbot added the A-rustc-dev-guide Area: rustc-dev-guide label Jul 9, 2025
Copy link
Member

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I tried it locally. Good job, it seem to work as expected! I can already see this becoming quite useful when dealing with reports of bootstrap being slow.

I have some suggestions/feedback:

  • Streaming Cargo commands (which are the most expensive) ones are not recorded. Could you also add support for profiling these?
  • The execution and cache hit timestamps are not very useful. I thought they would be, but in this aggregated output they aren't. Also outputting them as UNIX timestamp is not very useful, and we'd have to depend on e.g. time or chrono to improve that, and that's not worth it. You were right when you said that for the trace itself Chrome will be better, I agree now. Sorry! So let's just remove the timestamps and only remember the cache hit counts and the execution duration(s).
  • Some aggregated stats at the end of the report would be nice. Like the total number of unique commands (fingerprints) that were recorded, the total number of executions (and the aggregated duration of all executions), the total time of bootstrap outside of command executions (so that we can see how large % of bootstrap time was spent on commands specifically - just take a timestamp at the start of bootstrap and when writing the summary, compute the duration, and subtract the sum of command execution durations), the total number of cache hits, and the estimated amount of time saved due to cache hits (take avg. duration of each cached command and multiply it by the number of cache hits it had, and sum that up).

Taking this a bit further, it would be nice to add I/O operations tracing:

  • Add trace! calls to I/O operations (there is a lot of methods that do file/dir copy, symlinking, network downloads, etc.), so that it can be visualized in Chrome. Also ideally find places where I/O is done ad-hoc, and port it to the I/O helper functions on Builder.
  • Add these I/O operations to the profile summary, so that we can compare the total durations of I/O and command execution.

But that should be left for a follow-up PR.

@@ -101,4 +101,4 @@ tests/rustdoc-gui/src/**.lock
flake.lock
/default.nix

# Before adding new lines, see the comment at the top.
# Before adding new lines, see the comment at the top.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Newline change, please revert :)

@@ -147,6 +148,10 @@ fn main() {
t!(file.write_all(lines.join("\n").as_bytes()));
}
}

if env::var("BOOTSTRAP_PROFILE").is_ok_and(|v| v == "1") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already check this environment variable elsewhere. Could you please introduce a function so that the logic is shared? Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-rustc-dev-guide Area: rustc-dev-guide T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants