Add profiler to bootstrap command #143525
Draft
+169
−20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
the
T-bootstrap
Kobzol
reviewed
Jul 6, 2025
This comment has been minimized.
This comment has been minimized.
Shourya742
force-pushed
the
2025-07-06-add-profiler
branch
from
2730ce9 to
aeec9c4
Compare
This comment has been minimized.
This comment has been minimized.
Kobzol
reviewed
Jul 9, 2025
Shourya742
force-pushed
the
2025-07-06-add-profiler
branch
from
424e09b to
dba3b7f
Compare
Kobzol
reviewed
Jul 9, 2025
There was a problem hiding this comment.
Ok I tried it locally. Good job, it seem to work as expected! I can already see this becoming quite useful when dealing with reports of bootstrap being slow.
I have some suggestions/feedback:
- Streaming Cargo commands (which are the most expensive) ones are not recorded. Could you also add support for profiling these?
- The execution and cache hit timestamps are not very useful. I thought they would be, but in this aggregated output they aren't. Also outputting them as UNIX timestamp is not very useful, and we'd have to depend on e.g.
timeorchronoto improve that, and that's not worth it. You were right when you said that for the trace itself Chrome will be better, I agree now. Sorry! So let's just remove the timestamps and only remember the cache hit counts and the execution duration(s). - Some aggregated stats at the end of the report would be nice. Like the total number of unique commands (fingerprints) that were recorded, the total number of executions (and the aggregated duration of all executions), the total time of bootstrap outside of command executions (so that we can see how large % of bootstrap time was spent on commands specifically - just take a timestamp at the start of bootstrap and when writing the summary, compute the duration, and subtract the sum of command execution durations), the total number of cache hits, and the estimated amount of time saved due to cache hits (take avg. duration of each cached command and multiply it by the number of cache hits it had, and sum that up).
Taking this a bit further, it would be nice to add I/O operations tracing:
- Add
trace!calls to I/O operations (there is a lot of methods that do file/dir copy, symlinking, network downloads, etc.), so that it can be visualized in Chrome. Also ideally find places where I/O is done ad-hoc, and port it to the I/O helper functions onBuilder. - Add these I/O operations to the profile summary, so that we can compare the total durations of I/O and command execution.
But that should be left for a follow-up PR.
| @@ -101,4 +101,4 @@ tests/rustdoc-gui/src/**.lock | |||
| flake.lock | |||
| /default.nix | |||
|
|
|||
| # Before adding new lines, see the comment at the top. | |||
| # Before adding new lines, see the comment at the top. | |||
| @@ -147,6 +148,10 @@ fn main() { | |||
| t!(file.write_all(lines.join("\n").as_bytes())); | |||
| } | |||
| } | |||
|
|
|||
| if env::var("BOOTSTRAP_PROFILE").is_ok_and(|v| v == "1") { | |||
There was a problem hiding this comment.
We already check this environment variable elsewhere. Could you please introduce a function so that the logic is shared? Thanks.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds command profiling to the bootstrap command. It tracks the total execution time and records cache hits for each command. It also provides the ability to export execution result to a JSON file. Integrating this with Chrome tracing could further enhance observability.
r? @Kobzol