-
Notifications
You must be signed in to change notification settings - Fork 13.5k
Add profiler to bootstrap command #143525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! I don't think that we'll need JSON output for now, I think that the main output should be some lightly formatted text that will be human interpretable.
This comment has been minimized.
This comment has been minimized.
2730ce9
to
aeec9c4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gonna try it tomorrow.
This comment has been minimized.
This comment has been minimized.
424e09b
to
dba3b7f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I tried it locally. Good job, it seem to work as expected! I can already see this becoming quite useful when dealing with reports of bootstrap being slow.
I have some suggestions/feedback:
- Streaming Cargo commands (which are the most expensive) ones are not recorded. Could you also add support for profiling these?
- The execution and cache hit timestamps are not very useful. I thought they would be, but in this aggregated output they aren't. Also outputting them as UNIX timestamp is not very useful, and we'd have to depend on e.g.
time
orchrono
to improve that, and that's not worth it. You were right when you said that for the trace itself Chrome will be better, I agree now. Sorry! So let's just remove the timestamps and only remember the cache hit counts and the execution duration(s). - Some aggregated stats at the end of the report would be nice. Like the total number of unique commands (fingerprints) that were recorded, the total number of executions (and the aggregated duration of all executions), the total time of bootstrap outside of command executions (so that we can see how large % of bootstrap time was spent on commands specifically - just take a timestamp at the start of bootstrap and when writing the summary, compute the duration, and subtract the sum of command execution durations), the total number of cache hits, and the estimated amount of time saved due to cache hits (take avg. duration of each cached command and multiply it by the number of cache hits it had, and sum that up).
Taking this a bit further, it would be nice to add I/O operations tracing:
- Add
trace!
calls to I/O operations (there is a lot of methods that do file/dir copy, symlinking, network downloads, etc.), so that it can be visualized in Chrome. Also ideally find places where I/O is done ad-hoc, and port it to the I/O helper functions onBuilder
. - Add these I/O operations to the profile summary, so that we can compare the total durations of I/O and command execution.
But that should be left for a follow-up PR.
@@ -101,4 +101,4 @@ tests/rustdoc-gui/src/**.lock | |||
flake.lock | |||
/default.nix | |||
|
|||
# Before adding new lines, see the comment at the top. | |||
# Before adding new lines, see the comment at the top. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Newline change, please revert :)
@@ -147,6 +148,10 @@ fn main() { | |||
t!(file.write_all(lines.join("\n").as_bytes())); | |||
} | |||
} | |||
|
|||
if env::var("BOOTSTRAP_PROFILE").is_ok_and(|v| v == "1") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We already check this environment variable elsewhere. Could you please introduce a function so that the logic is shared? Thanks.
This PR adds command profiling to the bootstrap command. It tracks the total execution time and records cache hits for each command. It also provides the ability to export execution result to a JSON file. Integrating this with Chrome tracing could further enhance observability.
r? @Kobzol