Have rustc produce a local log of execution metrics

[estebank]

Proposal

Introduce a mechanism into rustc for tracking useful per-invocation metrics during regular compilation. This mechanism would exclusively record information locally.

This is a follow up to my blog-post "No telemetry in the Rust compiler: metrics without betraying user privacy", where I present my objectives and, more importantly, my non-objectives for this feature.

Data points collected should be in a format that easy to extend, to be determined through experimentation and research of existing analogue systems in other spaces. Any tool that consumes these will be required to deal with format changes itself. Summary presentation and analysis of the collected data will be performed on the raw data by spearate tools and are outside of the scope of this MCP, rustc only needs to dump this information.

The initial set of data that I would be interested in collecting would be equivalent to (but not required to be) the contents of -Zhir-stats, -Zself-profile, -Zllvm-lines, -Zmono-items and cargo build --timings (a priori it seems that this could partly be gathered from -Zdump-dep-graph). A potential source of metrics could be existing info!() level debug statements, but this 1) needs to be explored on how it will operate and 2) existing info statements must be audited for suitability as source of useful metrics. If this option proves to be unworkable, an explicit opt-in metric!(level, key, value) macro would be introduced for targetted collection. Eventually I would also appreciate having enough information to produce a histogram of errors encountered by users (this last metric might need to be restricted explicitly to opt-in collection).

Collection would be restricted to nightly releases and enabled through an environment variable, but with an eye for eventually having it enabled by default with an available opt-out, first in nightly and some time in the future stable, only once we have battle tested the entirety of the feature, with a proper automatic clean up mechanism to avoid unbounded growth, and with these changes requiring new MCPs. We will likely require to provide different levels of metrics collection for users to customize what they desire to track or not. The rationale behind eventually having some level of on-by-default metrics collection is in order to have enough information for post-hoc debugging of transient issues, where re-running rustc with the appropriate command to dump its internal state will cause the error to no longer reproduce.

Situations like "strict incremental fingerprint checking" where we desire to ensure that a certain codepath is not being executed in the wild before changing behavior (like enabling a new feature or removing an old deprecated assumed to be inert one) would benefit from this mechanism, as we could instruct users regularly to grep their metrics files for the existence of specific entries. I could also envision us eventually distributing tools, either through rustup, VSCode plug-in or as part of existing tools like rust-analyzer, that perform local analysis of these metrics to identify cases of interest to answer questions the development team has, prompting users to file tickets to inform us of this.

Be advised that telemetry is an explicit non-goal of this MCP. Data collected must have an explicit documented rationale, and it should behave no different than any kind of local logging preserved to disk. rustc should be responsible of no more than producing this data, in the same way that compiler developers are already able to build with debug level information and use RUSTC_LOG.

This is related to, and a super-set of, the rustc ability to write ICE errors to disk.

Mentors or Reviewers

• @estebank

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[oli-obk]

@rustbot second

[estebank]

@rustbot label -final-comment-period +major-change-accepted