Do not get proc_macro from the sysroot in rustc #141595

bjorn3 · 2025-05-26T15:51:46Z

With the stage0 refactor the proc_macro version found in the sysroot will no longer always match the proc_macro version that proc-macros get compiled with by the rustc executable that uses this proc_macro. This will cause problems as soon as the ABI of the bridge gets changed to implement new features or change the way existing features work.

To fix this, this commit changes rustc crates to depend directly on the local version of proc_macro which will also be used in the sysroot that rustc will build.

rustbot · 2025-05-26T15:51:50Z

r? @compiler-errors

rustbot has assigned @compiler-errors.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-05-26T15:51:52Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

bjorn3 · 2025-05-26T15:51:54Z

cc #119899 (comment)

bjorn3 · 2025-05-26T15:53:26Z

src/bootstrap/src/core/build_steps/dist.rs

@@ -776,7 +776,8 @@ impl Step for RustcDev {
        copy_src_dirs(
            builder,
            &builder.src,
-            &["compiler"],
+            // The compiler has a path dependency on proc_macro, so make sure to include it.
+            &["compiler", "library/proc_macro"],


I've tested that cargo metadata and cargo check still work in the built rustc-dev tarball. It may be possible to do the same trick for rustc-literal-escaper to pull it back into this repo.

The rustup component was not the only problem; the other problem are out-of-tree consumers -- RA has to be able to import some of the rustc crates, and IIRC the literal-escaper occurred in the dependency tree.

It'd be good to involve them here as well in case rustc_builtin_macros is in their dependency tree -- Cc @Veykril

It is not so that should be fine

I am also iffy about the fact that the compiler tree relies on this somewhat magical hack, but it looks fairly simple/maintainable and works perfectly fine. It seems like no one in the project is bothered by it (yet), so I would say let's move forward and revisit it if someone raises an issue.

RalfJung · 2025-05-26T15:59:02Z

compiler/rustc_builtin_macros/Cargo.toml

@@ -8,6 +8,9 @@ doctest = false

 [dependencies]
 # tidy-alphabetical-start
+# We must use the proc_macro version that we will compile proc-macros against,
+# not the one from our own sysroot.
+proc_macro = { path = "../../library/proc_macro", default-features = false }


Will this work in the sources as they are distributed by rustup? IIRC last time we tried to share a file between library and compiler, a bunch of stuff broke.

Issue from last time this happened: #138647.

That was the exact same thing, a path dependency from compiler to library -- so it'll almost certainly fail in the exact same way this time again.

EDIT: I didn't look closely, the rustc-dev dist component was addressed by #141595 (comment)

jieyouxu · 2025-05-26T16:16:31Z

r? @onur-ozkan

onur-ozkan · 2025-05-27T07:40:18Z

@rustbot author

bjorn3 · 2025-05-27T10:31:55Z

I had to switch from using a path dependency for proc_macro to creating a separate package which includes the same source as the proc_macro package.

@rustbot ready

RalfJung · 2025-05-27T11:32:55Z

compiler/rustc_proc_macro/Cargo.toml

+edition = "2024"
+
+[lib]
+path = "../../library/proc_macro/src/lib.rs"


This will break for the same reason, no? That path just doesn't exist when rustc-dev is distributed via rustup.

I explicitly added it to the rustc-dev component. See #141595 (comment).

Hm... if you do that then you can go back to making it a regular path dependency, you don't need this separate crate, do you?

That didn't work. Building two crates both named proc_macro (once as part of stdlib and once as part of rustc) causes ambiguity for extern crate proc_macro; when the rustc libraries are part of the sysroot. Because of this I'm forced to build the rustc copy with a different crate name, which requires a separate Cargo.toml.

With the stage0 refactor the proc_macro version found in the sysroot will no longer always match the proc_macro version that proc-macros get compiled with by the rustc executable that uses this proc_macro. This will cause problems as soon as the ABI of the bridge gets changed to implement new features or change the way existing features work. To fix this, this commit changes rustc crates to depend directly on the local version of proc_macro which will also be used in the sysroot that rustc will build.

onur-ozkan · 2025-05-28T04:36:30Z

@bors r+ rollup=never (to make it easier to revert if any issues come up)

bors · 2025-05-28T04:36:32Z

📌 Commit 026baa1 has been approved by onur-ozkan

It is now in the queue for this repository.

onur-ozkan · 2025-05-29T11:30:58Z

This is blocking #119899, so @bors p=1

bors · 2025-05-29T12:07:57Z

⌛ Testing commit 026baa1 with merge 13718eb...

bors · 2025-05-29T15:19:24Z

☀️ Test successful - checks-actions
Approved by: onur-ozkan
Pushing 13718eb to master...

github-actions · 2025-05-29T15:22:16Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8afd710 (parent) -> 13718eb (this PR)

Test differences

No test diffs found

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 13718eb788622ef8c998650451174570230d2971 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

dist-apple-various: 7291.2s -> 6303.7s (-13.5%)
aarch64-apple: 4784.4s -> 5398.9s (12.8%)
x86_64-apple-1: 7755.6s -> 8418.6s (8.5%)
x86_64-msvc-2: 6681.9s -> 7171.1s (7.3%)
x86_64-msvc-1: 8787.5s -> 8144.4s (-7.3%)
x86_64-gnu-llvm-20-1: 3870.4s -> 4147.2s (7.2%)
dist-x86_64-apple: 8830.8s -> 8277.5s (-6.3%)
dist-powerpc64le-linux-musl: 5370.3s -> 5657.9s (5.4%)
x86_64-gnu-llvm-19-1: 3953.8s -> 4119.1s (4.2%)
dist-i586-gnu-i586-i686-musl: 5099.9s -> 5307.4s (4.1%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2025-05-29T18:43:32Z

Finished benchmarking commit (13718eb): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.4%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (secondary 0.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.2%	[0.6%, 2.7%]	7
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.1%	[-2.6%, -0.6%]	4
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary 0.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.7%	[0.5%, 1.3%]	7
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.1%	[-1.6%, -0.6%]	3
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 777.423s -> 780.508s (0.40%)
Artifact size: 368.47 MiB -> 368.46 MiB (-0.00%)

…r=onur-ozkan Do not get proc_macro from the sysroot in rustc With the stage0 refactor the proc_macro version found in the sysroot will no longer always match the proc_macro version that proc-macros get compiled with by the rustc executable that uses this proc_macro. This will cause problems as soon as the ABI of the bridge gets changed to implement new features or change the way existing features work. To fix this, this commit changes rustc crates to depend directly on the local version of proc_macro which will also be used in the sysroot that rustc will build.

rustbot assigned compiler-errors May 26, 2025

bjorn3 commented May 26, 2025

View reviewed changes

bjorn3 mentioned this pull request May 26, 2025

redesign stage 0 std #119899

Merged

RalfJung reviewed May 26, 2025

View reviewed changes