Description
Describe the problem you are trying to solve
Compilation pipelining can currently compile dependent crates as soon as the crate metadata is written in most cases. This improves cpu utilization for from-scratch builds. It is however not possible to apply pipelining on crates that need to be linked like executables and dylibs as all object files need to be available to link something.
Describe the solution you'd like
Using the -Zno-link
option of rustc it is possible to separate the compilation and linking step. This makes it possible to pipeline the compilation and only wait for all crates to be fully compiled once we want to link the final executable. This option was introduced in rust-lang/rust#68487 and fixed in rust-lang/rust#80187 (now with a test to prevent regressions).
Notes
cc rust-lang/rust#64191
To implement this rustc will first need to be invoked with -Zno-link
as argument. This will produce all object files and a .rlink
file. Once all dependencies are compiled you need to invoke rustc a second time with -Zlink-only
as argument and the .rlink
file instead of the source file.
Activity
weihanglo commentedon Jan 11, 2021
I am interested in this issue and plan to implement it. Just some questions need to make sure I didn't get it wrong.
Below is my understanding of a possible flow. @bjorn3, could you help me check its correctness?
The firstrustc
invocation is for all dependencies with-Zno-link
.And the next one is also for all dependencies with-Zlink-only
.Then compile and link theTarget
(crate root).If there is any dependency attempting to produce a staticlib/dylib/proc-macro which requires upstream objects, the compilation pipeline must invokerustc
with-Zlink-only
for all its upstream dependencies before proceeding.Edit: Wrong. Was too young at that time though. See #9019 (comment).
@alexcrichton, since you are the person who implemented pipeline compilation and opened rust-lang/rust#64191, it would be very helpful if you can provide some advice and tips for prototyping. Thanks 😄
bjorn3 commentedon Jan 11, 2021
The first step is to compile all crates, with the crates that need to be linked getting an extra
-Zno-link
argument. Non-linked crates must not get-Zno-link
as argument. This step can be fully pipelined, so once the metadata file of a dependency is written, all dependent crates can immediately be compiled without having to wait on codegen. This pipelining is unchanged from master except for also pipelining crates that need to be linked. The second step is after all crates are done compiling completely, only running rustc on the crates that need to be linked with-Zlink-only
and with the.rmeta
file from-Zno-link
instead of the source file. If I am right, all crate types except forrlib
need to be linked.alexcrichton commentedon Jan 11, 2021
This is likely to be a somewhat nontrivial change in Cargo. My best guess for the way to implement this would be to introduce a
Unit
for-Zno-link
and a second unit for-Zlink-only
for compilations that today are a singleUnit
which invoke the linker (e.g. everything other than an rlib basically). Cargo's unit of execution of a process is aUnit
, and today there's aUnit
-per-rustc-invocation basically. AUnit
also tracks things like execution of a build script.Pipelining with rlib dependencies is different since we don't invoke rustc twice. Instead we invoke rustc once and only once we've received the message that the rmeta is available do we start executing other units. There's special-casing that handles this (see
JobQueue::enqueue
andArtifact::Metadata
), but if we're executing rustc twice then that probably doesn't need to happen.I suspect functions like
requires_upstream_objects
andonly_requires_rmeta
may not need to be refactored, but you'll want to eye them carefully in case new support here impacts them.weihanglo commentedon May 14, 2025
Take another look of ii a few years after, I now have some doubts upon the benefit to compilation pipeline. For the real world crates, most of the dependencies are rlibs, not any other linkable artifacts. The most common linkable artifacts are bins from build scripts, which can't really be pipelined because their dependents still need to wait for build script executions. Given this, the benefit of splitting compile/link phase is more about configurability and cachability than pipelining.
That said, for crates depending on dylibs it does help, but it is a bit rare.
bjorn3 commentedon May 18, 2025
rustc_driver would benefit significantly from this. It is currently split into rustc_driver and rustc_driver_impl to effectively emulate
-Zno-link
/-Zlink-only
.weihanglo commentedon May 23, 2025
Other future possibilities along with this change:
RUSTFLAGS
to something like RUSTLINKFLAGS, so that normal rlib caches have a higher chance of being hit. And that would help a lot in per-user cachebjorn3 commentedon May 23, 2025
Not as is, the
-Zlink-only
args need to match the-Zno-link
invocation. And if you want to rerun-Zlink-only
, you have to rerun-Zno-link
to as the former removes all temporary files that the latter produced.weihanglo commentedon May 23, 2025
Is it a limitation of the current implementation, or the future possibility just doesn't make sense?
bjorn3 commentedon May 23, 2025
I think we could add it in the future, but it would require explicitly specifying which cli arguments are "link" arguments that can be set for
-Zlink-only
as well as moving a bunch of computations from the-Zno-link
invocation to the-Zlink-only
invocation. It should be doable, but it is not trivial. That said, I do think it would be nice to have in the future.weihanglo commentedon May 28, 2025
Status update: currently blocked on rust-lang/rust#141706.