Split sysroot into runtime libs and compile-time bits

[lf-]

I have built this rust executable:

fn main() {
    println!("hello world");
}

like so:

$ rustc -Cprefer-dynamic -Crpath=yes foo.rs

It has this runpath:

~/lix » readelf -d foo

Dynamic section at offset 0xfd50 contains 30 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstd-998872f6596bd1c9.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../nix/store/y4wsjfgpavqs0y371frl5vgpc50fyr3w-rustc-1.82.0/lib/rustlib/aarch64-unknown-linux-gnu/lib:/home/jade/lix/outputs/out/lib:/nix/store/y4wsjfgpavqs0y371frl5vgpc50fyr3w-rustc-1.82.0/lib/rustlib/aarch64-unknown-linux-gnu/lib:/nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36/lib:/nix/store/b4mrzszd04v235l3dvfj0xkjwy7
rg9dj-gcc-13.3.0-lib/lib:/nix/store/31lhx9gjb4d4rsy3l9vxc63482z6vb8v-gcc-13.3.0-libgcc/lib]
 0x000000000000000c (INIT)               0x998
 0x000000000000000d (FINI)               0xd98
 0x0000000000000019 (INIT_ARRAY)         0x1fd00
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x1fd08
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x2d0
 0x0000000000000005 (STRTAB)             0x428
 0x0000000000000006 (SYMTAB)             0x2f0
 0x000000000000000a (STRSZ)              757 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x1ff70
 0x0000000000000002 (PLTRELSZ)           192 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x8d8
 0x0000000000000007 (RELA)               0x788
 0x0000000000000008 (RELASZ)             336 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001e (FLAGS)              ORIGIN BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW ORIGIN PIE
 0x000000006ffffffe (VERNEED)            0x738
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x71e
 0x000000006ffffff9 (RELACOUNT)          9
 0x0000000000000000 (NULL)               0x0

Of particular note here is the /nix/store/y4wsjfgpavqs0y371frl5vgpc50fyr3w-rustc-1.82.0/lib/rustlib/aarch64-unknown-linux-gnu/lib: this is a problem! rustc and its transitive dependencies are over 1GB in size, which means that executables distributed with dynamic libstd cannot be distributed if they link to the toolchain's copy of libstd.

However, it's pretty reasonable to actually use the toolchain's libraries at runtime, they just have to be able to be configured to go somewhere else.

See, from the above, these libs from gcc (libgcc_s.so, libasah.so, etc etc etc) which basically everything is linked to:

$ nix path-info -rSsh /nix/store/b4mrzszd04v235l3dvfj0xkjwy7rg9dj-gcc-13.3.0-lib/lib /nix/store/31lhx9gjb4d4rsy3l9vxc63482z6vb8v-gcc-13.3.0-libgcc/lib
/nix/store/8rywcj8s7gx9iy4hwipfz7nb87s9rib9-libunistring-1.2       1.8M    1.8M
/nix/store/04aq2w58qlqjvwamcljh1hahz744hlzd-libidn2-2.3.7        359.0K    2.2M
/nix/store/31lhx9gjb4d4rsy3l9vxc63482z6vb8v-gcc-13.3.0-libgcc    147.4K  147.4K
/nix/store/9zk5kpadlnzhz4aq319hykb6anf71g7x-xgcc-13.3.0-libgcc   147.4K  147.4K
/nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36         39.8M   42.1M
/nix/store/b4mrzszd04v235l3dvfj0xkjwy7rg9dj-gcc-13.3.0-lib         8.6M   50.9M

It needs to be possible to ship rustc's runtime libs like libstd-foo.so without shipping an entire rest of a sysroot and compiler along with them. We can hack around this at a packaging level by doing some truly horrible fixups to every rust build after the fact, but it's not really reasonable to do to every single package: the compiler needs to be able to look somewhere different for its libstd dynamic libs so it just works.

The way that this would probably look like is that rustc has a separate install directory of runtime libs which is set up at configure time in a similar way to the main install directory and then that's the paths it would give to the linker when linking.

Discussion on Zulip: https://rust-lang.zulipchat.com/#narrow/channel/182449-t-compiler.2Fhelp/topic/libstd.2C.20rpath.2C.20dylibs.20and.20packaging

Relevant (I believe!) code:

rust/compiler/rustc_session/src/filesearch.rs

Lines 48 to 59 in 84ce2e1

	pub fn make_target_lib_path(sysroot: &Path, target_triple: &str) -> PathBuf {
	let rustlib_path = rustc_target::relative_target_rustlib_path(sysroot, target_triple);
	sysroot.join(rustlib_path).join("lib")
	}
	/// Returns a path to the target's `bin` folder within its `rustlib` path in the sysroot. This is
	/// where binaries are usually installed, e.g. the self-contained linkers, lld-wrappers, LLVM tools,
	/// etc.
	pub fn make_target_bin_path(sysroot: &Path, target_triple: &str) -> PathBuf {
	let rustlib_path = rustc_target::relative_target_rustlib_path(sysroot, target_triple);
	sysroot.join(rustlib_path).join("bin")
	}

Meta

rustc --version --verbose:

rustc 1.82.0 (f6e511eec 2024-10-15) (built from a source tarball)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: aarch64-unknown-linux-gnu
release: 1.82.0
LLVM version: 18.1.8

[Noratrieb]

Creating a separate derivation with just the runtime libraries should be pretty simple. Then instead of using -Crpath you can set a custom rpath (-Clink-arg=-Wl,rpath=$whatever) to this other derivation. I think that should solve your problem?

[lf-]

Oh if it were only so easy. Currently I'm making a fake sysroot made of symlinks so i can replace libstd-foo.{so,dylib}, giving it to rustc, and in spite of my best efforts I'm still going to have to patchelf the final product because on Linux, unlike on macOS, libraries don't give their own reference name in an executable header and there's two separate things adding wrong rpaths for me. ld-wrapper puts the path to my fake sysroot (or the real one if not specified) into the rpath because it's given as -L by rustc. Meson adds rpath based on rustc --print whatsit, but not on macOS (which is a large part of why i have to mess with libstd's executable headers to change the link id).

Furthermore, an rpath (or this hack) based solution requires intervention on every single separate package to do this patchelf hacking (or a post build hook).

At the scale of a distro this hacking would be barely maintainable. It probably could be done by uniformly adding a patchelf post build hook to deal with meson and ld-wrapper's problems and hacking up the rustc derivation to split out dynamic libraries from the 400mb of static ones as symlinks in the sysroot and so on.

It would be infinitely cleaner if rustc could have a separate directory for dynamic libraries used at runtime so that you can distribute rust software without including an entire 1gb of rustc.