Binaries created using rules_nixpkgs have rpath for libraries from nix store

[malt3]

Describe the bug

I have a go binary that is built in Baze using rules_nixpkgs Go and CC toolchains. The Go binary uses cgo to link cryptsetup from nixpkgs.
The resulting binary uses an RPATH to link the top level libcryptsetup.so.12, while all transitive deps are linked with their absolute store path.

To Reproduce

• Wrap a C library from nix in a cc_library
• Link it with any native code (could be a cc_binary or go_binary or something else)
• Run ldd on the result

Expected behavior

The resulting binary should have only absolute paths in the ldd output, since a nix store is required at runtime anyways.

Actual behavior

The top level library that I link against (libcryptsetup.so.12) uses a location relative to the binary (requiring runfiles to be shipped with the binary itself):

rpath looks like this:

patchelf --print-rpath bootstrapper 
$ORIGIN/../../../../_solib_k8/_U@cryptsetup_Ux86_U64-linux_S_S_Ccryptsetup___Ulib:$ORIGIN/bootstrapper.runfiles/constellation/_solib_k8/_U@cryptsetup_Ux86_U64-linux_S_S_Ccryptsetup___Ulib:/nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib:/nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib

ldd bazel-bin/bootstrapper/cmd/bootstrapper/bootstrapper_/bootstrapper
        linux-vdso.so.1 (0x00007ffff7fc8000)
        libm.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libm.so.6 (0x00007ffff7ee2000)
        libcryptsetup.so.12 => /home/malte/github.com/edgelesssys/constellation/bazel-bin/bootstrapper/cmd/bootstrapper/bootstrapper_/../../../../_solib_k8/_U@cryptsetup_Ulib_S_S_Ccryptsetup___Ulib/libcryptsetup.so.12 (0x00007ffff7e60000)
        libpthread.so.0 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libpthread.so.0 (0x00007ffff7e5b000)
        libc.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libc.so.6 (0x00007ffff7c73000)
        /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/ld-linux-x86-64.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fca000)
        libuuid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libuuid.so.1 (0x00007ffff7c67000)
        libdevmapper.so.1.02 => /nix/store/4iajj5xvjys2bis1g8kvrj43c2khw9aj-lvm2-2.03.22-lib/lib/libdevmapper.so.1.02 (0x00007ffff7c0c000)
        libssl.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libssl.so.3 (0x00007ffff7b5f000)
        libcrypto.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libcrypto.so.3 (0x00007ffff7600000)
        libargon2.so.1 => /nix/store/x3q760hm3rqi72vpqcv4552wyx7x3z9i-libargon2-20190702/lib/libargon2.so.1 (0x00007ffff7b53000)
        librt.so.1 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/librt.so.1 (0x00007ffff7b4e000)
        libdl.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libdl.so.2 (0x00007ffff7b49000)
        libjson-c.so.5 => /nix/store/ndc2l8gyajdfpagccpiymacnmpnmn50i-json-c-0.16/lib/libjson-c.so.5 (0x00007ffff7b38000)
        libblkid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libblkid.so.1 (0x00007ffff7adc000)
        libgcc_s.so.1 => /nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib/libgcc_s.so.1 (0x00007ffff7ab9000)
        libudev.so.1 => /nix/store/62sdl1d0qd0qr9llfvx6w4yd2pr1v0bh-systemd-minimal-254.3/lib/libudev.so.1 (0x00007ffff75bf000)
        libcap.so.2 => /nix/store/c3nvhic2lcxbbyln5v4x9bnl9jdcw8m5-libcap-2.69-lib/lib/libcap.so.2 (0x00007ffff7aad000)

Environment

• NixOS 23.05 (amd64)
• Latest HEAD: e632be8

Additional context

I'm currently thinking about using `patchelf as a postprocessing step to normalize library paths (and always use absolute paths into the nix store). Not sure if that's a good idea but it is the best I came up with so far.

As an example, I tried "repairing" the binary, by setting a working rpath:

patchelf --set-rpath /nix/store/36r63mp6pi1mymcqzwb0rnc7z2x249i6-cryptsetup-2.6.1/lib:/nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib:/nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib bootstrapper-fixed 
ldd bootstrapper-fixed 
        linux-vdso.so.1 (0x00007ffff7fc8000)
        libm.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libm.so.6 (0x00007ffff7ee2000)
        libcryptsetup.so.12 => /nix/store/36r63mp6pi1mymcqzwb0rnc7z2x249i6-cryptsetup-2.6.1/lib/libcryptsetup.so.12 (0x00007ffff7e60000)
        libpthread.so.0 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libpthread.so.0 (0x00007ffff7e5b000)
        libc.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libc.so.6 (0x00007ffff7c73000)
        /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/ld-linux-x86-64.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fca000)
        libuuid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libuuid.so.1 (0x00007ffff7c67000)
        libdevmapper.so.1.02 => /nix/store/4iajj5xvjys2bis1g8kvrj43c2khw9aj-lvm2-2.03.22-lib/lib/libdevmapper.so.1.02 (0x00007ffff7c0c000)
        libssl.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libssl.so.3 (0x00007ffff7b5f000)
        libcrypto.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libcrypto.so.3 (0x00007ffff7600000)
        libargon2.so.1 => /nix/store/x3q760hm3rqi72vpqcv4552wyx7x3z9i-libargon2-20190702/lib/libargon2.so.1 (0x00007ffff7b55000)
        librt.so.1 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/librt.so.1 (0x00007ffff7b4e000)
        libdl.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libdl.so.2 (0x00007ffff7b49000)
        libjson-c.so.5 => /nix/store/ndc2l8gyajdfpagccpiymacnmpnmn50i-json-c-0.16/lib/libjson-c.so.5 (0x00007ffff7b38000)
        libblkid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libblkid.so.1 (0x00007ffff7adc000)
        libgcc_s.so.1 => /nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib/libgcc_s.so.1 (0x00007ffff7abb000)
        libudev.so.1 => /nix/store/62sdl1d0qd0qr9llfvx6w4yd2pr1v0bh-systemd-minimal-254.3/lib/libudev.so.1 (0x00007ffff75bf000)
        libcap.so.2 => /nix/store/c3nvhic2lcxbbyln5v4x9bnl9jdcw8m5-libcap-2.69-lib/lib/libcap.so.2 (0x00007ffff7aad000)

[malt3]

Small update: I wrote a rule for automating the patchelf step. If you are interested, I could upstream this work to rules_nixpkgs.

[benradf]

Hi @malt3, thanks for opening this issue.

The story for deploying binaries created with rules_nixpkgs to other machines isn't great at the moment. As you say, they often depend on libraries in the Nix store. We have an open issue to document solutions to this.

Your patchelf rule is an interesting approach, but it seems like it still requires the Nix store on the target machine to have all the necessary paths? My concern is there is no way to know what these store paths are. In the past we've discussed adding an aspect to nixpkgs_package to collect the transitive closure of the store paths and write it to an output file, e.g. required-nix-paths.txt. This could then be used either to bundle all those paths as a single deployable artifact, or as an indication of what packages need to be installed on the target machine as prerequisite.

[malt3]

Your patchelf rule is an interesting approach, but it seems like it still requires the Nix store on the target machine to have all the necessary paths?

Yes. That's correct. I'm creating a tar of the transitive closure of required nix store paths that I can ship with the binary.