Nixification #81
Comments
|
I'm not familiar with Nix, so bear with me if I ask silly questions. What is the point of using Nix instead of Dockerized Gentoo? Can I avoid using Docker if I adopt Nix? |
|
@piegamesde I am not familiar enough with the structure of nixpkgs to comment authoritatively. However, from reading it over, I see that you are patching stdenv and binutils to use mold as the linker, which sounds right. What about those packages that explicitly opt in to using lld such as mold itself? Scroll to the bottom of https://github.com/NixOS/nixpkgs/pull/128889/files#diff-ab5748dc9567516fefba8344056b51ec1866adeace380f46e58a7af3d619ea22R13818. |
Well, mainly because I can :) There probably are some advantages, but I don't know a lot about Gentoo so I can't really compare. Here are some things that are cool about Nix, you judge if Gentoo+Docker can do this too:
Yes. You only need to have Nix installed.
They won't use mold. This holds of all packages that have a custom I'm still trying to build the above derivation. A few packages are already failing. I'll have to disable mold and check that they aren't broken upstream, but otherwise I may have found some bugs? Also, brainstorming some ideas how to move this prototype forward:
|
|
Note so self: the following packages fail with mold ┃ ⚠ valgrind-3.16.1 on ssh://builder1 failed with exit code 2 after ⏲ 00:01:03 For comparison, without mold: ┃ ⚠ python3.8-python-lz4-3.1.3 on ssh://builder1 failed with exit code 137 after ⏲ 00:01:07 The difference should be subject to closer investigation (and I'll open bug reports on nixpkgs for the latter). |
|
valgrind's failure is known -- valgrind assumes some specific section layout which mold doesn't comply. Technically, it's not mold's fault, and I think I need to submit a patch to valgrind to fix it. Other failures are not known. |
|
I did some further research, filtered out false positives. The following packages have interesting failures that are very likely directly caused by mold: kexec-tools, pixman, glibc-locales, mariadb-connector-c. And the following packages fail too, but with less obvious error messages: gfortran, gmp, python3.8-cffi |
|
mariadb-connector-c (version script support is incomplete?): pixman (something off with 'common' symbols: being treated as undefined?): kexectools (no support in mold for -Text=<> or --section-start): glibcLocales (some assumptions in configure script break): gmp (in tests/mpf, make check fails for 'reuse' which throws a floating-point exception!) gfortran: fails because it can't build gmp as a dependency python3.pkgs.cffi (not sure if related to mold): |
|
mariadb-connector-c is failing because mold doesn't support the If pixman is failing due to common symbols in an archive file, it is hard to fix. It looks like BFD linker's common symbol handling is not consistent and highly dependent on the order of files in the command line, and it is hard to mimic the exact behavior. Even gold behaves slightly different from BFD, and so do LLVM lld. It's usually pretty easy to fix an application (you can just compile with As to kexectools, let me implement -Ttext. |
|
The |
|
How can I build gmp with mold on Nix? I just tried building the most recent version of gmp using mold on my Ubuntu, but the tests just pass, so I think I have to built it on Nix. |
|
To build gmp with nix: |
|
@rui314 I've changed the nix file a bit to make it easier to build and test individual packages: let
# Pin some fairly new nixpkgs
sources = builtins.fetchTarball {
name = "nixpkgs-unstable-2021-07-06";
url = "https://github.com/nixos/nixpkgs/archive/291b3ff5af268eb7a656bb11c73f2fe535ea2170.tar.gz";
sha256 = "1z2l7q4cmiaqb99cd8yfisdr1n6xbwcczr9020ss47y2z1cn1x7x";
};
# For bootstrapping
pkgs = import sources {
overlays = [
(pkgs: super: {
# TODO replace with proper packaging once https://github.com/NixOS/nixpkgs/pull/128889 is merged
mold = pkgs.stdenv.mkDerivation {
pname = "mold";
version = "0.9.1";
src = pkgs.fetchgit {
url = "https://github.com/rui314/mold";
rev = "v0.9.1";
sha256 = "sha256-yIkW6OCXhlHZ1jC8/yMAdJbSgY9K40POT2zWv6wYr5E=";
};
nativeBuildInputs = with pkgs; [ clang_12 cmake lld_12 tbb xxHash zlib openssl git ];
dontUseCmakeConfigure = "true";
buildPhase = "make -j $NIX_BUILD_CORES";
installPhase = "mkdir -p $out $out/bin $out/share/man/man1 && PREFIX=$out make install";
};
binutils_mold = pkgs.wrapBintoolsWith {
bintools = pkgs.binutils-unwrapped.overrideAttrs (old: {
postInstall = ''
rm $out/bin/ld.gold
rm $out/bin/ld.bfd
ln -sf ${pkgs.mold}/bin/mold $out/bin/ld.bfd
'';
});
};
stdenv_mold = super.overrideCC super.stdenv (super.wrapCCWith rec {
cc = super.gcc-unwrapped;
bintools = pkgs.binutils_mold;
});
})
];
};
# Actual nixpkgs with patched linker in all packages
pkgsMold = import sources {
overlays = [
(self: super: {
stdenv = pkgs.stdenv_mold;
mold = pkgs.mold;
})
];
};
in {
inherit pkgs;
inherit pkgsMold;
# Use -A testPackages to build all of these
testPackages = with pkgsMold; linkFarmFromDrvs "packages-with-mold" [
# mold
binutils
stdenv
#opencv
jack2
chromium
];
}You can now do I'm planning to add a |
Previously, a GOT relocation (e.g. R_X86_64_REX_GOTPCRELX) and a R_X86_64_64 relocation referring the same imported symbols were resolved to different addresses. Here is why: - When we saw a R_X86_64_64 relocation against an imported symbol, we created a PLT and resolve the relocation there. - GOT relocation is resolved to a GOT entry, which has a true address of an imported function at runtime, which is different from PLT entries that redirect calls to the real function. With this patch, we no longer create a PLT entry for R_X86_64_64. Instead, we emit a dynamic relocation so that it is always resolved to a real function address. Fixes GNU MP's `make check` failure, which was reported at #81
|
Thanks guys for teaching me how to use Nix! @piegamesde That config file works for me, and I succeeded to reproduce the test failure of GNU MP. It was due to a subtle bug in mold, and I submitted a fix. |
|
Hello! 👋 Thank you for working on this project! Rather coincidentally, I have recently decided to write a package for I have never used Nix before, but I know people say it’s similar to Guix. (Indeed, Guix is inspired by Nix, at least from what the developers say.) I came across this issue, and decided to try to do something similar for Guix, to see how well it would work, mostly for fun. It actually ended up going fairly well, honestly! Using my package, I was able to write the following file: (define-module (mold-ld))
(use-modules
(guix packages)
(guix build-system trivial))
(include "packages/compilers/mold.scm")
(define-public mold-ld
(package
(name "mold-ld")
(version (package-version mold))
(synopsis (package-synopsis mold))
(home-page (package-home-page mold))
(description (package-description mold))
(license (package-license mold))
(inputs `(("mold" ,mold)))
(source #f)
(build-system trivial-build-system)
(arguments
'(#:builder
(let ((out (assoc-ref %outputs "out")) (mold (assoc-ref %build-inputs "mold")))
(begin
(mkdir out)
(mkdir (string-append out "/bin"))
(symlink (string-append mold "/bin/mold") (string-append out "/bin/ld"))))))))The file is fairly staright forward, it simply creates a new package that links its Using this file, I was able to run a build of some projects using mold with the Unfortunately, it failed while building binutils. GNU hello worked, though, and I was able to verify it was indeed using mold with binutils build outputHere’s the last couple lines of the output of the failed binutils build anyway, in case it’s relevant or useful. For some reason, it seems to be a problem regarding |
|
@zamfofex Awesome! This is honestly the first time for me to hear about Guix, but I'm happy to hear that mold could be used with that package system. I used to write lots of Scheme code at one point of my life, so it is interesting in that sense too. I'm not sure why binutils is failing to be built with mold. I'll try to reproduce the issue locally. |
|
@zamfofex I installed guix using the installer script as explained in https://guix.gnu.org/manual/en/html_node/Binary-Installation.html. I then executed the following commands as you instructed: This is what I got: What am I missing? |
|
I added a let
# Pin some fairly new nixpkgs
sources = builtins.fetchTarball {
name = "nixpkgs-unstable-2021-07-06";
url = "https://github.com/nixos/nixpkgs/archive/291b3ff5af268eb7a656bb11c73f2fe535ea2170.tar.gz";
sha256 = "1z2l7q4cmiaqb99cd8yfisdr1n6xbwcczr9020ss47y2z1cn1x7x";
};
# For bootstrapping
pkgs = import sources {
overlays = [
(pkgs: super: {
# TODO replace with proper packaging once https://github.com/NixOS/nixpkgs/pull/128889 is merged
mold = pkgs.stdenv.mkDerivation {
pname = "mold";
version = "0.9.1";
src = pkgs.fetchgit {
url = "https://github.com/rui314/mold";
rev = "v0.9.1";
sha256 = "sha256-yIkW6OCXhlHZ1jC8/yMAdJbSgY9K40POT2zWv6wYr5E=";
};
nativeBuildInputs = with pkgs; [ clang_12 cmake lld_12 tbb xxHash zlib openssl git ];
dontUseCmakeConfigure = "true";
buildPhase = "make -j $NIX_BUILD_CORES";
installPhase = "mkdir -p $out $out/bin $out/share/man/man1 && PREFIX=$out make install";
};
binutils_mold = pkgs.wrapBintoolsWith {
bintools = pkgs.binutils-unwrapped.overrideAttrs (old: {
postInstall = ''
rm $out/bin/ld.gold
rm $out/bin/ld.bfd
ln -sf ${pkgs.mold}/bin/mold $out/bin/ld.bfd
'';
});
};
stdenv_mold = super.overrideCC super.stdenv (super.wrapCCWith rec {
cc = super.gcc-unwrapped;
bintools = pkgs.binutils_mold;
});
})
];
};
# Actual nixpkgs with patched linker in all packages
pkgsMold = import sources {
overlays = [
(self: super: {
stdenv = pkgs.stdenv_mold;
mold = pkgs.mold;
})
];
};
# nixpkgs version where all packages are linked with mold individually, but not their transitive dependencies
pkgsMoldLight = import sources {
overlays = [
# This is really hacky in quite a few ways, but it works
(self: super:
let
overridePackage = name: let
package = pkgs.${name};
in
if package ? override && (builtins.hasAttr "stdenv" package.override.__functionArgs) then
(package.override { stdenv = pkgs.stdenv_mold; })
else
package;
moldPackages = pkgs.lib.trivial.pipe pkgs [
pkgs.lib.attrNames
(map (name: pkgs.lib.nameValuePair name (overridePackage name)))
builtins.listToAttrs
];
in
moldPackages
)
];
};
in {
inherit pkgs pkgsMold pkgsMoldLight;
# Packages we want to build
testPackages = with pkgsMold; linkFarmFromDrvs "packages-with-mold" [
# mold
binutils
stdenv
#opencv
jack2
chromium
];
# Packages we already know that work in order to catch regressions
ciPackages = with pkgsMold; linkFarmFromDrvs "packages-with-mold" [
# TODO
];
}Sadly, it doesn't really catch the things I'd hoped it would: For example, |
|
@piegamesde Building not only a specified package but its all dependencies with mold isn't bad to find as many bugs as possible. The only catch is, if there's a package that can't be built with mold, all other packages that depend on it will also transitively fail. But we can workaround it by writing a simple shim that wraps mold. The shim invokes ld.bfd instead of mold only when it is used to build a known-broken package. |
@rui314: I think you might need to run I’m going to try to execute it on a fresh Guix VM to see what’s needed, if there are additional steps, I can share them here too. |
|
A little update about this: I was able to build binutils without its dependencies. I’m going to try to figure out what the issue could be with texinfo, and maybe also try other projects like Chromium or Jack. It should also be possible to use the default Though maybe this should move to a different issue, instead of resuing this one for Guix too? It’s up to you, really. information about building binutilsThe binutils build (without dependencies) went fairly smoothly. All I had to do was add the following to the end of file I shared: (define moldify (options->transformation '((with-input . "ld-wrapper=mold-ld"))))
(moldify binutils)But I also had to add the whole fileFrom there, I built the modified binutils package using I was able to use the built |
|
@zamfofex Yeah, I'm starting to get confused which comments are referring to Nix and which ones are based on Guix, so a separate issue might be a good idea. |
|
Since Nix package is already available, I'm closing this issue. |
As soon as I read that mold uses Gentoo for testing because it's source-based, I instantly thought about Nix being a perfect fit for it. Achieving this however turned out to be a lot harder than I initially thought, because the bootstrapping code is rather complex compared to the simple
overrides one usually does.Nevertheless, here's a file that builds any selected packages (list at the bottom) and all their transitive dependencies using
moldas links (if they usedbfdbefore). (Edit: updated file here)Let me know what you think.
CC @nitsky
The text was updated successfully, but these errors were encountered: