dwarffs

dwarffs is a FUSE-based filesystem that fetches DWARF debug info files automatically from cache.nixos.org, based on the build ID embedded in ELF executables and libraries.

Usage

NixOS

To enable dwarffs, add it to your NixOS configuration flake like this:

{
  inputs.dwarffs.url = "github:edolstra/dwarffs";

  outputs = { self, nixpkgs, dwarffs }: {
    nixosConfigurations.my-machine = nixpkgs.lib.nixosSystem {
      modules =
        [ dwarffs.nixosModules.dwarffs
          # ... other configuration ...
        ];
    };
  };
}

You may also need an unstable version of Nix, which can be obtained from the nix flake:

{
  inputs.dwarffs.url = "github:edolstra/dwarffs";
  inputs.nix.url = "github:NixOS/nix";

  outputs = { self, nixpkgs, dwarffs, nix }: {
    nixosConfigurations.my-machine = nixpkgs.lib.nixosSystem {
      modules =
        [ dwarffs.nixosModules.dwarffs
          { nixpkgs.overlays = [ nix.overlays.default ]; }
          # ... other configuration ...
        ];
    };
  };
}

This creates an automount unit on /run/dwarffs. It also sets the environment variable NIX_DEBUG_INFO_DIRS to /run/dwarffs. The gdb and elfutils packages in Nixpkgs have been patched to use that environment variable, allowing them to use dwarffs automatically.

For example, to get line number information from eu-stack for a running process:

$ eu-stack -p 28080 -s
PID 28080 - process
TID 28080:
#0  0x00007f1ee343f99d read
    ../sysdeps/unix/syscall-template.S:84
#1  0x00007f1ee4147864 nix::FdSource::readUnbuffered(unsigned char*, unsigned long)
    /nix/store/50jw5m7lda3rylirxyly9diy55lh149z-glibc-2.25-49-dev/include/bits/unistd.h:44
#2  0x00007f1ee4146d66 nix::BufferedSource::read(unsigned char*, unsigned long)
    src/libutil/serialise.cc:94
...

Similarly, to get line number information from a core dump:

$ eu-stack --core ./core  -s
PID 22154 - core
TID 22154:
#0  0x00007fd98db9bfcd __read
    ../sysdeps/unix/syscall-template.S:84
#1  0x000000000041cf13 linenoise
    /nix/store/50jw5m7lda3rylirxyly9diy55lh149z-glibc-2.25-49-dev/include/bits/unistd.h:44
#2  0x0000000000471f98 nix::NixRepl::getLine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    src/nix/repl.cc:187
...

gdb should also obtain debug info automatically:

$ gdb -p 22178
(gdb) bt
#0  0x00007fe96642436f in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fffa120e3f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fffa120e300, cond=0x7fffa120e3d0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fffa120e3d0, mutex=0x7fffa120e300) at pthread_cond_wait.c:655
#3  0x00007fe966c0dd1c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from target:/nix/store/lc9cdddv2xv45ighz8znsanjfgkcdgbx-gcc-5.4.0-lib/lib/libstdc++.so.6
#4  0x00007fe96712a044 in nix::Sync<nix::ThreadPool::State, std::mutex>::Lock::wait (cv=..., this=0x7fffa120e0c0) at src/libutil/sync.hh:56
#5  nix::ThreadPool::process (this=this@entry=0x7fffa120e2f0) at src/libutil/thread-pool.cc:60
#6  0x00007fe96747c306 in nix::Store::queryMissing (this=this@entry=0xfc57b0, targets=..., willBuild_=..., willSubstitute_=..., unknown_=..., downloadSize_=@0x7fffa120e500: 0, narSize_=@0x7fffa120e508: 0) at src/libstore/misc.cc:237
#7  0x0000000000415672 in opServe (opFlags=..., opArgs=...) at src/nix-store/nix-store.cc:836
#8  0x000000000040b632 in <lambda()>::operator()(void) const (__closure=<optimized out>) at src/nix-store/nix-store.cc:1070
#9  0x00007fe967720642 in std::function<void ()>::operator()() const (this=0x7fffa120f6f0) at /nix/store/2m6v6lnmnf9521ixd5v0x4qrmjkn30ws-gcc-5.4.0/include/c++/5.4.0/functional:2267
#10 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName=..., fun=...) at src/libmain/shared.cc:271
#11 0x00000000004079b7 in main (argc=3, argv=0x7fffa120f818) at src/nix-store/nix-store.cc:1071

(gdb) select-frame 5
(gdb) info locals
state = {s = 0x7fffa120e300, lk = {_M_device = 0x7fffa120e300, _M_owns = true}}
(gdb) p state.s.data.active
$2 = 0

Operation

ELF binaries have a unique build ID embedded in them, allowing separate debug info files to be looked up. For example, here is the build ID of a particular build of libnixexpr.so:

$ readelf -a /nix/store/c0srp6xfqyrjrmqhd1pgw6hcrhcghg87-nix-1.12pre5619_346aeee1/lib/libnixexpr.so | grep 'Build ID'
    Build ID: bde350fa1f1bbde3649bfce3ae143b87683d8bf9

Given this build ID, gdb and elfutils will look in /run/dwarffs/.build-id/bd/e350fa1f1bbde3649bfce3ae143b87683d8bf9.debug to obtain the debug info for this binary. dwarffs will then look in https://cache.nixos.org/debuginfo/<build-ID>:

$ curl https://cache.nixos.org/debuginfo/bde350fa1f1bbde3649bfce3ae143b87683d8bf9
{"archive":"../nar/0jz52hawk8wida5b2544i484mws9ddkl8jjpjmw6rzrjf83jr5zw.nar.xz","member":"lib/debug/.build-id/bd/e350fa1f1bbde3649bfce3ae143b87683d8bf9.debug"}

That is, the debug info is contained in the NAR file https://cache.nixos.org/nar/0jz52hawk8wida5b2544i484mws9ddkl8jjpjmw6rzrjf83jr5zw.nar.xz (along with the debug info for other binaries from the same package). dwarffs will unpack all these debug info files to a cache (/var/cache/dwarffs/) and serve them from there.