Abort (assertion failure) in nix::daemon::TunnelLogger::enqueueMsg

[the-sun-will-rise-tomorrow]

Describe the bug

I saw this when trying to build a derivation:

error: Nix daemon disconnected unexpectedly (maybe it crashed?)

Looking at the system journal, I see that the daemon exited with SIGABRT. Here is the demangled stack trace from the journal:

#0  0x00007fe506c9916c __pthread_kill_implementation (libc.so.6 + 0x9916c)
#1  0x00007fe506c40e86 raise (libc.so.6 + 0x40e86)
#2  0x00007fe506c2893a abort (libc.so.6 + 0x2893a)
#3  0x00007fe5070adc3d __gnu_cxx::__verbose_terminate_handler() [clone .cold] (libstdc++.so.6 + 0xadc3d)
#4  0x00007fe5070bf20a __cxxabiv1::__terminate(void (*)()) (libstdc++.so.6 + 0xbf20a)
#5  0x00007fe5070ad7e9 std::terminate() (libstdc++.so.6 + 0xad7e9)
#6  0x00007fe5070ad800 __cxa_rethrow.cold (libstdc++.so.6 + 0xad800)
#7  0x00007fe5078a2ab1 nix::daemon::TunnelLogger::enqueueMsg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .cold] (libnixstore.so + 0xa2ab1)
#8  0x00007fe507955e66 nix::daemon::TunnelLogger::log(nix::Verbosity, std::basic_string_view<char, std::char_traits<char> >) (libnixstore.so + 0x155e66)
#9  0x00007fe507d508dc nix::ignoreExceptionExceptInterrupt(nix::Verbosity) (libnixutil.so + 0x608dc)
#10 0x00007fe507d4dfe9 nix::ThreadPool::doWork(bool) [clone .cold] (libnixutil.so + 0x5dfe9)
#11 0x00007fe5070ed0a4 execute_native_thread_routine (libstdc++.so.6 + 0xed0a4)
#12 0x00007fe506c972e3 start_thread (libc.so.6 + 0x972e3)
#13 0x00007fe506d1b2fc __clone3 (libc.so.6 + 0x11b2fc)

Stack trace of thread 2286721:
#0  0x00007fe506c9381e __futex_abstimed_wait_common (libc.so.6 + 0x9381e)
#1  0x00007fe506c96490 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x96490)
#2  0x00007fe507a018fd nix::Pool<nix::RemoteStore::Connection>::get() (libnixstore.so + 0x2018fd)
#3  0x00007fe507a02261 nix::RemoteStore::getConnection() (libnixstore.so + 0x202261)
#4  0x00007fe507a093ad nix::RemoteStore::queryRealisationUncached(nix::DrvOutput const&, nix::Callback<std::shared_ptr<nix::Realisation const> >) (libnixstore.so + 0x2093ad)
#5  0x00007fe507a303fc nix::Store::queryRealisation(nix::DrvOutput const&, nix::Callback<std::shared_ptr<nix::Realisation const> >) (libnixstore.so + 0x2303fc)
#6  0x00007fe507a30a87 nix::Store::queryRealisation(nix::DrvOutput const&) (libnixstore.so + 0x230a87)
#7  0x00007fe5079e1268 std::_Function_handler<void (nix::DerivedPath), nix::Store::queryMissing(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, unsigned long&, unsigned long&)::{lambda(nix::DerivedPath const&)#1}>::_M_invoke(std::_Any_data const&, nix::DerivedPath&&) [clone .lto_priv.0] (libnixstore.so + 0x1e1268)
#8  0x00007fe5079d91ce std::_Function_handler<void (), std::_Bind<std::function<void (nix::DerivedPath)> (nix::DerivedPathBuilt)> >::_M_invoke(std::_Any_data const&) (libnixstore.so + 0x1d91ce)
#9  0x00007fe507dd735b nix::ThreadPool::doWork(bool) (libnixutil.so + 0xe735b)
#10 0x00007fe5070ed0a4 execute_native_thread_routine (libstdc++.so.6 + 0xed0a4)
#11 0x00007fe506c972e3 start_thread (libc.so.6 + 0x972e3)
#12 0x00007fe506d1b2fc __clone3 (libc.so.6 + 0x11b2fc)

Stack trace of thread 2286482:
#0  0x00007fe506c9381e __futex_abstimed_wait_common (libc.so.6 + 0x9381e)
#1  0x00007fe506c98ed3 __pthread_clockjoin_ex (libc.so.6 + 0x98ed3)
#2  0x00007fe5070ed11b std::thread::join() (libstdc++.so.6 + 0xed11b)
#3  0x00007fe507dd3798 nix::ThreadPool::shutdown() (libnixutil.so + 0xe3798)
#4  0x00007fe507d4e171 nix::ThreadPool::process() [clone .cold] (libnixutil.so + 0x5e171)
#5  0x00007fe5079dc351 nix::Store::queryMissing(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, std::set<nix::StorePath, std::less<nix::StorePath>, std::allocator<nix::StorePath> >&, unsigned long&, unsigned long&) (libnixstore.so + 0x1dc351)
#6  0x00007fe507959f6e nix::daemon::performOp(nix::daemon::TunnelLogger*, nix::ref<nix::Store>, nix::TrustedFlag, nix::daemon::RecursiveFlag, nix::WorkerProto::BasicServerConnection&, nix::WorkerProto::Op) (libnixstore.so + 0x159f6e)
#7  0x00007fe50795e9a5 nix::daemon::processConnection(nix::ref<nix::Store>, nix::FdSource&&, nix::FdSink&&, nix::TrustedFlag, nix::daemon::RecursiveFlag) (libnixstore.so + 0x15e9a5)
#8  0x000000000053cf8e n/a (n/a + 0x0)
ELF object binary architecture: AMD x86-64

Steps To Reproduce

Unfortunately I don't have a reproducer right now. It happens for me after I:

1. Enable extra-experimental-features = ca-derivations
2. Try to build an internal package with importing nixpkgs with config.contentAddressedByDefault = true;.

Though, looking at the stack trace, CA-derivations might be unrelated to this crash.

Expected behavior

No crash.

Metadata

nix-env (Nix) 2.26.3

Additional context

I noticed a similar crash was reported in an unrelated comment: #3377 (comment)

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[the-sun-will-rise-tomorrow]

I don't know how reproducible it is for others, but I noticed that this also happens if I run nix --extra-experimental-features 'nix-command flakes' develop in a checkout of this repository (NixOS/nix) with extra-experimental-features = ca-derivations enabled.

[edolstra]

nix::Store::queryRealisation() is related to CA derivations.