Skip to content

Store path metadata has the potential to get corrupted #13384

@winterqt

Description

@winterqt
Member

Describe the bug

In rare cases, the metadata of a store path (e.g. what drv an out path was derived from) has the potential to be incorrect.

Steps To Reproduce

  1. Check out Nixpkgs 0f64ecd19d54377664bc296469fb623c7c7470f8
  2. Create bug.nix:
let
  pkgs = import ./. { config.allowUnfree = true; };
in
{
  normal = pkgs._1password-gui;
  bad = pkgs._1password-gui.override { polkitPolicyOwners = ["winter"];};
}
$ nix-instantiate bug.nix
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/z26zmwrfb6shlrsv61ia5slxwdj3cw7q-1password-8.10.78.drv
/nix/store/vssmdq8nw78iwwdim6l87jr1qgbkpvm2-1password-8.10.78.drv

$ nix-build bug.nix --no-out-link
/nix/store/0q4kz73yzd1w0xqph2hvc2g4z62y20kk-1password-8.10.78
/nix/store/xxx2xvymaqr448k0f7d94m38d3khmvb0-1password-8.10.78

$ nix derivation show /nix/store/0q4kz73yzd1w0xqph2hvc2g4z62y20kk-1password-8.10.78 | head
{
  "/nix/store/ny6wd0bswldsn5l151xzmgw1mj30x4c7-1password-8.10.78.drv": {
    "args": [
      "-e",
      "/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh",
      "/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh"
    ],
    "builder": "/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin/bash",
    "env": {
      "__structuredAttrs": "",
$ nix derivation show /nix/store/xxx2xvymaqr448k0f7d94m38d3khmvb0-1password-8.10.78 | head
{
  "/nix/store/vssmdq8nw78iwwdim6l87jr1qgbkpvm2-1password-8.10.78.drv": {
    "args": [
      "-e",
      "/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh",
      "/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh"
    ],
    "builder": "/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin/bash",
    "env": {
      "__structuredAttrs": "",

Expected behavior

For the output of nix derivation show /nix/store/0q4kz73yzd1w0xqph2hvc2g4z62y20kk-1password-8.10.78 to be /nix/store/z26zmwrfb6shlrsv61ia5slxwdj3cw7q-1password-8.10.78.drv.

Metadata

nix-env (Nix) 2.28.3

(on x86_64-linux)

Additional context

  1. If I make a small change to the expression (e.g. changing the value of polkitPolicyOwners), this issue does not occur, which implies this would likely not happen I was to nix-store --delete the affected path(s).
  2. These expressions were initially instantiated with Nix 2.24.10.

Checklist

  • checked open bug issues and pull requests for possible duplicates

Add 👍 to issues you find important.

Activity

winterqt

winterqt commented on Jun 21, 2025

@winterqt
MemberAuthor

I forgot to note that this also has very perplexing behavior for dependent drvs; can send a concrete example next week.

roberth

roberth commented on Jun 25, 2025

@roberth
Member

Triaged in Nix meeting today:

This command relies on a mapping from derivation outputs to derivations, which is not 1:1.
Multiple derivations can result in the same output path.
A trivial example is that of two fixed output derivations that declare the same output hash, but this 1:many relation propagates to other kinds of derivations as well. This is beneficial because it allows early cutoff for this CA output (FODs are class of CA).
So what probably happened here is that the output was substituted from an earlier version of Nixpkgs, before something about some FOD was changed, and the store db's deriver field points to that derivation instead of your mostly-equivalent (probably "newer") derivation. (mostly-equivalent, because if the FOD needs to be fetched, it is different)

  1. If I make a small change to the expression (e.g. changing the value of polkitPolicyOwners), this issue does not occur,

This is also consistent with the FOD change theory just described.

Most likely, this is an expected behavior, although we agree it is surprising. Storing more provenance information will help a bit with diagnosing this on your own, as you may be able to see that the "wrong" derivation was from a different version of Nixpkgs.

We'll close this issue under that assumption. We'll happily reopen it if you have a reproducer that is not explained by the above.

nixos-discourse

nixos-discourse commented on Jun 26, 2025

@nixos-discourse

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-06-25-nix-team-meeting-minutes-232/66102/1

winterqt

winterqt commented on Jun 26, 2025

@winterqt
MemberAuthor

Thanks for the detailed (and fast) response! While I think I understand what you’re saying, I’m not sure this applies here given the drv in question isn’t a FOD.

The override in question is this, and definitely was not substituted beforehand.

So what probably happened here is that the output was substituted from an earlier version of Nixpkgs, before something about some FOD was changed, and the store db's deriver field points to that derivation instead of your mostly-equivalent (probably "newer") derivation. (mostly-equivalent, because if the FOD needs to be fetched, it is different)

Right, but the issue is that said old Nixpkgs drv is being pulled into my closures, when it shouldn’t be given that version isn’t even present on my system anymore.

This is also consistent with the FOD change theory just described.

To clarify: I meant that I rebuilt the same expr with a different override value.

roberth

roberth commented on Jun 27, 2025

@roberth
Member

The distinction between derivation hashes and output hashes is very important here.
Output hashing treats FODs as a black box, only including the name and hash (or output path? I'd have to look that up, but that amount of detail is not needed).
Any discrepancy like this between derivation hashes and output hashes gives rise to the 1:many relation between outputs and their derivations.

not sure this applies here given the drv in question isn’t a FOD.

The only necessary conditions for the hypothesis are

  1. A different derivation has existed for a given output path
  2. An output of such a different derivation is available in your store

(1) can happen when any FOD anywhere in the closure has a different derivation, but still produces the same output hash.
(2) can happen when you've built the different derivation before in your own store, or when a substituter has built it for you

Right, but the issue is that said old Nixpkgs drv is being pulled into my closures

It does not really become part of your closure, unless you use a very broad definition of closure, such as used in GC when (iirc) keep-derivations is enabled.

To clarify: I meant that I rebuilt the same expr with a different override value.

Understood. We're talking about two distinct "orderings"

  • Before/after the change in an FOD somewhere in the closure
  • Before/after the override is applied

Discussion of hashing behavior relates to the former.
The latter is only relevant in that the override probably created a presumably "fresh" derivation where aforementioned necessary conditions are not met.

winterqt

winterqt commented on Jun 27, 2025

@winterqt
MemberAuthor

Let me clarify myself a bit with an example, as I think I confused even myself:

$ cat aa.nix
let
  pkgs = import ./. { config.allowUnfree = true; };
in
{
  normal = pkgs._1password-gui;
  bad = pkgs._1password-gui.override { polkitPolicyOwners = ["winter"];};
}

$ nix-instantiate aa.nix
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/z26zmwrfb6shlrsv61ia5slxwdj3cw7q-1password-8.10.78.drv
/nix/store/vssmdq8nw78iwwdim6l87jr1qgbkpvm2-1password-8.10.78.drv

$ cat aa.nix
let
  pkgs = import ./. { config.allowUnfree = true; };
in
{
  normal = pkgs._1password-gui;
  bad = pkgs._1password-gui.override { polkitPolicyOwners = ["winte"];};
}

$ nix-instantiate aa.nix
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/7vlgplg9dayibnjwj6gfvwp7qxz9pkia-1password-8.10.78.drv
/nix/store/vssmdq8nw78iwwdim6l87jr1qgbkpvm2-1password-8.10.78.drv

I'm not sure how changing bad would affect good, am I just misunderstanding?

winterqt

winterqt commented on Jul 2, 2025

@winterqt
MemberAuthor

Ah, looking at this again, I think the ordering of the output was screwing me up. Let me come back to this and see if I can articulate it better without an incorrect understanding of what’s actually happening. Sorry!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @roberth@nixos-discourse@winterqt

        Issue actions

          Store path metadata has the potential to get corrupted · Issue #13384 · NixOS/nix