Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

treewide: substitute pname for strings #387769

Merged
merged 1 commit into from
Mar 11, 2025

Conversation

pbsds
Copy link
Member

@pbsds pbsds commented Mar 7, 2025

Inspired by #387725 (comment), script is based on #336172 using what i learned in #386865, this time limited to pkgs/by-name, part of #346453

Should be zero rebuilds.

All candidates were made using:

#!/usr/bin/env nix-shell
#!nix-shell -I nixpkgs=. --pure -i bash -p ripgrep sd jq git git-wait lix util-linux

export NIXPKGS_ALLOW_UNFREE=1
export NIXPKGS_ALLOW_INSECURE=1
export NIXPKGS_ALLOW_BROKEN=1

git-wait restore .

test -s packages.json || ( set -x;
  time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages.json
)

list_attrpath_fname_col() {
    jq <packages.json 'to_entries[] | select(.value.meta.position==null|not) | "\(.key)\t\(.value.meta.position)"' -r |
        sed -e "s#\t$(realpath .)/#\t#" |
        sed -e 's#:\([0-9]*\)$#\t\1#' |
        grep . |
        grep -iv haskell |
        grep -iv /top-level/ |
        grep -iv chicken |
        grep pkgs/by-name/ |
        grep -iv build |
        grep -E '/(package|default)\.nix'
}

FLOCKDIR="$(mktemp -d)"
N_WORKERS=4
while read attrpath fname col; do
    grep -qE 'repo *= *("\$\{pname\}"|pname);' "$fname" || continue

    echo | (
        # mutex on fname
        flock --nonblock 200 || {
            >&2 echo "failed to aquire lock for $fname"
            exit 1
        }

        echo "$attrpath"
        data="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)" || exit
        test -n "$data" || exit
        pname="$(jq <<<"$data" .pname -r)"
        test -n "$pname" || exit

        (set -x
            sd -F '${pname}'  "$pname"         "$fname"
            sd -F ' = pname;' " = \"$pname\";" "$fname"
        )

        data2="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)"
        if [[ "$data" = "$data2" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
            exit
        fi

        (set -x
            sd -F ' rec {' ' {' "$fname"
        )

        data3="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json 2>/dev/nul)"

        if [[ "$data" = "$data3" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
        fi

    ) 200>"$FLOCKDIR"/"$(sha256sum - <<<"$fname" | cut -d' ' -f1)".lock &

    while [[ $(jobs -p | wc -l) -ge $N_WORKERS ]]; do
        wait -n < <(jobs -p) || true
    done

done < <(list_attrpath_fname_col)

wait

git restore .

time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages2.json

diff packages{,2}.json is empty, indicating that no package nor src derivation has changed, nor have i introduced any eval failures.
I checked and cherry-picked the changes using GIT_DIFF_OPTS='-u15' git -c interactive.singleKey=true add --patch while watching the planet nix stream

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 6.topic: cinnamon Desktop environment 6.topic: xen-project The Xen Project hypervisor labels Mar 7, 2025
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 7, 2025
@nix-owners nix-owners bot requested a review from mkg20001 March 7, 2025 00:45
@pbsds pbsds force-pushed the migrate-pname-1741296491 branch from 53d0161 to d1c0914 Compare March 7, 2025 00:46
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 7, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Mar 7, 2025
Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good. It's quite hard to review in the web interface, but I read through the diff on my machine with git and I trust you Peder (@pbsds). The lack of rebuilds means it's good too.

@pbsds pbsds force-pushed the migrate-pname-1741296491 branch from d1c0914 to f946332 Compare March 11, 2025 22:19
Inspired by NixOS#387725 (comment), script is based on NixOS#336172 using what i learned in NixOS#386865, part of NixOS#346453

Should be zero rebuilds.

All candidates were made using:

```shell

export NIXPKGS_ALLOW_UNFREE=1
export NIXPKGS_ALLOW_INSECURE=1
export NIXPKGS_ALLOW_BROKEN=1

git-wait restore .

test -s packages.json || ( set -x;
  time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages.json
)

list_attrpath_fname_col() {
    jq <packages.json 'to_entries[] | select(.value.meta.position==null|not) | "\(.key)\t\(.value.meta.position)"' -r |
        sed -e "s#\t$(realpath .)/#\t#" |
        sed -e 's#:\([0-9]*\)$#\t\1#' |
        grep . |
        grep -iv haskell |
        grep -iv /top-level/ |
        grep -iv chicken |
        grep pkgs/by-name/ |
        grep -iv build |
        grep -E '/(package|default)\.nix'
}

FLOCKDIR="$(mktemp -d)"
N_WORKERS=4
while read attrpath fname col; do
    grep -qE 'repo *= *("\$\{pname\}"|pname);' "$fname" || continue

    echo | (
        # mutex on fname
        flock --nonblock 200 || {
            >&2 echo "failed to aquire lock for $fname"
            exit 1
        }

        echo "$attrpath"
        data="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)" || exit
        test -n "$data" || exit
        pname="$(jq <<<"$data" .pname -r)"
        test -n "$pname" || exit

        (set -x
            sd -F '${pname}'  "$pname"         "$fname"
            sd -F ' = pname;' " = \"$pname\";" "$fname"
        )

        data2="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)"
        if [[ "$data" = "$data2" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
            exit
        fi

        (set -x
            sd -F ' rec {' ' {' "$fname"
        )

        data3="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json 2>/dev/nul)"

        if [[ "$data" = "$data3" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
        fi

    ) 200>"$FLOCKDIR"/"$(sha256sum - <<<"$fname" | cut -d' ' -f1)".lock &

    while [[ $(jobs -p | wc -l) -ge $N_WORKERS ]]; do
        wait -n < <(jobs -p) || true
    done

done < <(list_attrpath_fname_col)

wait

git restore .

time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages2.json
```

`diff packages{,2}.json` is empty, indicating that no package nor src derivation has changed.
I checked and cherry-picked the changes using `GIT_DIFF_OPTS='-u15' git -c interactive.singleKey=true add --patch`
@pbsds pbsds force-pushed the migrate-pname-1741296491 branch from f946332 to 357d253 Compare March 11, 2025 22:55
@pbsds pbsds merged commit e607ef1 into NixOS:master Mar 11, 2025
23 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: cinnamon Desktop environment 6.topic: xen-project The Xen Project hypervisor 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants