Skip to content

Nix profile silently emptied (all removals, no additions) coinciding with repeated FlakeHub auth failures and daemon restarts #201

Description

@WACampelo

Environment

  • Determinate Nixd version: 3.21.9
  • OS: openSUSE Tumbleweed (x86_64-linux)
  • Profile managed via nix profile install (imperative, flake refs to nixpkgs) plus one custom local flake at ~/.config/nix-global#packages.x86_64-linux.default
  • No home-manager, no NixOS module — plain Determinate Nix install with a systemd --user service (kanata.service) depending on ~/.nix-profile/bin/kanata

Summary

On 2026-08-30, my nix profile (~/.local/state/nix/profiles/profile) jumped from generation 5 to generation 6. Generation 6 removed every previously installed package and added nothing: kanata, kdotool, borgbackup, and a custom nix-global closure (which itself provided rustc, cargo, nodejs_22, python3, pip) all disappeared in one transition. This silently broke a systemd --user service depending on one of the removed binaries.

I did not run any nix profile remove, nix profile rollback, or similar command around that time (confirmed via full shell history review), and this is not a NixOS/home-manager system where a declarative rebuild could explain a profile replacement.

Evidence

1. nix profile history output — the generation transition

Version 5 (2026-08-27) <- 4:
  flake:nixpkgs#legacyPackages.x86_64-linux.borgbackup: 1.4.5, 1.4.5-man added
Version 6 (2026-08-30) <- 5:
  flake:nixpkgs#legacyPackages.x86_64-linux.borgbackup: 1.4.5, 1.4.5-man removed
  flake:nixpkgs#legacyPackages.x86_64-linux.kanata: 1.12.0 removed
  flake:nixpkgs#legacyPackages.x86_64-linux.kdotool: 0.2.3 removed
  path:/home/walter/.config/nix-global#packages.x86_64-linux.default: (no version) removed

2. nix profile diff-closures — confirms zero additions, ~1GB+ of closures removed

Generation 5→6 shows dozens of packages removed (rustc, cargo, llvm, nodejs, python3, openssh, systemd-minimal, borgbackup, kanata, kdotool, etc.) and not a single addition — inconsistent with a normal nix profile install/upgrade operation, which always adds at least the requested package.

3. journalctl — repeated daemon restarts with FlakeHub auth failures throughout the same day

ago 30 01:52:20 CampSUSE determinate-nixd[1613]: ... INFO determinate_nixd::gc: enable_automatic_garbage_collection=true; GC task will auto collect
ago 30 01:52:20 CampSUSE determinate-nixd[1613]: ...   WARN determinate_nixd::auth: Auth failed (transient error), trying again after 500ms, attempt: 1, max_attempts: 5
[... retries at 1000ms, 2000ms, 4000ms ...]
ago 30 01:52:28 CampSUSE determinate-nixd[1613]: ... ERROR determinate_nixd::auth: Authentication failure, s: Permanent

This same pattern (daemon restart → GC-enabled log line → retried auth → permanent auth failure) repeats at 01:51, 01:56, 06:35, and 23:15 on the same day.

4. The last restart precedes the first observed failure by 8 seconds

ago 30 23:15:39 CampSUSE determinate-nixd[1683]: ... in determinate_nixd::config::load with inner_config: DaemonInnerConfig { root: None, garbage_collector: GarbageCollectorConfig { strategy: Automatic }, authentication: AuthenticationConfig { additional_netrc_sources: [] }, builder: BuilderConfig { state: Enabled, memory_bytes: None, cpu_count: None }, telemetry: TelemetryConfig { sentry: None }, edge_cache_substituters: [] }
ago 30 23:15:47 CampSUSE (kanata)[2993]: kanata.service: Unable to locate executable '/home/walter/.nix-profile/bin/kanata': No such file or directory
ago 30 23:15:47 CampSUSE (kanata)[2993]: kanata.service: Failed at step EXEC spawning /home/walter/.nix-profile/bin/kanata: No such file or directory

(This EXEC failure then repeats every ~2 seconds for the rest of the day via systemd's restart-on-failure, since the service unit was still enabled.)

What I ruled out

  • btrfs/snapper rollback: the root filesystem (which includes /nix, no separate mount) had been rolled back via snapper earlier that day for an unrelated kernel/QEMU issue, but the old store paths for the removed packages were still physically present on disk (/nix/store/0zgbbqalm1nvg4mjk4n4b4mfm63m2847-kanata-1.12.0/bin/kanata still resolves and exists). This rules out a store-level rollback as the cause — only the profile manifest changed, not the store.
  • Disk-pressure-triggered "urgent" GC: root filesystem was at 61% used / ~197GB free at the time — nowhere near the documented urgent-GC threshold (<5% free) or even the steady-state band (5–20% free).
  • A command I ran: reviewed both history and .zsh_history in full; no nix profile remove/rollback/install --replace-all around the incident window.
  • home-manager: not installed on this system (which home-manager → not found).
  • A systemd timer/unit: systemctl --user list-timers and list-units | grep nix show nothing relevant.

Hypothesis

The only correlating event I could find in any log is the daemon restart + FlakeHub authentication permanently failing, happening in the same second-level window as the profile collapsing to a stripped generation with zero additions. I don't have visibility into determinate-nixd's internal GC/profile-management logic to confirm the mechanism, but the pattern suggests that automatic garbage collection or some profile-reconciliation step may be mishandling the case where FlakeHub auth is failing — e.g. treating the current profile's referenced packages as collectible, or falling back to an unexpectedly minimal profile state.

Local mitigation

Disabled automatic GC via /etc/determinate/config.json:

{ "garbageCollector": { "strategy": "disabled" } }

Request

Any insight into what could cause nix profile entries to be dropped this way — especially given the auth failures occurring at the same moments — would be appreciated. Happy to share the full journalctl output for that day if useful for reproducing or diagnosing this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions