Installing NixOS on Hetzner Online dedicated servers

Enter rescue system

SSH into rescue system

The rescue system can be accessed with the credentials given by Hetzner.

Get inside, save this script somewhere and execute it. It will kexec into a NixOS live system. Swap the SSH key as needed.

# Let root run the nix installer
mkdir -p /etc/nix
echo "build-users-group =" > /etc/nix/nix.conf

# Install Nix in single-user mode
curl -L https://nixos.org/nix/install | sh
. $HOME/.nix-profile/etc/profile.d/nix.sh

# Install nixos-generators
# This might take a while, so the verbose flag `-v` is included to monitor progress
# You may need to use an old channel like NixOS 22.05 because kexec is not included anymore in nixos-generators(?!!)
nix-env -iA nixpkgs.nixos-generators

# Create a initial config, just to kexec into
cat <<EOF > /root/config.nix
{
  services.openssh.enable = true;
  users.users.root.openssh.authorizedKeys.keys = [
    # Replace with your public key
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDs+LyhedR8+3W2xjQglnL9ZQMkpA/69rE9nyPptcj4a hal@arch"
  ];
}
EOF

# Generate the kexec script
nixos-generate -o /root/result  -f kexec-bundle -c /root/config.nix

# Switch to the new system
/root/result

You will most likely need to delete the host key entry in authorized_hosts after it hangs. SSH again into the system...

The live system may have a pending shutdown. Cancel it. shutdown -c

Install NixOS

Here you have the magic script. yolo.

!/usr/bin/env bash

# Installs NixOS on a Hetzner server, wiping the server.
set -euox pipefail

# Replace with your key
SSH_PUB_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDs+LyhedR8+3W2xjQglnL9ZQMkpA/69rE9nyPptcj4a hal@arch"
MY_HOSTNAME=fafserver
MY_HOSTID=f4f4f4f4

# Undo existing setups to allow running the script multiple times to iterate on it.
# We allow these operations to fail for the case the script runs the first time.
umount -R /mnt || true
sleep 2
zpool destroy -f tank

devices=(/dev/nvme0n1 /dev/nvme1n1)

##
## Partition disks
##

# Generate "NixOS1", "NixOS2" etc for each device and store in the labels list
labels=()
for i in "${!devices[@]}"; do
    labels+=("NixOS$((i+1))")
done

efi_size=512M


for i in "${!devices[@]}"; do
    device="${devices[$i]}"
    label="${labels[$i]}"

    sgdisk --zap-all "$device"

    sgdisk --clear \
           --new=1:0:"+$efi_size" \
           --typecode=1:ef00 \
           --change-name=1:"EFI System Partition" \
           --new=2:0:0 \
           --typecode=2:8300 \
           --change-name=2:"$label" \
           "$device"

done

##
## Create ZFS pool
##
zpool_args=(
    "-m none"
    "-o ashift=12"
    "-f"
    "tank mirror"
)

for i in "${!labels[@]}"; do
    disk_path="/dev/disk/by-partlabel/${labels[$i]}"
    zpool_args+=("$disk_path")
done

echo Waiting for disks to appear
sleep 2
eval "zpool create ${zpool_args[@]}"

# Create datasets, NixOS will be installed to its corresponding one
zfs create -o mountpoint=legacy -o atime=on -o relatime=on -o compression=lz4 -o xattr=sa -o acltype=posixacl tank/nixos

# This is ancient wisdom about reserving space that may not be needed anymore
# But we've encountered issues with full disks before, so it doesn't hurt to keep a gig spare
zfs create -o refreservation=1G -o mountpoint=none tank/reserved

# this creates a special volume for db data see https://wiki.archlinux.org/index.php/ZFS#Databases
zfs create -o mountpoint=legacy \
    -o recordsize=8K \
    -o primarycache=metadata \
    -o logbias=throughput \
    tank/postgres

##
## NixOS pre-installation mounts
##

# Mount the filesystems manually. The nixos installer will detect these mountpoints
# and save them to /mnt/nixos/hardware-configuration.nix during the install process.
mount -t zfs tank/nixos /mnt

# With the OS ready
for i in "${!devices[@]}"; do
    device="${devices[$i]}"
    partition="${device}p1"  # Assuming the first partition is suffixed with "p1"
    mount_path="/mnt/boot$((i+1))"

    mkfs.vfat "$partition"

    mkdir -p "$mount_path"
    mount "$partition" "$mount_path"

done


nixos-generate-config --root /mnt

NIXOS_INTERFACE=$(ip route get 8.8.8.8 | grep -Po '(?<=dev )(\S+)')
IP_V4=$(ip route get 8.8.8.8 | grep -Po '(?<=src )(\S+)')
read _ _ DEFAULT_GATEWAY _ < <(ip route list match 0/0); echo "$DEFAULT_GATEWAY"
echo "Determined DEFAULT_GATEWAY as $DEFAULT_GATEWAY"

# Generate `configuration.nix`. Note that we splice in shell variables.
cat > /mnt/etc/nixos/configuration.nix <<EOF
{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use GRUB2 as the boot loader.
  # We don't use systemd-boot because GRUB2 has a NixOS module for dual EFI partitions
  boot.loader.systemd-boot.enable = false;

  boot.loader.grub = {
    enable = true;
    efiSupport = true;
    efiInstallAsRemovable = true;
    mirroredBoots = [
      {
       devices = [ "nodev" ];
       path = "/boot1";
      }
      {
       devices = [ "nodev" ];
       path = "/boot2";
      }
    ];
    copyKernels = true;
  };

  boot.supportedFilesystems = [ "zfs" ];

  networking.hostName = "$MY_HOSTNAME";
  networking.hostId = "$MY_HOSTID";

  # Network (Hetzner uses static IP assignments, and we don't use DHCP here)
  networking.useDHCP = false;
  networking.interfaces."$NIXOS_INTERFACE".ipv4.addresses = [
    {
      address = "$IP_V4";
      prefixLength = 24;
    }
  ];
  networking.defaultGateway = "$DEFAULT_GATEWAY";
  networking.defaultGateway6 = { address = "fe80::1"; interface = "$NIXOS_INTERFACE"; };
  networking.nameservers = [ "8.8.8.8" ];

  # Initial empty root password for easy login:
  users.users.root.initialHashedPassword = "";
  services.openssh.permitRootLogin = "prohibit-password";

  users.users.root.openssh.authorizedKeys.keys = ["$SSH_PUB_KEY"];

  services.openssh.enable = true;

}
EOF

# Install NixOS
PATH="$PATH" NIX_PATH="$NIX_PATH" `which nixos-install` \
  --no-root-passwd --root /mnt --max-jobs 40

echo Take a final look at the installed system before rebootin