Install NixOS on Hetzner Online

This article is about installing NixOS on Hetzner Online, which provides dedicated bare-metal servers.

This is not to be confused with Hetzner Cloud, that provides VMs (an example for how to install NixOS there is shown here).

There are three ways at the time to install NixOS on Hetzner dedicated:

1. From Hetzner's rescue image one can boot into the nixos installer using a custom kexec image that is configured with the fixed IPv6 provided by Hetzner and also contain your ssh key. Tip: The kexec tarball as generated by nixos-generators can remain put into the /boot partition for future use.
2. Hetzner also provides an interface to upload your own ISO-images. Also here you may want to build your own iso-image, which has openssh with ssh keys due the lack of a remote console.
3. An easier method to install NixOS on Hetzner, is to use the existing integration into NixOps.
4. An example to install NixOS in the Hetzner rescue mode, including full RAID partitioning, is available here.

Network configuration

From Hetzner's web interface, one can obtain both ipv4/ipv6 addresses and gateways. Hetzner does announce ipv6 addresses servers, so you need to assign those statically. In this example we use networkd to configure the interface. The same configuration can be used for both the kexec installation image and the final server configuration.

{ ... }: {
  # This make sure that our interface is named `eth0`.
  # This should be ok as long as you don't have multiple physical network cards
  # For multiple cards one could add a netdev unit to rename the interface based on the mac address
  networking.usePredictableInterfaceNames = false;
  systemd.network = {
    enable = true;
    networks."eth0".extraConfig = ''
      [Match]
      Name = eth0
      [Network]
      # Add your own assigned ipv6 subnet here here!
      Address = 2a01:4f9:ffff::1/64
      Gateway = fe80::1
      # optionally you can do the same for ipv4 and disable DHCP (networking.dhcpcd.enable = false;)
      # Address =  144.x.x.x/26
      # Gateway = 144.x.x.1
    '';
  };
}

Another possibility is to use networking.interfaces:

let
  external-mac = "00:11:22:33:44:55";
  ext-if = "et0";
  external-ip = "144.x.x.x";
  external-gw = "144.x.x.255";
  external-ip6 = "2a01:XXXX:XXXX::1";
  external-gw6 = "fe80::1";
  external-netmask = 27;
  external-netmask6 = 64;
in {
  # rename the external interface based on the MAC of the interface
  services.udev.extraRules = ''SUBSYSTEM=="net", ATTR{address}=="${external-mac}", NAME="${ext-if}"'';
  networking = {
    interfaces."${ext-if}" = {
      ipv4.addresses = [{
        address = external-ip;
        prefixLength = external-netmask;
      }];
      ipv6.addresses = [{
        address = external-ip6;
        prefixLength = external-netmask6;
      }];
    };
    defaultGateway6 = {
      address = external-gw6;
      interface = ext-if;
    };
    defaultGateway = external-gw;
  };
}

Bootstrap from the Rescue System

Here are some quick notes on how to bootstrap.

The nixos-install-scripts repo may also be a valuable resource:

https://github.com/nix-community/nixos-install-scripts/tree/master/hosters/hetzner-dedicated

Otherwise, inspiration for the kexec approach below comes from https://github.com/ofborg/infrastructure/commit/0712a5cf871b7a6d2fbbd2df539d3cd90ab8fa1f and https://github.com/andir/infra/tree/master/bootstrap

The main principle is that we will go from: Rescue system, install Nix, kexec into a NixOS system, finally install the system.

First, reboot the machine in Rescue mode. Make sure to select your SSH public key. SSH into the machine:

# Let root run the nix installer
mkdir -p /etc/nix
echo "build-users-group =" > /etc/nix/nix.conf

# Install Nix in single-user mode
curl -L https://nixos.org/nix/install | sh
. $HOME/.nix-profile/etc/profile.d/nix.sh

# Install nixos-generators
# This might take a while, so the verbose flag `-v` is included to monitor progress
nix-env -f https://github.com/nix-community/nixos-generators/archive/master.tar.gz -i -v

# Create a initial config, just to kexec into
cat <<EOF > /root/config.nix
{
  services.openssh.enable = true;
  users.users.root.openssh.authorizedKeys.keys = [
    # Replace with your public key
    "ssh-rsa AAAA..."
  ];
}
EOF

# Generate the kexec script
nixos-generate -o /root/result  -f kexec-bundle -c /root/config.nix

# Switch to the new system
/root/result

At this point the shell should stop responding. Kill the shell and ssh back into the machine. The server public key will have changed.

format() {
  parted -s "$1" -- mklabel msdos
  parted -s "$1" -- mkpart primary 1MiB 512MiB
  parted -s "$1" -- set 1 boot on
  parted -s "$1" -- mkpart primary 512MiB 100%
  parted -s "$1" -- print
}

# In this particular machine we have two NVMe disks
format /dev/nvme0n1
format /dev/nvme1n1

# Here we create a single btrfs volume using both disks. Change as needed

# TODO: Use boot.loader.grub.mirroredBoots
mkfs.ext2 /dev/nvme0n1p1
mkfs.btrfs -d raid0 -m raid1 -L nixos /dev/nvme0n1p2 /dev/nvme1n1p2

# Mount the disks
mount /dev/disk/by-label/nixos /mnt
mkdir /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot

# Generate the NixOS configuration.
nixos-generate-config --root /mnt

At this point, edit the /mnt/etc/nixos/configuration.nix and tune as needed. I just added the following lines:

boot.loader.grub.device = "/dev/nvme0n1";
services.openssh.enable = true;
users.users.root.openssh.authorizedKeys.keys = [
  "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGB1Pog97SWdV2UEA40V+3bML+lSZXEd48zCRlS/eGbY3rsXfgUXb5FIBulN9cET9g0OOAKeCZBR1Y2xXofiHDYkhk298rHDuir6cINuoMGUO7VsygUfKguBy63QMPHYnJBE1h+6sQGu/3X9G2o/0Ys2J+lZv4+N7Hqolhbg/Cu6/LUCsJM/udqTVwJGEqszDWPtuuTAIS6utB1QdL9EZT5WBb1nsNyHnIlCnoDKZvrrO9kM0FGKhjJG2skd3+NqmLhYIDhRhZvRnL9c8U8uozjbtj/N8L/2VCRzgzKmvu0Y1cZMWeAAdyqG6LoyE7xGO+SF4Vz1x6JjS9VxnZipIB zimbatm@nixos"
];

Finally run nixos-install, and then reboot the machine.

Voila! (after 1000 steps)

Bootstrap with cloud-init

Create a server with the Debian 11 image (using the web interface or the hcloud cli tool) and provide the following "user data" for cloud-init:

#cloud-config
runcmd:
- set -e
- /root/install-nix
- /root/.nix-profile/bin/nix --extra-experimental-features "nix-command flakes" build -L github:BBBSnowball/nixcfg#nixosConfigurations.hetzner-temp.config.system.build.toplevel --out-link /root/system && /root/.nix-profile/bin/nix-env -p /nix/var/nix/profiles/system --set /root/system
# NixOS manual suggests this. Not really needed here because we do a multi-user install.
- chown -R 0.0 /nix
- touch /etc/NIXOS
- echo "etc/nixos" >/etc/NIXOS_LUSTRATE
- echo "root/.ssh/authorized_keys" >>/etc/NIXOS_LUSTRATE
- echo "etc/ssh-shared-secret" >>/etc/NIXOS_LUSTRATE
- echo "var/lib/systemd/random-seed" >>/etc/NIXOS_LUSTRATE
- rm -rf /boot/efi/*
- mkdir /boot/efi/{EFI/systemd,EFI/BOOT} -p
- /nix/var/nix/profiles/system/bin/switch-to-configuration boot
- reboot
write_files:
- path: /root/install-nix
  permissions: '0755'
  content: |
    #!/bin/bash
    set -xe
    apt update && apt install -y gnupg2 sudo
    # Nix installer refuses to run as root...
    useradd --create-home -G sudo user
    sudo -u user  curl --fail -o ~user/install-nix-2.5.1 https://releases.nixos.org/nix/nix-2.5.1/install
    sudo -u user  curl --fail -o ~user/install-nix-2.5.1.asc https://releases.nixos.org/nix/nix-2.5.1/install.asc
    cp ~user/install-nix-2.5.1{,.asc} ~/
    sha256sum -c <<<'e265dfd8e80223633a9726009b42c534ac3d5f2b6da5ad6432ca1f6ea88206d0  /root/install-nix-2.5.1'
    echo 'user ALL=(ALL:ALL) NOPASSWD: ALL' >>/etc/sudoers
    yes | sudo -u user  sh ~user/install-nix-2.5.1 --daemon --no-channel-add

Change the third command to use your own flake (or fetch your config in some other way). The authorized_keys file is not removed so you can login with the SSH keys that you choose when creating the server. If it doesn't reboot to NixOS, check the syntax of your user data (and keep the first line!) and have a look at the log: journalctl --unit cloud-final

Your config has to replace the Debian bootloader. There is an EFI partition but the server uses legacy boot so make sure to replace the legacy bootloader, i.e. configure boot.loader.* to not use EFI.

I use this helper script to create the server and retrieve its SSH host key.

You can remove /old-root after booting into NixOS, which will free up about 1 GB. There is some interesting metadata in /old-root/var/lib/cloud/instances (e.g. the IP address) but you can also fetch similar information from Hetzner's API.