Install NixOS on Scaleway X86 Virtual Cloud Server

Create Scaleway Instance

If you know the Scaleway API you can do equivalent steps via API instead of using the webinterface.

• Open Scaleway "Create a server" page
• Enter name
• Select region
• Select server type
  • This will only work for the V and X types of servers (e.g. VC1M or X64-15GB)
• Select "Debian Sid" from the distributions tab

Note: At the time of writing of this guide the Ubuntu Xenial did not have a kexec enabled kernel and will therefore not work!

• Create server
• Wait until server has been provisioned and is done booting
  • The status of the system can be checked on by looking at the console in the webinterface

Build NixOS kexec Tarball

remote build

First you will need to build a NixOS kexec tarball. To do this follow the steps outlined below:

git clone https://github.com/cleverca22/nix-tests.git
cd nix-tests/kexec/
# Edit configuration.nix according to your needs. An example can be found below:
nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_tarball -I nixos-config=./configuration.nix -Q -j 4

copy the NixOS tarball

• Copy the prepared NixOS kexec tarball to the server

scp result/tarball/nixos-system-x86_64-linux.tar.xz root@51.YY.XX.93:

• Use ssh to access the server
  • Use the corrensponding SSH key you configured at Scaleway before
• Extract the tarball into /

cd /; tar -xf /root/nixos-system-x86_64-linux.tar.xz

local (Scaleway instance) build

apt-get update
apt-get install -y git
curl https://nixos.org/nix/install | sh
source ~/.nix-profile/etc/profile.d/nix.sh
git clone https://github.com/cleverca22/nix-tests.git
cd nix-tests/kexec/
vim configuration.nix

remove autoreboot.nix, comment out boot.supportedFilesystems = [ "zfs" ]; (as in the sample config below)

nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_tarball -I nixos-config=./configuration.nix -Q -j 4
cp result/tarball/nixos-system-x86_64-linux.tar.xz /nixos.tar.xz
cd /
tar -xf nixos.tar.xz

Example configuration.nix

(removed autoreboot.nix from imports, added SSH key)

# new cmd: nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_tarball -I nixos-config=./configuration.nix -Q -j 4

{ lib, pkgs, config, ... }:

with lib;

{
  imports = [ <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix> ./kexec.nix ./justdoit.nix ];

  #boot.supportedFilesystems = [ "zfs" ];
  boot.loader.grub.enable = false;
  boot.kernelParams = [
    "console=ttyS0,115200"          # allows certain forms of remote access, if the hardware is setup right
    "panic=30" "boot.panic_on_fail" # reboot the machine upon fatal boot issues
  ];
  systemd.services.sshd.wantedBy = mkForce [ "multi-user.target" ];
  networking.hostName = "kexec";
  # example way to embed an ssh pubkey into the tar
  users.users.root.openssh.authorizedKeys.keys = [ "ssh-rsa AAAAB3.... Your-SSH-key" ];
}

Start NixOS kexec System

• Execute the kexec_nixos script

./kexec_nixos

• The output should look something like this:

root@scalenix:/# ./kexec_nixos 
++ mktemp -d
+ cd /tmp/tmp.iDXuzu8Ec4
+ pwd
/tmp/tmp.iDXuzu8Ec4
+ mkdir initrd
+ pushd initrd
/tmp/tmp.iDXuzu8Ec4/initrd /tmp/tmp.iDXuzu8Ec4
+ cat /ssh_pubkey
cat: /ssh_pubkey: No such file or directory
+ find -type f
+ cpio -o -H newc
+ gzip -9
1 block
+ popd
/tmp/tmp.iDXuzu8Ec4
+ cat /nix/store/2lmw78k2ralvpn6fa270b53nz1xgqk8b-image/initrd extra.gz
+ kexec -l /nix/store/2lmw78k2ralvpn6fa270b53nz1xgqk8b-image/kernel --initrd=final.gz '--append=init=/nix/store/sv9hndbkrdxr1psi2jr82hkm1ba0j8bx-nixos-system-kexec-17.09.git.f3841ab/init loglevel=4 console=ttyS0,115200 panic=30 boot.panic_on_fail'
+ sync
+ echo 'executing kernel, filesystems will be improperly umounted'
executing kernel, filesystems will be improperly umounted
+ kexec -e
packet_write_wait: Connection to 51.XX.XX.93 port 22: Broken pipe

• Once the ssh connection is broken take a look at the servers console in the webinterface. After a while you should see a NixOS root session:

Installing NixOS

Scaleway peculiarities

Scaleway starts up VMs differently than other providers. This results in the following things that need to be done so that NixOS can boot:

• Scaleway start scripts mount /dev/vda -> No partitioning of the root volume possible
• You have to configure 3 "tags" in the Scaleway configuration of the system to tell the start script which kernel and initrd to load
  • KEXEC_KERNEL=/boot/nixos-kernel
  • KEXEC_INITRD=/boot/nixos-initrd
  • KEXEC_APPEND=init=/boot/nixos-init

    File:Scaleway-tags.png

    Screenshot of Scaleway webinterface tag setting showing 2 of the 3 tags

• You need to configure NixOS with the following options

boot.loader.initScript.enable = true;
boot.loader.generationsDir.enable = true;
boot.loader.generationsDir.copyKernels = true;
boot.loader.grub.enable = false;

Scaleway Aarch64 peculiarities

If your server is aarch64, there is no working kexec on initrd image provided by Scaleway. So KEXEC_KERNEL tag won't work. This can be worked around by compiling static kexec and copying in to /boot and also by providing fake /sbin/init (a file which Scaleway's initrd launches at the end)

system.build.installBootLoader = pkgs.writeScript "installBootLoader.sh" ''
  #!/bin/sh

  export TOPLEVEL=$(readlink -f /nix/var/nix/profiles/system)

  cp -f $TOPLEVEL/kernel /boot/kernel
  cp -f $TOPLEVEL/initrd /boot/initrd
  cp -f ${pkgs.pkgsMusl.kexectools.overrideDerivation(old: {configureFlags = old.configureFlags ++ ["LDFLAGS=-static"]; }) }/bin/kexec /boot/kexec

  mkdir -p /sbin
  cat > /sbin/init <<EOF
  #!/bin/sh
  /boot/kexec -d -l /boot/kernel --initrd=/boot/initrd --append="init=$TOPLEVEL/init systemConfig=$TOPLEVEL $(cat $TOPLEVEL/kernel-params)"
  sync
  /boot/kexec -d -e
  EOF
  chmod 0777 /sbin/init
'';
boot.loader.grub.enable = false;

Step by Step Guide

• Mount /dev/vda

  mount /dev/vda /mnt/
  

• Delete all existing data from the previous Debian system

  rm -rf /mnt/*
  

• Create mountpoint for /nix on the 2nd volume (I want to use it for the nix store)

  mkdir /mnt/nix
  

• Create filesystem on the 2nd volume

  mkfs.ext4 /dev/vdb -L nixstore
  

• Mount 2nd volume

  mount /dev/vdb /mnt/nix/
  

• Generate NixOS config

  nixos-generate-config --root /mnt
  

• Edit /mnt/etc/nixos/configuration.nix
  • Remove the default boot.loader.grub entries
  • Add the following

boot.loader.initScript.enable = true;
boot.loader.generationsDir.enable = true;
boot.loader.generationsDir.copyKernels = true;
boot.loader.grub.enable = false;

• • Make sure to add users/keys to access the system later via SSH
• Install NixOS nixos-install --root /mnt
  • This might take some time
  • Set root password at the end of
• reboot