Install NixOS on Scaleway X86 Virtual Cloud Server

Create Scaleway Instance

If you know the Scaleway API you can do equivalent steps via API instead of using the webinterface.

Open Scaleway "Create a server" page
Enter name
Select region
Select server type
- This will only work for the V and X types of servers (e.g. VC1M or X64-15GB)
Select "Debian Stretch" or "Debian Mini Stretch" from the distributions tab

Note: At the time of writing of this guide the Ubuntu Xenial did not have a kexec enabled kernel and will therefore not work!

Create server
Wait until server has been provisioned and is done booting
- The status of the system can be checked on by looking at the console in the web interface

Build NixOS kexec Tarball

remote build

First you will need to build a NixOS kexec tarball. To do this follow the steps outlined below:

git clone https://github.com/cleverca22/nix-tests.git
cd nix-tests/kexec/
# Edit configuration.nix according to your needs. An example can be found below:
nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_tarball -I nixos-config=./configuration.nix -Q -j 4

copy the NixOS tarball

Copy the prepared NixOS kexec tarball to the server

scp result/tarball/nixos-system-x86_64-linux.tar.xz root@51.YY.XX.93:

Use ssh to access the server
- Use the corrensponding SSH key you configured at Scaleway before
Extract the tarball into /

cd /; tar -xf /root/nixos-system-x86_64-linux.tar.xz

local (Scaleway instance) build

We need to start from 17.09 because that comes with Linux kernel 4.9 which can be Kexec'd into from the Scaleway Debian Stretch. The system to be installed later be any version since it uses the regular boot procedure.

apt-get update
apt-get install -y git
sysctl kernel.unprivileged_userns_clone=1
adduser nix
addgroup sudo nix
su nix # nix installer only runs as non-root
cd
curl https://nixos.org/nix/install | sh
source ~/.nix-profile/etc/profile.d/nix.sh
git clone https://github.com/cleverca22/nix-tests.git
cd nix-tests/kexec/
vim configuration.nix

remove autoreboot.nix, comment out boot.supportedFilesystems = [ "zfs" ]; (as in the sample config below)

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-17.03.tar.gz nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_tarball -I nixos-config=./configuration.nix -Q -j 4
exit # back to root
cp /home/nix/nix-tests/kexec/result/tarball/nixos-system-x86_64-linux.tar.xz /nixos.tar.xz
cd /
tar -xf nixos.tar.xz

Example configuration.nix

(removed autoreboot.nix from imports, added SSH key)

# new cmd: nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_tarball -I nixos-config=./configuration.nix -Q -j 4

{ lib, pkgs, config, ... }:

with lib;

{
  imports = [ <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix> ./kexec.nix ./justdoit.nix ];

  #boot.supportedFilesystems = [ "zfs" ];
  boot.loader.grub.enable = false;
  boot.kernelParams = [
    "console=ttyS0,115200"          # allows certain forms of remote access, if the hardware is setup right
    "panic=30" "boot.panic_on_fail" # reboot the machine upon fatal boot issues
  ];
  systemd.services.sshd.wantedBy = mkForce [ "multi-user.target" ];
  networking.hostName = "kexec";
  # example way to embed an ssh pubkey into the tar
  users.users.root.openssh.authorizedKeys.keys = [ "ssh-rsa AAAAB3.... Your-SSH-key" ];
}

Start NixOS kexec System

Execute the kexec_nixos script

./kexec_nixos

The output should look something like this:

root@scalenix:/# ./kexec_nixos 
++ mktemp -d
+ cd /tmp/tmp.iDXuzu8Ec4
+ pwd
/tmp/tmp.iDXuzu8Ec4
+ mkdir initrd
+ pushd initrd
/tmp/tmp.iDXuzu8Ec4/initrd /tmp/tmp.iDXuzu8Ec4
+ cat /ssh_pubkey
cat: /ssh_pubkey: No such file or directory
+ find -type f
+ cpio -o -H newc
+ gzip -9
1 block
+ popd
/tmp/tmp.iDXuzu8Ec4
+ cat /nix/store/2lmw78k2ralvpn6fa270b53nz1xgqk8b-image/initrd extra.gz
+ kexec -l /nix/store/2lmw78k2ralvpn6fa270b53nz1xgqk8b-image/kernel --initrd=final.gz '--append=init=/nix/store/sv9hndbkrdxr1psi2jr82hkm1ba0j8bx-nixos-system-kexec-17.09.git.f3841ab/init loglevel=4 console=ttyS0,115200 panic=30 boot.panic_on_fail'
+ sync
+ echo 'executing kernel, filesystems will be improperly umounted'
executing kernel, filesystems will be improperly umounted
+ kexec -e
packet_write_wait: Connection to 51.XX.XX.93 port 22: Broken pipe

Once the ssh connection is broken take a look at the servers console in the webinterface. After a while you should see a NixOS root session:

Installing NixOS

Scaleway peculiarities

Scaleway used to start up VMs differently than other providers but new instances are configured for "local boot" which means they can just boot from EFI.

Scaleway Aarch64 peculiarities

If your server is aarch64, there is no working kexec on initrd image provided by Scaleway. So KEXEC_KERNEL tag won't work. This can be worked around by compiling static kexec and copying in to /boot and also by providing fake /sbin/init (a file which Scaleway's initrd launches at the end)

system.build.installBootLoader = pkgs.writeScript "installBootLoader.sh" ''
  #!/bin/sh

  export TOPLEVEL="$1"
  export PATH=${pkgs.coreutils}/bin

  cp -f $TOPLEVEL/kernel /boot/kernel
  cp -f $TOPLEVEL/initrd /boot/initrd
  cp -f ${pkgs.pkgsMusl.kexectools.overrideDerivation(old: {configureFlags = old.configureFlags ++ ["LDFLAGS=-static"]; }) }/bin/kexec /boot/kexec
  cp -f ${pkgs.pkgsMusl.busybox.override {enableStatic = true;}}/bin/busybox /boot/sh

  mkdir -p /sbin
  cat > /sbin/init <<EOF
  #!/boot/sh
  /boot/kexec -d -l /boot/kernel --initrd=/boot/initrd --append="init=$TOPLEVEL/init systemConfig=$TOPLEVEL $(cat $TOPLEVEL/kernel-params)"
  /boot/kexec -d -e
  EOF
  chmod 0777 /sbin/init
'';
boot.loader.grub.enable = false;

Step by Step Guide

Mount root volume
```
mount /dev/vda1 /mnt/
```
Delete all existing data from the previous Debian system
```
rm -rf /mnt/*
```
Create mountpoint for /nix on the 2nd volume (I want to use it for the nix store)
```
mkdir /mnt/nix
```
Create filesystem on the 2nd volume
```
mkfs.ext4 /dev/vdb -L nixstore
```
Mount 2nd volume
```
mount /dev/vdb /mnt/nix/
```
Create mountpoint for /boot to set up EFI (I want to use it for the nix store)
```
mkdir /mnt/boot
```
Mount EFI boot partition
```
mount /dev/vda15 /mnt/boot
```
Generate NixOS config (make sure after mounting /mnt/boot so it recognizes it)
```
nixos-generate-config --root /mnt
```
Edit /mnt/etc/nixos/configuration.nix
- Make sure to add users/keys to access the system later via SSH
Install NixOS nixos-install --root /mnt
- This might take some time
- Set root password at the end of it
reboot