User:2r/NixOS on ZFS

This is a userspace draft and is not supported by NixOS Wiki.

Enable ZFS on Existing Installation

Add the following lines to configuration:

boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "deadbeef";

Host ID should be unique, generate one with head -c 8 /etc/machine-id.

Rebuild system with nixos-rebuild switch.

Install NixOS on ZFS

Layout

Partitions

As swap on ZFS will cause deadlock and does not support hibernation, a separate swap partition should be created.

GPT partition table

	ESP	bpool	rpool	swap	BIOS boot sector
Filesystem	vfat	ZFS, feature limited for GRUB compatibility.	ZFS	swap	N/A
Content	grubx64.efi	/boot	/	swap	N/A
Encryption	No, can be validated with Secure Boot	LUKS1	ZFS Encryption	random/LUKS2	N/A

Datasets

As NixOS lacks a service to handle native ZFS mounting at boot, such as zfs-mount-generator, all mountable datasets must be created with mountpoint=legacy to be mounted with fileSystems option.

Datasets with canmount=off mountpoint=none are used as containers, that is, no data is stored directly under such datasets, but child datasets can inherit their properties or imitate directory structures, such as /var/log.

Dataset layout

Containers				mountpoint	canmount	comment
bpool	sys	BOOT	default	/boot	noauto
rpool	sys	ROOT	default	/	noauto
		DATA	local	/	off	container for datasets that do not need backup, such as /nix
			safe	/	off	container for datasets that need backup, such as /{root,home,home/user}

Encryption

Boot pool can be encrypted with LUKS1 to prevent initrd tempering, however ZFS on LUKS is discouraged on root pool as LUKS abstracts physical devices and thus not desirable. Also, data needs to be encrypted per disk, thus slower than per file, as with ZFS native encryption.

ZFS native encryption does not encrypt dataset paths and default properties. Custom properties containing colon custom:property is encrypted.

Also, as ZFS currently does not support replacing master key, once the passphrase/keyfile is compromised, the encrypted dataset must be destroyed to protect confidentiality. Therefore, users are advised to choose a strong password at the beginning.

Preparations

Download NixOS 64 Bit Minimal ISO image and SHA-256 checksum. Verify checksum and write the image to an external disk. Boot the computer from the disk afterwards.

After booting the computer, optionally configure SSH server and connect from another computer via SSH.

Identify target disks

List available disks with

ls -d /dev/disk/by-id/* | grep -v part

Partition

for i in {/dev/disk/by-id/disk1,/dev/disk/by-id/disk2}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+1G -t1:EF00 $i
sgdisk -n2:0:+4G -t2:BE00 $i
sgdisk -n3:0:0 -t3:BF00 $i

# with swap
# sgdisk -n3:0:-8G -t3:BF00 $i
# sgdisk -n4:0:0   -t4:8308 $i

# with BIOS
# sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done

Create pools

Multi-disk

Change

zpool create \
... \
/dev/disk/by-id/disk1-partX

to

zpool create \
... \
mirror \ # or raidz1 (discouraged), raidz2, raidz3
/dev/disk/by-id/disk1-partX \
/dev/disk/by-id/disk2-partX

Boot pool

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -d -o feature@async_destroy=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@extensible_dataset=enabled \
    -o feature@filesystem_limits=enabled \
    -o feature@hole_birth=enabled \
    -o feature@large_blocks=enabled \
    -o feature@lz4_compress=enabled \
    -o feature@spacemap_histogram=enabled \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=none \
    -R /mnt \
    bpool \
    /dev/disk/by-id/disk1-part2

Root pool

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R /mnt \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=none \
    rpool \
    /dev/disk/by-id/disk1-part3

Create datasets

Boot container

zfs create \
 -o canmount=off \
 -o mountpoint=none \
 bpool/sys

Root container

Encrypted

zfs create \
 -o canmount=off \
 -o mountpoint=none \
 -o encryption=on \
 -o keylocation=prompt \
 -o keyformat=passphrase \
 rpool/sys

Unencrypted

zfs create \
 -o canmount=off \
 -o mountpoint=none \
 rpool/sys

System datasets

zfs create -o canmount=off -o mountpoint=none bpool/NixOS/BOOT
zfs create -o canmount=off -o mountpoint=none rpool/NixOS/ROOT
zfs create -o canmount=off -o mountpoint=none rpool/NixOS/DATA
zfs create -o canmount=off -o mountpoint=/ rpool/NixOS/DATA/local
zfs create -o canmount=off -o mountpoint=/ rpool/NixOS/DATA/safe
zfs create -o mountpoint=legacy -o canmount=noauto bpool/NixOS/BOOT/default
zfs create -o mountpoint=legacy -o canmount=noauto rpool/NixOS/ROOT/default
mount -t zfs rpool/NixOS/ROOT/default /mnt
mkdir /mnt/boot
mount -t zfs bpool/NixOS/BOOT/default /mnt/boot
for i in {nix,}; do
zfs create -o canmount=on -o mountpoint=legacy rpool/NixOS/DATA/local/$i
mkdir -p /mnt/$i
mount -t zfs rpool/NixOS/DATA/local/$i /mnt/$i
done
for i in {root,home,home/user}; do
zfs create -o canmount=on -o mountpoint=legacy rpool/NixOS/DATA/safe/$i
mkdir -p /mnt/$i
mount -t zfs rpool/NixOS/DATA/safe/$i /mnt/$i
done
chmod 750 /mnt/root
chmod 700 /mnt/home/user

System Installation

Except a few additional options, the procedure is identical with normal installation.

General

Generate Host ID

head -c 8 /etc/machine-id

Generate and edit config file

nixos-generate-config --root /mnt
vim /mnt/etc/nixos/configuration.nix

Add ZFS:

boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "deadbeef";

If not using /dev/disk/by-id:

boot.zfs.devNodes = "/dev/disk/by-path";

If swap partition:

swapDevices = [
{
device = "/dev/disk/by-id/disk1-part4";
randomEncryption.enable = true;
}
];

GRUB

UEFI

For UEFI, delete systemd-boot and other boot.loader options and use GRUB:

boot.loader = {
# grub.efiInstallAsRemovable = true;
efi.canTouchEfiVariables = true;
efi.efiSysMountPoint = "/boot/efi";
grub.enable = true;
grub.version = 2;
grub.copyKernels = true;
grub.efiSupport = true;
grub.devices = [ "nodev" ];
grub.zfsSupport = true;
};

For multiple disks, mirror EFI system partition /boot/efi with boot.loader.grub.mirroredBoots option. See man configuration.nix.

BIOS

For BIOS, delete boot.loader options and use GRUB:

boot.loader = {
grub.enable = true;
grub.version = 2;
grub.copyKernels = true;
grub.zfsSupport = true;
grub.devices = [ "/dev/disk/by-id/disk1" "/dev/disk/by-id/disk2" ];
};

Installation

After finishing configuration, install system:

nixos-install --root /mnt

Finishing installation

Take an initial snapshot:

zfs snapshot -r rpool/sys@install
zfs snapshot -r bpool/sys@install

Unmount EFI system partition:

umount /mnt/boot/efi

Also unmount mirrored ESP, if any. Export pools:

zpool export bpool
zpool export rpool

LUKS Boot pool

Install cryptsetup

environment.systemPackages = [ pkgs.cryptsetup ];

nixos-rebuild switch

Generate encryption keys

mkdir /root/cryptkey.d/
chmod 700 /root/cryptkey.d/
dd bs=32 count=1 if=/dev/urandom of=/root/cryptkey.d/lukskey-bpool
dd bs=32 count=1 if=/dev/urandom of=/root/cryptkey.d/zfskey-rpool

Back up boot pool

zfs snapshot -r bpool/sys@pre-luks
zfs send -R bpool/sys@pre-luks > /root/bpool-pre-luks

Back up boot pool creation command, used for recreation:

zpool history bpool | head -n2 \
| grep 'zpool create' > /root/bpool-cmd

Unmount EFI system partitions

umount /boot/efi

Destroy boot pool

zpool destroy bpool

Enter LUKS password:

LUKS_PWD=rootpool

Create LUKS1 containers

for i in {/dev/disk/by-id/disk1,/dev/disk/by-id/disk2}; do
 cryptsetup luksFormat -q --type luks1 $i-part2 --key-file /root/cryptkey.d/lukskey-bpool
 echo $LUKS_PWD | cryptsetup luksAddKey $i-part2 --key-file /root/cryptkey.d/lukskey-bpool
 cryptsetup open $i-part2 luks-bpool-$i-part2 --key-file /root/cryptkey.d/lukskey-bpool
 echo luks-bpool-$i-part2 $i-part2 /root/cryptkey.d/lukskey-bpool discard >> /etc/crypttab
done

You can also convert crypttab entries to boot.initrd.luks.devices options.

Recreate boot pool:

cat /root/bpool-cmd

Remove -R /mnt and use devices from ls -d /dev/disk/by-id/dm*. Restore boot pool:

cat /root/bpool-pre-luks | zfs recv bpool/sys

Mount EFI system partition:

mount /boot/efi

Optionally, if you don't want to enter 2 passwords, one for LUKS bpool, one for rpool, use embedded keyfile:

zfs change-key -l \
-o keylocation=file:///root/cryptkey.d/zfskey-rpool \
-o keyformat=raw \
rpool/sys

Embed keyfile and enable LUKS GRUB:

boot.initrd.secrets = {
"/root/cryptkey.d/lukskey-bpool" = "/root/cryptkey.d/lukskey-bpool";
"/root/cryptkey.d/zfskey-rpool" = "/root/cryptkey.d/zfskey-rpool";
};
boot.loader.grub = {
enableCryptodisk = true;
};

Add import bpool service

systemd.services.zfs-import-bpool-mapper = {
wantedBy = [ "zfs-import.target" ];
description = "Import encrypted boot pool";
after = [ "cryptsetup.target" ];
before = [ "zfs-import-bpool.service" ];
serviceConfig = {
Type = "oneshot";
ExecStart = ''${pkgs.zfs}/bin/zpool import -d /dev/mapper bpool'';
};
};

Finally, apply changes

nixos-rebuild --install-bootloader boot

Ensure Installing the GRUB 2 is displayed, or else your system will not boot! Try again in local text console (not SSH, not terminal emulator), logged in as root.

Important: After reboot, back up /root/cryptkey.d/zfskey-rpool, In the possible event of LUKS header corruption, data on root dataset will only be available with this key.

Hibernation

This requires boot pool to be encrypted and a separate swap partition.

Unload swap

cat /proc/swaps
# /dev/dm-0
swapoff /dev/dm-0
cryptsetup close /dev/dm-0

Identify partition

ls -d /dev/disk/by-id/*-part4

Create keyfile and LUKS2 container

dd bs=32 count=1 if=/dev/urandom of=/root/cryptkey.d/lukskey-cryptswap-$DISK-part4
cryptsetup luksFormat -q --type luks2 /dev/disk/by-id/disk1-part4 --key-file /root/cryptkey.d/lukskey-cryptswap-$DISK-part4
cryptsetup luksOpen /dev/disk/by-id/disk1-part4 cryptswap-$DISK-part4 --key-file /root/cryptkey.d/lukskey-cryptswap-$DISK-part4 --allow-discards
mkswap /dev/mapper/cryptswap-$DISK-part4
swapon /dev/mapper/cryptswap-$DISK-part4

Embed keyfile in initrd and configure swap, remove random swap if needed

boot.initrd.secrets = {
"/root/cryptkey.d/lukskey-cryptswap-$DISK-part4" = "/root/cryptkey.d/lukskey-cryptswap-$DISK-part4";
};
swapDevices = [
{
device = "/dev/mapper/cryptswap-$DISK-part4";
}
];
boot.initrd.luks.devices = {
cryptswap-$DISK-part4 = {
device = "$DISK-part4";
allowDiscards = true;
keyFile = "/root/cryptkey.d/lukskey-cryptswap-$DISK-part4";
};
};
boot.resumeDevice = "/dev/mapper/cryptswap-$DISK-part4";

Apply changes

nixos-rebuild boot
reboot

However some devices might not support hibernation. My laptop, EliteBook 735 G5 with AMD Ryzen 5 PRO 2500U, the screen is either black or freezes when resuming from disk. Resuming from disk works intermittently with Ubuntu 20.10. This might be a issue related to integrated graphics (Radeon Vega Mobile).

Enter LUKS password in GRUB

GRUB will not let you retry password when the first attempt failed,

Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a): 
error: access denied.
error: no such cryptodisk found.
Entering rescue mode...
grub rescue>

Retry password with

grub rescue> cryptomount hd0,gpt2
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
Slot 1 opened
grub rescue> insmod normal
grub rescue> normal

Switch prefix disk

GRUB encodes the prefix disk in the bootloader, that is, if it fails to unlock the disk, although redundant disks are available, it will refuse to boot:

error: no such cryptodisk found.
Attempting to decrypt master key...
Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a): 
Slot 1 opened
error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
Entering rescue mode...
grub rescue>

Check prefix

grub rescue > set
prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/default@/grub
root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf

Change prefix

grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
grub rescue> root=crypto0

Load GRUB from other disks

grub rescue> insmod normal
grub rescue> normal