User:2r/NixOS on ZFS

This is a userspace draft and is not supported by NixOS Wiki.

Enable ZFS on Existing Installation

Add the following lines to configuration:

boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "deadbeef";

Host ID should be unique, generate one with head -c 8 /etc/machine-id.

Rebuild system with nixos-rebuild switch.

Install NixOS on ZFS

Layout

Partitions

As swap on ZFS will cause deadlock and does not support hibernation, a separate swap partition should be created.

GPT partition table

	ESP	bpool	rpool	swap	BIOS boot sector
Filesystem	vfat	ZFS, feature limited for GRUB compatibility.	ZFS	swap	N/A
Content	grubx64.efi	/boot	/	swap	N/A
Encryption	No, can be validated with Secure Boot	LUKS1	ZFS Encrytion	random/LUKS2	N/A

Datasets

As NixOS lacks a service to handle native ZFS mounting at boot, such as zfs-mount-generator, all mountable datasets must be created with mountpoint=legacy to be mounted with fileSystems option.

Datasets with canmount=off mountpoint=none are used as containers, that is, no data is stored directly under such datasets, but child datasets can inherit their properties or imitate directory structures, such as /var/log.

Dataset layout

Containers				mountpoint	canmount	comment
bpool	sys	BOOT	default	/boot	noauto
rpool	sys	ROOT	default	/	noauto
		DATA	local	/	off	container for datasets that do not need backup, such as /nix
			safe	/	off	container for datasets that need backup, such as /{root,home,home/user}

Encryption

Boot pool can be encrypted with LUKS1 to prevent initrd tempering, however ZFS on LUKS is discouraged on root pool as LUKS abstracts physical devices and thus not desirable. Also, data needs to be encrypted per disk, thus slower than per file, as with ZFS native encryption.

ZFS native encryption does not encrypt dataset paths and default properties. Custom properties containing colon custom:property is encrypted.

Also, as ZFS currently does not support replacing master key, once the passphrase/keyfile is compromised, the encrypted dataset must be destroyed to protect confidentiality. Therefore, users are advised to choose a strong password at the beginning.

Preparations

Download NixOS 64 Bit Minimal ISO image and SHA-256 checksum. Verify checksum and write the image to an external disk. Boot the computer from the disk afterwards.

After booting the computer, optionally configure SSH server and connect from another computer via SSH.

Identify target disks

List available disks with

ls -d /dev/disk/by-id/* | grep -v part

Partition

for i in {/dev/disk/by-id/disk1,/dev/disk/by-id/disk2}; do
sgdisk --zap-all $i
sgdisk -n1:1M:+1G -t1:EF00 $i
sgdisk -n2:0:+4G -t2:BE00 $i
sgdisk -n3:0:0 -t3:BF00 $i

# with swap
# sgdisk -n3:0:-8G -t3:BF00 $i
# sgdisk -n4:0:0   -t4:8308 $i

# with BIOS
# sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
done

Create pools

Multi-disk

Change

zpool create \
... \
/dev/disk/by-id/disk1-partX

to

zpool create \
... \
mirror \ # or raidz1 (discouraged), raidz2, raidz3
/dev/disk/by-id/disk1-partX \
/dev/disk/by-id/disk2-partX

Boot pool

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -d -o feature@async_destroy=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@extensible_dataset=enabled \
    -o feature@filesystem_limits=enabled \
    -o feature@hole_birth=enabled \
    -o feature@large_blocks=enabled \
    -o feature@lz4_compress=enabled \
    -o feature@spacemap_histogram=enabled \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=none \
    -R /mnt \
    bpool \
    /dev/disk/by-id/disk1-part2

Root pool

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R /mnt \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=none \
    rpool \
    /dev/disk/by-id/disk1-part3

Create datasets

Boot container

zfs create \
 -o canmount=off \
 -o mountpoint=none \
 bpool/sys

Root container

Encrypted

zfs create \
 -o canmount=off \
 -o mountpoint=none \
 -o encryption=on \
 -o keylocation=prompt \
 -o keyformat=passphrase \
 rpool/sys

Unencrypted

zfs create \
 -o canmount=off \
 -o mountpoint=none \
 rpool/sys

System datasets

zfs create -o canmount=off -o mountpoint=none bpool/NixOS/BOOT
zfs create -o canmount=off -o mountpoint=none rpool/NixOS/ROOT
zfs create -o canmount=off -o mountpoint=none rpool/NixOS/DATA
zfs create -o canmount=off -o mountpoint=/ rpool/NixOS/DATA/local
zfs create -o canmount=off -o mountpoint=/ rpool/NixOS/DATA/safe
zfs create -o mountpoint=legacy -o canmount=noauto bpool/NixOS/BOOT/default
zfs create -o mountpoint=legacy -o canmount=noauto rpool/NixOS/ROOT/default
mount -t zfs rpool/NixOS/ROOT/default /mnt
mkdir /mnt/boot
mount -t zfs bpool/NixOS/BOOT/default /mnt/boot
for i in {nix,}; do
zfs create -o canmount=on -o mountpoint=legacy rpool/NixOS/DATA/local/$i
mkdir -p /mnt/$i
mount -t zfs rpool/NixOS/DATA/local/$i /mnt/$i
done
for i in {root,home,home/user}; do
zfs create -o canmount=on -o mountpoint=legacy rpool/NixOS/DATA/safe/$i
mkdir -p /mnt/$i
mount -t zfs rpool/NixOS/DATA/safe/$i /mnt/$i
done
chmod 750 /mnt/root
chmod 700 /mnt/home/user

System Installation

Except a few additional options, the procedure is identical with normal installation.

General

Generate Host ID

head -c 8 /etc/machine-id

Generate and edit config file

nixos-generate-config --root /mnt
vim /mnt/etc/nixos/configuration.nix

Add ZFS:

boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "deadbeef";

If not using /dev/disk/by-id:

boot.zfs.devNodes = "/dev/disk/by-path";

If swap partition:

swapDevices = [
{
device = "/dev/disk/by-id/disk1-part4";
randomEncryption.enable = true;
}
];

GRUB

UEFI

For UEFI, delete systemd-boot and other boot.loader options and use GRUB:

boot.loader = {
# grub.efiInstallAsRemovable = true;
efi.canTouchEfiVariables = true;
efi.efiSysMountPoint = "/boot/efi";
grub.enable = true;
grub.version = 2;
grub.copyKernels = true;
grub.efiSupport = true;
grub.devices = [ "nodev" ];
};

For multiple disks, mirror EFI system partition /boot/efi with boot.loader.grub.mirroredBoots option. See man configuration.nix.

BIOS

For BIOS, delete boot.loader options and use GRUB:

boot.loader = {
grub.enable = true;
grub.version = 2;
grub.copyKernels = true;
grub.devices = [ "/dev/disk/by-id/disk1" "/dev/disk/by-id/disk2" ];
};

Installation

After finishing configuration, install system:

nixos-install --root /mnt

Finishing installation

Take an initial snapshot:

zfs snapshot -r rpool/sys@install
zfs snapshot -r bpool/sys@install

Unmount EFI system partition:

umount /mnt/boot/efi

Also unmount mirrored ESP, if any. Export pools:

zpool export bpool
zpool export rpool