|
|
| (2 intermediate revisions by one other user not shown) |
| Line 1: |
Line 1: |
| This is a userspace draft and is not supported by NixOS Wiki.
| | Moved to here: |
| | https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html |
|
| |
|
| == Enable ZFS on Existing Installation ==
| | Last version is: https://wiki.nixos.org/w/index.php?title=User:2r/NixOS_on_ZFS&oldid=5406 |
| Add the following lines to configuration:
| |
| <pre>boot.supportedFilesystems = [ "zfs" ];
| |
| networking.hostId = "deadbeef";</pre>
| |
| Host ID should be unique, generate one with <code>head -c 8 /etc/machine-id</code>.
| |
| | |
| Rebuild system with <code>nixos-rebuild switch</code>.
| |
| | |
| == Install NixOS on ZFS ==
| |
| === Layout ===
| |
| ==== Partitions ====
| |
| As swap on ZFS will cause deadlock and does not support hibernation, a separate swap partition should be created.
| |
| | |
| {| class="wikitable"
| |
| |+ GPT partition table
| |
| |-
| |
| |
| |
| | ESP
| |
| | bpool
| |
| | rpool
| |
| | swap
| |
| | BIOS boot sector
| |
| |-
| |
| |width=5% valign=top| Filesystem
| |
| |width=20% valign=top| vfat
| |
| |width=20% valign=top| ZFS, feature limited for GRUB compatibility.
| |
| |width=20% valign=top| ZFS
| |
| |width=20% valign=top| swap
| |
| |width=5% valign=top| N/A
| |
| |-
| |
| |width=5% valign=top| Content
| |
| |width=20% valign=top| <code>grubx64.efi</code>
| |
| |width=20% valign=top| <code>/boot</code>
| |
| |width=20% valign=top| <code>/</code>
| |
| |width=20% valign=top| swap
| |
| |width=5% valign=top| N/A
| |
| |-
| |
| |width=5% valign=top| Encryption
| |
| |width=20% valign=top| No, can be validated with Secure Boot
| |
| |width=20% valign=top| LUKS1
| |
| |width=20% valign=top| ZFS Encryption
| |
| |width=20% valign=top| random/LUKS2
| |
| |width=5% valign=top| N/A
| |
| |}
| |
| ==== Datasets ====
| |
| As NixOS lacks a service to handle native ZFS mounting at boot, such as <code>zfs-mount-generator</code>, all mountable datasets must be created with <code>mountpoint=legacy</code> to be mounted with <code>fileSystems</code> option.
| |
| | |
| Datasets with <code>canmount=off mountpoint=none</code> are used as containers, that is, no data is stored directly under such datasets, but child datasets can inherit their properties or imitate directory structures, such as <code>/var/log</code>.
| |
| | |
| {| class="wikitable"
| |
| |+ Dataset layout
| |
| |-
| |
| |colspan="3"| Containers
| |
| |
| |
| | mountpoint
| |
| | canmount
| |
| | comment
| |
| |-
| |
| | bpool
| |
| | sys
| |
| | BOOT
| |
| | default
| |
| | /boot
| |
| | noauto
| |
| |-
| |
| |rowspan="3"| rpool
| |
| |rowspan="3"| sys
| |
| | ROOT
| |
| | default
| |
| | /
| |
| | noauto
| |
| |-
| |
| |rowspan="2"| DATA
| |
| | local
| |
| | /
| |
| | off
| |
| | container for datasets that do not need backup, such as /nix
| |
| |-
| |
| | safe
| |
| | /
| |
| | off
| |
| | container for datasets that need backup, such as /{root,home,home/user}
| |
| |}
| |
| ==== Encryption ====
| |
| Boot pool can be encrypted with LUKS1 to prevent initrd tempering, however ZFS on LUKS is discouraged on root pool as LUKS abstracts physical devices and thus not desirable. Also, data needs to be encrypted per disk, thus slower than per file, as with ZFS native encryption.
| |
| | |
| ZFS native encryption does not encrypt dataset paths and default properties. Custom properties containing colon <code>custom:property</code> is encrypted.
| |
| | |
| Also, as ZFS currently does not support replacing master key, once the passphrase/keyfile is compromised, the encrypted dataset must be destroyed to protect confidentiality. Therefore, users are advised to choose a strong password at the beginning.
| |
| | |
| === Preparations ===
| |
| Download NixOS 64 Bit Minimal ISO image and SHA-256 checksum. Verify checksum and write the image to an external disk. Boot the computer from the disk afterwards.
| |
| | |
| After booting the computer, optionally configure SSH server and connect from another computer via SSH.
| |
| | |
| === Identify target disks ===
| |
| List available disks with
| |
| <pre>ls -d /dev/disk/by-id/* | grep -v part</pre>
| |
| | |
| === Partition ===
| |
| <pre>for i in {/dev/disk/by-id/disk1,/dev/disk/by-id/disk2}; do
| |
| sgdisk --zap-all $i
| |
| sgdisk -n1:1M:+1G -t1:EF00 $i
| |
| sgdisk -n2:0:+4G -t2:BE00 $i
| |
| sgdisk -n3:0:0 -t3:BF00 $i
| |
| | |
| # with swap
| |
| # sgdisk -n3:0:-8G -t3:BF00 $i
| |
| # sgdisk -n4:0:0 -t4:8308 $i
| |
| | |
| # with BIOS
| |
| # sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i
| |
| done</pre>
| |
| | |
| === Create pools ===
| |
| ==== Multi-disk ====
| |
| Change
| |
| <pre>zpool create \
| |
| ... \
| |
| /dev/disk/by-id/disk1-partX</pre>
| |
| to
| |
| <pre>zpool create \
| |
| ... \
| |
| mirror \ # or raidz1 (discouraged), raidz2, raidz3
| |
| /dev/disk/by-id/disk1-partX \
| |
| /dev/disk/by-id/disk2-partX</pre>
| |
| ==== Boot pool ====
| |
| <pre>zpool create \
| |
| -o ashift=12 \
| |
| -o autotrim=on \
| |
| -d -o feature@async_destroy=enabled \
| |
| -o feature@bookmarks=enabled \
| |
| -o feature@embedded_data=enabled \
| |
| -o feature@empty_bpobj=enabled \
| |
| -o feature@enabled_txg=enabled \
| |
| -o feature@extensible_dataset=enabled \
| |
| -o feature@filesystem_limits=enabled \
| |
| -o feature@hole_birth=enabled \
| |
| -o feature@large_blocks=enabled \
| |
| -o feature@lz4_compress=enabled \
| |
| -o feature@spacemap_histogram=enabled \
| |
| -O acltype=posixacl \
| |
| -O canmount=off \
| |
| -O compression=lz4 \
| |
| -O devices=off \
| |
| -O normalization=formD \
| |
| -O relatime=on \
| |
| -O xattr=sa \
| |
| -O mountpoint=none \
| |
| -R /mnt \
| |
| bpool \
| |
| /dev/disk/by-id/disk1-part2</pre>
| |
| ==== Root pool ====
| |
| <pre>zpool create \
| |
| -o ashift=12 \
| |
| -o autotrim=on \
| |
| -R /mnt \
| |
| -O acltype=posixacl \
| |
| -O canmount=off \
| |
| -O compression=zstd \
| |
| -O dnodesize=auto \
| |
| -O normalization=formD \
| |
| -O relatime=on \
| |
| -O xattr=sa \
| |
| -O mountpoint=none \
| |
| rpool \
| |
| /dev/disk/by-id/disk1-part3</pre>
| |
| === Create datasets ===
| |
| ==== Boot container ====
| |
| <pre>zfs create \
| |
| -o canmount=off \
| |
| -o mountpoint=none \
| |
| bpool/sys</pre>
| |
| ==== Root container ====
| |
| Encrypted
| |
| <pre>zfs create \
| |
| -o canmount=off \
| |
| -o mountpoint=none \
| |
| -o encryption=on \
| |
| -o keylocation=prompt \
| |
| -o keyformat=passphrase \
| |
| rpool/sys</pre>
| |
| Unencrypted
| |
| <pre>zfs create \
| |
| -o canmount=off \
| |
| -o mountpoint=none \
| |
| rpool/sys</pre>
| |
| ==== System datasets ====
| |
| <pre>zfs create -o canmount=off -o mountpoint=none bpool/NixOS/BOOT
| |
| zfs create -o canmount=off -o mountpoint=none rpool/NixOS/ROOT
| |
| zfs create -o canmount=off -o mountpoint=none rpool/NixOS/DATA
| |
| zfs create -o canmount=off -o mountpoint=/ rpool/NixOS/DATA/local
| |
| zfs create -o canmount=off -o mountpoint=/ rpool/NixOS/DATA/safe
| |
| zfs create -o mountpoint=legacy -o canmount=noauto bpool/NixOS/BOOT/default
| |
| zfs create -o mountpoint=legacy -o canmount=noauto rpool/NixOS/ROOT/default
| |
| mount -t zfs rpool/NixOS/ROOT/default /mnt
| |
| mkdir /mnt/boot
| |
| mount -t zfs bpool/NixOS/BOOT/default /mnt/boot
| |
| for i in {nix,}; do
| |
| zfs create -o canmount=on -o mountpoint=legacy rpool/NixOS/DATA/local/$i
| |
| mkdir -p /mnt/$i
| |
| mount -t zfs rpool/NixOS/DATA/local/$i /mnt/$i
| |
| done
| |
| for i in {root,home,home/user}; do
| |
| zfs create -o canmount=on -o mountpoint=legacy rpool/NixOS/DATA/safe/$i
| |
| mkdir -p /mnt/$i
| |
| mount -t zfs rpool/NixOS/DATA/safe/$i /mnt/$i
| |
| done
| |
| chmod 750 /mnt/root
| |
| chmod 700 /mnt/home/user</pre>
| |
| === System Installation ===
| |
| Except a few additional options, the procedure is identical with normal installation.
| |
| | |
| ==== General ====
| |
| Generate Host ID<pre>head -c 8 /etc/machine-id</pre>
| |
| | |
| Generate and edit config file<pre>nixos-generate-config --root /mnt
| |
| vim /mnt/etc/nixos/configuration.nix</pre>
| |
| Add ZFS:
| |
| <pre>boot.supportedFilesystems = [ "zfs" ];
| |
| networking.hostId = "deadbeef";</pre>
| |
| | |
| If not using <code>/dev/disk/by-id</code>:<pre>boot.zfs.devNodes = "/dev/disk/by-path";</pre>
| |
| | |
| If swap partition:<pre>swapDevices = [
| |
| {
| |
| device = "/dev/disk/by-id/disk1-part4";
| |
| randomEncryption.enable = true;
| |
| }
| |
| ];</pre>
| |
| | |
| ==== GRUB ====
| |
| ===== UEFI =====
| |
| For UEFI, delete systemd-boot and other <code>boot.loader</code> options and use GRUB:
| |
| <pre>boot.loader = {
| |
| # grub.efiInstallAsRemovable = true;
| |
| efi.canTouchEfiVariables = true;
| |
| efi.efiSysMountPoint = "/boot/efi";
| |
| grub.enable = true;
| |
| grub.version = 2;
| |
| grub.copyKernels = true;
| |
| grub.efiSupport = true;
| |
| grub.devices = [ "nodev" ];
| |
| grub.zfsSupport = true;
| |
| };</pre>
| |
| For multiple disks, mirror EFI system partition <code>/boot/efi</code> with <code>boot.loader.grub.mirroredBoots</code> option. See <code>man configuration.nix</code>.
| |
| ===== BIOS =====
| |
| For BIOS, delete <code>boot.loader</code> options and use GRUB:
| |
| <pre>boot.loader = {
| |
| grub.enable = true;
| |
| grub.version = 2;
| |
| grub.copyKernels = true;
| |
| grub.zfsSupport = true;
| |
| grub.devices = [ "/dev/disk/by-id/disk1" "/dev/disk/by-id/disk2" ];
| |
| };</pre>
| |
| ==== Installation ====
| |
| After finishing configuration, install system:
| |
| <pre>nixos-install --root /mnt</pre>
| |
| === Finishing installation ===
| |
| Take an initial snapshot:
| |
| <pre>zfs snapshot -r rpool/sys@install
| |
| zfs snapshot -r bpool/sys@install</pre>
| |
| Unmount EFI system partition:<pre>umount /mnt/boot/efi</pre>Also unmount mirrored ESP, if any.
| |
| | |
| Export pools:<pre>zpool export bpool
| |
| zpool export rpool</pre>
| |
| === LUKS Boot pool ===
| |
| Install cryptsetup<pre>environment.systemPackages = [ pkgs.cryptsetup ];</pre><pre>nixos-rebuild switch</pre>
| |
| Generate encryption keys<pre>mkdir /root/cryptkey.d/
| |
| chmod 700 /root/cryptkey.d/
| |
| dd bs=32 count=1 if=/dev/urandom of=/root/cryptkey.d/lukskey-bpool
| |
| dd bs=32 count=1 if=/dev/urandom of=/root/cryptkey.d/zfskey-rpool</pre>
| |
| Back up boot pool<pre>zfs snapshot -r bpool/sys@pre-luks
| |
| zfs send -R bpool/sys@pre-luks > /root/bpool-pre-luks</pre>
| |
| Back up boot pool creation command, used for recreation:<pre>zpool history bpool | head -n2 \
| |
| | grep 'zpool create' > /root/bpool-cmd</pre>
| |
| Unmount EFI system partitions<pre>umount /boot/efi</pre>
| |
| Destroy boot pool<pre>zpool destroy bpool</pre>
| |
| Enter LUKS password:<pre>LUKS_PWD=rootpool</pre>
| |
| Create LUKS1 containers<pre>for i in {/dev/disk/by-id/disk1,/dev/disk/by-id/disk2}; do
| |
| cryptsetup luksFormat -q --type luks1 $i-part2 --key-file /root/cryptkey.d/lukskey-bpool
| |
| echo $LUKS_PWD | cryptsetup luksAddKey $i-part2 --key-file /root/cryptkey.d/lukskey-bpool
| |
| cryptsetup open $i-part2 luks-bpool-$i-part2 --key-file /root/cryptkey.d/lukskey-bpool
| |
| echo luks-bpool-$i-part2 $i-part2 /root/cryptkey.d/lukskey-bpool discard >> /etc/crypttab
| |
| done</pre>
| |
| You can also convert crypttab entries to <code>boot.initrd.luks.devices</code> options.
| |
| | |
| Recreate boot pool:<pre>cat /root/bpool-cmd</pre>Remove <code>-R /mnt</code> and use devices from <code>ls -d /dev/disk/by-id/dm*</code>.
| |
| | |
| Restore boot pool:<pre>cat /root/bpool-pre-luks | zfs recv bpool/sys</pre>
| |
| | |
| Mount EFI system partition:<pre>mount /boot/efi</pre>
| |
| | |
| Optionally, if you don't want to enter 2 passwords, one for LUKS bpool, one for rpool, use embedded keyfile:<pre>zfs change-key -l \
| |
| -o keylocation=file:///root/cryptkey.d/zfskey-rpool \
| |
| -o keyformat=raw \
| |
| rpool/sys</pre>
| |
| Embed keyfile and enable LUKS GRUB:<pre>boot.initrd.secrets = {
| |
| "/root/cryptkey.d/lukskey-bpool" = "/root/cryptkey.d/lukskey-bpool";
| |
| "/root/cryptkey.d/zfskey-rpool" = "/root/cryptkey.d/zfskey-rpool";
| |
| };
| |
| boot.loader.grub = {
| |
| enableCryptodisk = true;
| |
| };</pre>
| |
| | |
| Add import bpool service<pre>systemd.services.zfs-import-bpool-mapper = {
| |
| wantedBy = [ "zfs-import.target" ];
| |
| description = "Import encrypted boot pool";
| |
| after = [ "cryptsetup.target" ];
| |
| before = [ "zfs-import-bpool.service" ];
| |
| serviceConfig = {
| |
| Type = "oneshot";
| |
| ExecStart = ''${pkgs.zfs}/bin/zpool import -d /dev/mapper bpool'';
| |
| };
| |
| };</pre>
| |
| | |
| Finally, apply changes<pre>nixos-rebuild --install-bootloader boot</pre>
| |
| Ensure <code>Installing the GRUB 2</code> is displayed, or else your system '''will not boot!''' Try again in local text console (not SSH, not terminal emulator), logged in as root.
| |
| | |
| '''Important''': After reboot, back up <code>/root/cryptkey.d/zfskey-rpool</code>, In the possible event of LUKS header corruption, data on root dataset will only be available with this key.
| |
| | |
| ==== Hibernation ====
| |
| This requires boot pool to be encrypted and a separate swap partition.
| |
| | |
| Unload swap<pre>cat /proc/swaps
| |
| # /dev/dm-0
| |
| swapoff /dev/dm-0
| |
| cryptsetup close /dev/dm-0</pre>
| |
| | |
| Identify partition<pre>ls -d /dev/disk/by-id/*-part4</pre>
| |
| | |
| Create keyfile and LUKS2 container<pre>dd bs=32 count=1 if=/dev/urandom of=/root/cryptkey.d/lukskey-cryptswap-$DISK-part4
| |
| cryptsetup luksFormat -q --type luks2 /dev/disk/by-id/disk1-part4 --key-file /root/cryptkey.d/lukskey-cryptswap-$DISK-part4
| |
| cryptsetup luksOpen /dev/disk/by-id/disk1-part4 cryptswap-$DISK-part4 --key-file /root/cryptkey.d/lukskey-cryptswap-$DISK-part4 --allow-discards
| |
| mkswap /dev/mapper/cryptswap-$DISK-part4
| |
| swapon /dev/mapper/cryptswap-$DISK-part4</pre>
| |
| | |
| Embed keyfile in initrd and configure swap, remove random swap if needed<pre>boot.initrd.secrets = {
| |
| "/root/cryptkey.d/lukskey-cryptswap-$DISK-part4" = "/root/cryptkey.d/lukskey-cryptswap-$DISK-part4";
| |
| };
| |
| swapDevices = [
| |
| {
| |
| device = "/dev/mapper/cryptswap-$DISK-part4";
| |
| }
| |
| ];
| |
| boot.initrd.luks.devices = {
| |
| cryptswap-$DISK-part4 = {
| |
| device = "$DISK-part4";
| |
| allowDiscards = true;
| |
| keyFile = "/root/cryptkey.d/lukskey-cryptswap-$DISK-part4";
| |
| };
| |
| };
| |
| boot.resumeDevice = "/dev/mapper/cryptswap-$DISK-part4";
| |
| </pre>
| |
| Apply changes<pre>nixos-rebuild boot
| |
| reboot</pre>
| |
| However some devices might not support hibernation. My laptop, EliteBook 735 G5 with AMD Ryzen 5 PRO 2500U, the screen is either black or freezes when resuming from disk. Resuming from disk works intermittently with Ubuntu 20.10. This might be a issue related to integrated graphics (Radeon Vega Mobile).
| |
| | |
| ==== Enter LUKS password in GRUB ====
| |
| GRUB will not let you retry password when the first attempt failed,
| |
| <pre>Attempting to decrypt master key...
| |
| Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
| |
| error: access denied.
| |
| error: no such cryptodisk found.
| |
| Entering rescue mode...
| |
| grub rescue></pre>
| |
| Retry password with
| |
| <pre>grub rescue> cryptomount hd0,gpt2
| |
| Attempting to decrypt master key...
| |
| Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
| |
| Slot 1 opened
| |
| grub rescue> insmod normal
| |
| grub rescue> normal</pre>
| |
| ==== Switch prefix disk ====
| |
| GRUB encodes the prefix disk in the bootloader, that is, if it fails to unlock the disk, although redundant disks are available, it will refuse to boot:<pre>error: no such cryptodisk found.
| |
| Attempting to decrypt master key...
| |
| Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a):
| |
| Slot 1 opened
| |
| error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found.
| |
| Entering rescue mode...
| |
| grub rescue></pre>
| |
| Check prefix<pre>grub rescue > set
| |
| prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/default@/grub
| |
| root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf</pre>
| |
| Change prefix<pre>grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub
| |
| grub rescue> root=crypto0</pre>
| |
| Load GRUB from other disks<pre>grub rescue> insmod normal
| |
| grub rescue> normal</pre>
| |
