ZFS: Difference between revisions
Add short command to know the difference between different disk/by-* paths
m (→boot.zfs.devNodes in virtual machines: this can happen in native NixOS installs as well) |
(Add short command to know the difference between different disk/by-* paths) |
||
(8 intermediate revisions by 4 users not shown) | |||
Line 19: | Line 19: | ||
==== boot.zfs.devNodes ==== | ==== boot.zfs.devNodes ==== | ||
If NixOS fails to import the zpool on reboot, you may need to add <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-path";</syntaxhighlight> or <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-partuuid";</syntaxhighlight> to your configuration.nix file. | If NixOS fails to import the zpool on reboot, you may need to add <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-path";</syntaxhighlight> or <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-partuuid";</syntaxhighlight> to your configuration.nix file. | ||
The differences can be tested by running <code>zpool import -d /dev/disk/by-id</code> when none of the pools are discovered, eg. a live iso. | |||
==== declarative mounting of ZFS datasets ==== | ==== declarative mounting of ZFS datasets ==== | ||
Line 47: | Line 49: | ||
Under manual partitioning [https://nixos.org/manual/nixos/stable/#sec-installation-manual-partitioning] do this instead: | Under manual partitioning [https://nixos.org/manual/nixos/stable/#sec-installation-manual-partitioning] do this instead: | ||
'''Partition your disk | '''Partition your disk with your favorite partition tool.''' | ||
We need the following partitions: | |||
* 1G for boot partition with "boot" as the partition label (also called name in some tools) and ef00 as partition code | |||
* 10G for a swap partition with "swap" as the partition label and 8200 as partition code. We will encrypt this with a random secret on each boot. | |||
* The rest of disk space for zfs with "root" as the partition label and 8300 as partition code (default code) | |||
Reason for swap partition: ZFS does use a caching mechanism that is different from the normal Linux cache infrastructure. | |||
In low-memory situations, ZFS therefore might need a bit longer to free up memory from its cache. The swap partition will help with that. | |||
Example output from fdisk: | Example output from fdisk: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
sudo gdisk /dev/nvme0n1 | |||
GPT fdisk (gdisk) version 1.0.9.1 | |||
... | |||
Command (? for help): p | |||
Disk /dev/nvme0n1: 500118192 sectors, 238.5 GiB | |||
Sector size (logical/physical): 512/512 bytes | |||
Disk identifier (GUID): CA926E8C-47F6-416A-AD1A-C2190CF5D1F8 | |||
Partition table holds up to 128 entries | |||
Main partition table begins at sector 2 and ends at sector 33 | |||
First usable sector is 34, last usable sector is 500118158 | |||
Partitions will be aligned on 2048-sector boundaries | |||
Total free space is 2669 sectors (1.3 MiB) | |||
Number Start (sector) End (sector) Size Code Name | |||
1 2048 2099199 1024.0 MiB EF00 boot | |||
2 2099200 23070719 10.0 GiB 8200 swap | |||
3 23070720 500117503 227.5 GiB 8300 root | |||
Command (? for help): | |||
</syntaxhighlight> | </syntaxhighlight> | ||
'''Make zfs pool with encryption and mount points:''' | '''Make zfs pool with encryption and mount points:''' | ||
Line 70: | Line 88: | ||
'''Note:''' zpool config can significantly affect performance (especially the ashift option) so you may want to do some research. The [https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ ZFS tuning cheatsheet] or [https://wiki.archlinux.org/title/ZFS#Storage_pools ArchWiki] is a good place to start. | '''Note:''' zpool config can significantly affect performance (especially the ashift option) so you may want to do some research. The [https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ ZFS tuning cheatsheet] or [https://wiki.archlinux.org/title/ZFS#Storage_pools ArchWiki] is a good place to start. | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
zpool create -O encryption=on -O keyformat=passphrase -O keylocation=prompt -O compression= | zpool create -O encryption=on -O keyformat=passphrase -O keylocation=prompt -O compression=zstd -O mountpoint=none -O xattr=sa -O acltype=posixacl -o ashift=12 zpool /dev/nvme0n1p2 | ||
zfs create | zfs create zpool/root | ||
zfs create | zfs create zpool/nix | ||
zfs create | zfs create zpool/var | ||
zfs create | zfs create zpool/home | ||
mkdir /mnt | mkdir -p /mnt | ||
mount -t zfs zpool/root /mnt | mount -t zfs zpool/root /mnt -o zfsutil | ||
mkdir /mnt/nix /mnt/var /mnt/home | mkdir /mnt/nix /mnt/var /mnt/home | ||
mount -t zfs zpool/nix /mnt/nix | mount -t zfs zpool/nix /mnt/nix -o zfsutil | ||
mount -t zfs zpool/var /mnt/var | mount -t zfs zpool/var /mnt/var -o zfsutil | ||
mount -t zfs zpool/home /mnt/home | mount -t zfs zpool/home /mnt/home -o zfsutil | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 104: | Line 122: | ||
mkfs.fat -F 32 -n boot /dev/nvme0n1p1 | mkfs.fat -F 32 -n boot /dev/nvme0n1p1 | ||
</syntaxhighlight> | </syntaxhighlight> | ||
'''Installation:''' | '''Installation:''' | ||
Line 113: | Line 132: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
mkdir -p /mnt/boot | mkdir -p /mnt/boot | ||
mount /dev/disk/by- | mount /dev/disk/by-partlabel/boot /mnt/boot | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 123: | Line 142: | ||
{ | { | ||
# Boot loader config for configuration.nix: | # Boot loader config for configuration.nix: | ||
boot.loader. | boot.loader.systemd-boot.enable = true; | ||
# for local disks that are not shared over the network, we don't need this to be random | |||
networking.hostId = "8425e349"; | |||
fileSystems."/" = { | |||
device = "zpool/root"; | |||
]; | fsType = "zfs"; | ||
# the zfsutil option is needed when mounting zfs datasets without "legacy" mountpoints | |||
options = [ "zfsutil" ]; | |||
}; | }; | ||
fileSystems."/" = | fileSystems."/nix" = { | ||
device = "zpool/nix"; | |||
fsType = "zfs"; | |||
}; | options = [ "zfsutil" ]; | ||
}; | |||
fileSystems."/ | fileSystems."/var" = { | ||
device = "zpool/var"; | |||
fsType = "zfs"; | |||
}; | options = [ "zfsutil" ]; | ||
}; | |||
fileSystems."/ | fileSystems."/home" = { | ||
device = "zpool/home"; | |||
fsType = "zfs"; | |||
}; | options = [ "zfsutil" ]; | ||
}; | |||
fileSystems."/ | fileSystems."/boot" = { | ||
device = "/dev/disk/by-partlabel/boot"; | |||
fsType = "vfat"; | |||
}; | |||
swapDevices = [{ | |||
device = "/dev/disk/by-partlabel/swap"; | |||
randomEncryption = true; | |||
}]; | |||
} | } | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Importing on boot == | == Importing on boot == | ||
Line 183: | Line 203: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Zpool created with bus-based disk names === | |||
If you used bus-based disk names in the <syntaxhighlight inline>zpool create</syntaxhighlight> command, e.g., <syntaxhighlight inline>/dev/sda</syntaxhighlight>, NixOS may run into issues importing the pool if the names change. Even if the pool is able to be mounted (with <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-partuuid";</syntaxhighlight> set), this may manifest as a <syntaxhighlight inline>FAULTED</syntaxhighlight> disk and a <syntaxhighlight inline>DEGRADED</syntaxhighlight> pool reported by <syntaxhighlight inline>zpool status</syntaxhighlight>. The fix is to re-import the pool using disk IDs: | If you used bus-based disk names in the <syntaxhighlight inline>zpool create</syntaxhighlight> command, e.g., <syntaxhighlight inline>/dev/sda</syntaxhighlight>, NixOS may run into issues importing the pool if the names change. Even if the pool is able to be mounted (with <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-partuuid";</syntaxhighlight> set), this may manifest as a <syntaxhighlight inline>FAULTED</syntaxhighlight> disk and a <syntaxhighlight inline>DEGRADED</syntaxhighlight> pool reported by <syntaxhighlight inline>zpool status</syntaxhighlight>. The fix is to re-import the pool using disk IDs: | ||
Line 190: | Line 211: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
The import setting is reflected in <syntaxhighlight inline>/etc/zfs/ | The import setting is reflected in <syntaxhighlight inline="" lang="bash">/etc/zfs/zpool.cache</syntaxhighlight>, so it should persist through subsequent boots. | ||
=== Zpool created with disk IDs === | |||
If you used disk IDs to refer to disks in the <code>zpool create</code> command, e.g., <code>/dev/disk/by-id</code>, then NixOS may consistently fail to import the pool unless <code>boot.zfs.devNodes = "/dev/disk/by-id"</code> is also set. | |||
== Mount datasets at boot == | == Mount datasets at boot == |