ZFS: Difference between revisions

[https://zfsonlinux.org/ {{PAGENAME}}] ([[wikipedia:en:{{PAGENAME}}]]), also known as [https://openzfs.org/ OpenZFS] ([[wikipedia:en:OpenZFS]]), is a modern filesystem which is well supported on [[NixOS]].

Besides the {{nixos:package|zfs}} package (''ZFS Filesystem Linux Kernel module'') itself, there are many packages in the ZFS ecosystem available.

==== Latest Kernel compatible with ZFS ====

If your config specifies a Kernel version that is not officially supported by upstream ZFS, the ZFS module will fail to evaluate with an error that the ZFS package is "broken". Upstream ZFS changed in 2.3 to refuse to build by default, regardless of Nixpkgs’ broken marking (or ignoring).  

===== Selecting the latest ZFS-compatible Kernel =====

{{Warning|This will often result in the Kernel version going backwards as Kernel versions become end-of-life and are removed from Nixpkgs. If you need more control over the Kernel version due to hardware requirements, consider simply pinning a specific version rather than calculating it as below.}}

To use the latest ZFS-compatible Kernel currently available, the following configuration may be used.

<syntaxhighlight lang="nix">

  ...

}:

  zfsCompatibleKernelPackages = lib.filterAttrs (

    (builtins.match "linux_[0-9]+_[0-9]+" name) != null

  boot.kernelPackages = latestKernelPackage;

 

 

 

 

 

 

 

ZFS will manage mounting non-legacy ZFS filesystems, but NixOS tries to manage mounting with systemd. ZFS native mountpoints are not managed as part of the system configuration (but better support hibernation with a separate swap partition). This can lead to conflicts if the ZFS mount service is also enabled for the same datasets.

 

 

 

 

 

 

 

 

* 4G for a swap partition with "swap" as the partition label and 8200 as partition code. We will encrypt this with a random secret on each boot.

* The rest of disk space for zfs with "root" as the partition label and 8300 as partition code (default code)

 

Reason for swap partition: ZFS does use a caching mechanism that is different from the normal Linux cache infrastructure.

In low-memory situations, ZFS therefore might need a bit longer to free up memory from its cache. The swap partition will help with that.

 

Example with gdisk using <code>/dev/nvme0n1</code> as the device (use <code>lsblk</code> to find the device</code>):

Partition number (1-128, default 1):

 

 

 

 

 

 

 

{{note|It is often recommended to specify the drive using the device ID/UUID to prevent incorrect configuration, but it is also possible to use the device name (e.g. /dev/sda). See also: [[#Zpool created with bus-based disk names]], [https://wiki.archlinux.org/title/Persistent_block_device_naming Persistent block device naming - ArchWiki]}}

 

{{Note|zpool config can significantly affect performance (especially the ashift option) so you may want to do some research. The ZFS tuning cheatsheet or ArchWiki is a good place to start.}}

zpool create -O encryption=on -O keyformat=passphrase -O keylocation=prompt -O compression=zstd -O mountpoint=none -O xattr=sa -O acltype=posixacl -o ashift=12 zpool $DISK

# enter the password to decrypt the pool at boot

Re-enter new passphrase:

# Create datasets

zfs create zpool/root

zfs create zpool/nix

zfs create zpool/var

mkdir -p /mnt

 

Output from <syntaxhighlight lang="bash" inline>zpool status</syntaxhighlight>:

<syntaxhighlight >

 

  nvme-eui.0025384b21406566-part2  ONLINE      0    0    0

 

</syntaxhighlight>

 

==== Format boot partition and enable swap ====

<syntaxhighlight lang="bash">

mkfs.fat -F 32 -n boot $BOOT

</syntaxhighlight>

<syntaxhighlight lang="bash">

==== Installation ====

writing /mnt/etc/nixos/hardware-configuration.nix...

writing /mnt/etc/nixos/configuration.nix...

For more hardware-specific settings, see https://github.com/NixOS/nixos-hardware.

Now edit the configuration.nix that was just created in <code>/mnt/etc/nixos/configuration.nix</code> and make sure to have at least the following content in it.

 

 

  # for local disks that are not shared over the network, we don't need this to be random

  # without this, "ZFS requires networking.hostId to be set" will be raised

 

Now check the hardware-configuration.nix in <code>/mnt/etc/nixos/hardware-configuration.nix</code> and add whats missing e.g. <code>options = [ "zfsutil" ]</code> for all filesystems except boot and <code>randomEncryption = true;</code> for the swap partition. Also change the generated swap device to the partition we created e.g. <code>/dev/disk/by-id/nvme-SKHynix_HFS512GDE9X081N_FNB6N634510106K5O-part2</code> in this case and <code>/dev/disk/by-id/nvme-SKHynix_HFS512GDE9X081N_FNB6N634510106K5O-part1</code> for boot.

 

 

 

 

  fileSystems."/home" = {

  fileSystems."/boot" = {

  device = "/dev/disk/by-id/nvme-SKHynix_HFS512GDE9X081N_FNB6N634510106K5O-part1";

  fsType = "vfat";

+    device = "/dev/disk/by-id/nvme-SKHynix_HFS512GDE9X081N_FNB6N634510106K5O-part2";

+    randomEncryption = true;

Now you may install NixOS with <code>nixos-install</code>.

If you create a zpool, it will not be imported on the next boot unless you either add the zpool name to <syntaxhighlight lang="nix" inline>boot.zfs.extraPools</syntaxhighlight>:

boot.zfs.extraPools = [ "zpool_name" ];

or if you are using legacy mountpoints, add a <syntaxhighlight lang="nix" inline>fileSystems</syntaxhighlight> entry and NixOS will automatically detect that the pool needs to be imported:

## In /etc/nixos/configuration.nix:

fileSystems."/mount/point" = {

   device = "zpool_name";

   fsType = "zfs";

If you used bus-based disk names in the <syntaxhighlight inline>zpool create</syntaxhighlight> command, e.g., <syntaxhighlight inline>/dev/sda</syntaxhighlight>, NixOS may run into issues importing the pool if the names change. Even if the pool is able to be mounted (with <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-partuuid";</syntaxhighlight> set), this may manifest as a <syntaxhighlight inline>FAULTED</syntaxhighlight> disk and a <syntaxhighlight inline>DEGRADED</syntaxhighlight> pool reported by <syntaxhighlight inline>zpool status</syntaxhighlight>. The fix is to re-import the pool using disk IDs:

<syntaxhighlight>

# zpool import -d /dev/disk/by-id zpool_name

The import setting is reflected in <syntaxhighlight inline="" lang="bash">/etc/zfs/zpool.cache</syntaxhighlight>, so it should persist through subsequent boots.  

=== Zpool created with disk IDs ===

If you used disk IDs to refer to disks in the <code>zpool create</code> command, e.g., <code>/dev/disk/by-id</code>, then NixOS may consistently fail to import the pool unless <code>boot.zfs.devNodes = "/dev/disk/by-id"</code> is also set.

zfs-mount service is enabled by default on NixOS 22.05.

To automatically mount a dataset at boot, you only need to set <code>canmount=on</code> and <code>mountpoint=/mount/point</code> on the respective datasets.

== Changing the Adaptive Replacement Cache size ==

To tune other attributes of ARC, L2ARC or of ZFS itself via runtime modprobe config, add this to your NixOS configuration (keys and values are examples only!):

    boot.extraModprobeConfig = ''

      options zfs l2arc_noprefetch=0 l2arc_write_boost=33554432 l2arc_write_max=16777216 zfs_arc_max=2147483648

</syntaxhighlight>

You can confirm whether any specified configuration/tuning got applied via commands like <code>arc_summary</code> and <code>arcstat -a -s " "</code>.

== Automatic scrubbing ==

<syntaxhighlight lang="nix">

services.zfs.autoScrub.enable = true;

</syntaxhighlight>

You can tweak the interval (defaults to once a week) and which pools should be scrubbed (defaults to all).

{{note|As of 22.05, rebuilding your config with the below directions may result in a situation where, if you want to revert the changes, you may need to do some pretty hairy nix-store manipulation to be able to successfully rebuild, see https://github.com/NixOS/nixpkgs/issues/101462#issuecomment-1172926129}}

In case you want unlock a machine remotely (after an update), having an ssh service in initrd for the password prompt is handy:

    # https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt

      # hostKeys paths must be unquoted strings, otherwise you'll run into issues with boot.initrd.secrets

      hostKeys = [ /path/to/ssh_host_rsa_key ];

</syntaxhighlight>

* In order to use DHCP in the initrd, network manager must not be enabled and <syntaxhighlight lang="nix" inline>networking.useDHCP = true;</syntaxhighlight> must be set.

boot.kernelModules = [ "r8169" ];

After that you can unlock your datasets using the following ssh command:

</syntaxhighlight>

Alternatively you could also add the commands as postCommands to your configuration.nix, then you just have to ssh into the initrd:

   initrd.network = {

     postCommands = ''

    echo "zfs load-key -a; killall zfs" >> /root/.profile

    '';

  };

};

</syntaxhighlight>

On ZFS, the performance will deteriorate significantly when more than 80% of the available space is used.  To avoid this, reserve disk space beforehand.

To reserve space create a new unused dataset that gets a guaranteed disk space of 10GB.

# zfs create -o refreservation=10G -o mountpoint=none zroot/reserved

== Auto ZFS trimming ==

 

 

See {{nixos:option|services.zfs.autoSnapshot}} or {{nixos:option|services.sanoid}} section in <code>man configuration.nix</code>.

With <code>sharenfs</code> property, ZFS has build-in support for generating <code>/etc/exports.d/zfs.exports</code> file, which in turn is processed by NFS service automatically.

{{warning|If you are intending on defining an IPv6 subnet as part of your sharenfs rule, as of ZFS 2.0.6 (2021-09-23) please note that due to a bug in openzfs '''your rule will not correctly apply''', and may result in a security vulnerability (CVE-2013-20001). A fix has been implemented in the next yet-to-be-released upstream version - [https://github.com/openzfs/zfs/pull/11939 openzfs/zfs#11939]}}

{{warning|<code>zfs share</code> or <code>sharenfs</code> does not work if the <code>mountpoint</code> is set to <code>legacy</code> (or <code>none</code>, of course). I was unable to find a source for this behaviour, but I was stuck on the problem for days, until I realized the problem.  ::Reply: sharenfs controlls what

is written into <code>/etc/exports</code>. If ZFS does not know the mountpoint, as is the case in

Then, set <code>sharenfs</code> property:

<syntaxhighlight lang="console">

zfs set sharenfs="ro=192.168.1.0/24,all_squash,anonuid=70,anongid=70" rpool/myData

</syntaxhighlight>

For more options, see <code>man 5 exports</code>.

== Mail notifications (ZFS Event Daemon) ==

ZFS Event Daemon (zed) monitors events generated by the ZFS Kernel module and runs configured tasks. It can be configured to send an email when a pool scrub is finished or a disk has failed. [https://search.nixos.org/options?query=services.zfs.zed zed options]

<syntaxhighlight lang="nix">

  age.secrets.msmtp = {

    file = "${inputs.self.outPath}/secrets/msmtp.age";

  # for zed enableMail, enable sendmailSetuidWrapper

  services.mail.sendmailSetuidWrapper.enable = true;

  programs.msmtp = {

        host = "smtp.mail.example.com";

        user = "myname@example.com";

        from = "myname@example.com";

Then, configure an alias for root account. With this alias configured, all mails sent to root, such as cron job results and failed sudo login events, will be redirected to the configured email account.

  environment.etc.aliases.text = ''

    root: admin@example.com

</syntaxhighlight>

  services.zfs.zed. = {

<syntaxhighlight lang="console">