Jump to content

ZFS: Difference between revisions

ZFS (view source)

Revision as of 02:31, 2 March 2022

689 bytes added ,  2 March 2022
fix some syntax highlighting and formatting
VisualWikitext
imported>Stites
m (${disk} -> ${disk}-part1 for single-disk instructions of creating a zpool)
imported>Wackbyte
(fix some syntax highlighting and formatting)
Line 26: Line 26:
* (ZFS, unrelated to Nix- see https://github.com/openzfs/zfs/issues/7734) You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure.
* (ZFS, unrelated to Nix- see https://github.com/openzfs/zfs/issues/7734) You shouldn't use a ZVol as a swap device, as it can deadlock under memory pressure.
* You should set the <code>mountpoint</code> property of your ZFS filesystems to be <code>legacy</code> and let NixOS mount them like any other filesystem (such as ''ext4'' or ''btrfs''), otherwise some filesystems may fail to mount due to ordering issues.
* You should set the <code>mountpoint</code> property of your ZFS filesystems to be <code>legacy</code> and let NixOS mount them like any other filesystem (such as ''ext4'' or ''btrfs''), otherwise some filesystems may fail to mount due to ordering issues.
* By default, all ZFS pools available to the system will be forcibly imported during boot, regardless if you had imported them before or not. You should be careful not to have any other system accessing them at the same time, otherwise it will corrupt your pools. Normally (for the common desktop user) this should not be a problem, as a hard disk is usually only directly connected to one machine. This behaviour can be disabled by setting <code>boot.zfs.forceImportAll = false</code>.
* By default, all ZFS pools available to the system will be forcibly imported during boot, regardless if you had imported them before or not. You should be careful not to have any other system accessing them at the same time, otherwise it will corrupt your pools. Normally (for the common desktop user) this should not be a problem, as a hard disk is usually only directly connected to one machine. This behaviour can be disabled by setting <syntaxhighlight lang="nix" inline>boot.zfs.forceImportAll = false;</syntaxhighlight>.
* If you create a zpool in the installer, make sure you run `zpool export <pool name>` after `nixos-install`, or else when you reboot into your new system, zfs will fail to import the zpool.
* If you create a zpool in the installer, make sure you run <code>zpool export <pool name></code> after <code>nixos-install</code>, or else when you reboot into your new system, zfs will fail to import the zpool.
* If you are running within a VM and NixOS fails to import the zpool on reboot, you may need to add <code>boot.zfs.devNodes = "/dev/disk/by-path";</code> to your configuration.nix file.
* If you are running within a VM and NixOS fails to import the zpool on reboot, you may need to add <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-path";</syntaxhighlight> to your configuration.nix file.




Line 37: Line 37:
Just add the following to your <code>configuration.nix</code> file:
Just add the following to your <code>configuration.nix</code> file:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
boot.initrd.supportedFilesystems = ["zfs"]; # boot from zfs
boot.initrd.supportedFilesystems = [ "zfs" ]; # boot from zfs
boot.supportedFilesystems = [ "zfs" ];
boot.supportedFilesystems = [ "zfs" ];
services.udev.extraRules = ''
services.udev.extraRules = ''
Line 47: Line 47:


To activate the configuration and load the ZFS kernel module, run:
To activate the configuration and load the ZFS kernel module, run:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
nixos-rebuild switch
# nixos-rebuild switch
</syntaxhighlight>
</syntaxhighlight>


Line 55: Line 55:
If you want NixOS to auto-mount your ZFS filesystems during boot, you should set their <code>mountpoint</code> property to <code>legacy</code> and treat it like if it were any other filesystem, i.e.: mount the filesystem manually and regenerate your list of filesystems, as such:
If you want NixOS to auto-mount your ZFS filesystems during boot, you should set their <code>mountpoint</code> property to <code>legacy</code> and treat it like if it were any other filesystem, i.e.: mount the filesystem manually and regenerate your list of filesystems, as such:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
zfs set mountpoint=legacy <pool>/<fs>
# zfs set mountpoint=legacy <pool>/<fs>
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
mount -t zfs <pool>/<fs> <mountpoint>
# mount -t zfs <pool>/<fs> <mountpoint>
</syntaxhighlight>
</syntaxhighlight>


This will regenerate your /etc/nixos/hardware-configuration.nix file:
This will regenerate your /etc/nixos/hardware-configuration.nix file:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
nixos-generate-config
# nixos-generate-config
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
nixos-rebuild switch
# nixos-rebuild switch
</syntaxhighlight>
</syntaxhighlight>


Line 76: Line 76:
The <code>nixos-generate-config</code> command regenerates your <code>/etc/nixos/hardware-configuration.nix</code> file, which includes the list of filesystems for NixOS to mount during boot, e.g.:
The <code>nixos-generate-config</code> command regenerates your <code>/etc/nixos/hardware-configuration.nix</code> file, which includes the list of filesystems for NixOS to mount during boot, e.g.:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  fileSystems."/home" =
fileSystems."/home" =
    { device = "rpool/home";
  { device = "rpool/home";
      fsType = "zfs";
    fsType = "zfs";
    };
  };


  fileSystems."/backup" =
fileSystems."/backup" =
    { device = "rpool/backup";
  { device = "rpool/backup";
      fsType = "zfs";
    fsType = "zfs";
    };
  };
</syntaxhighlight>
</syntaxhighlight>


Alternatively, if you do not mind maintaining some manual tweaks to your Nix hardware configuration, you can avoid using the ZFS legacy mounting option if you add <code>options = [ "zfsutil" ];</code> to your filesystem definitions.  e.g. the above would become.
Alternatively, if you do not mind maintaining some manual tweaks to your Nix hardware configuration, you can avoid using the ZFS legacy mounting option if you add <syntaxhighlight lang="nix" inline>options = [ "zfsutil" ];</syntaxhighlight> to your filesystem definitions.  e.g. the above would become.
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  fileSystems."/home" =
fileSystems."/home" =
    { device = "rpool/home";
  { device = "rpool/home";
      fsType = "zfs";
    fsType = "zfs";
      options = [ "zfsutil" ];
    options = [ "zfsutil" ];
    };
  };


  fileSystems."/backup" =
fileSystems."/backup" =
    { device = "rpool/backup";
  { device = "rpool/backup";
      fsType = "zfs";
    fsType = "zfs";
      options = [ "zfsutil" ];
    options = [ "zfsutil" ];
    };
  };
</syntaxhighlight>
</syntaxhighlight>


Keep your filesystem defintions in a file separate from <code>/etc/nixos/hardware-configuration.nix</code>, since this is overwritten whenever you run <code>nixos-generate-config</code>.
Keep your filesystem defintions in a file separate from <code>/etc/nixos/hardware-configuration.nix</code>, since this is overwritten whenever you run <code>nixos-generate-config</code>.


== Changing the Cache Size ==
== Changing the cache size ==


ZFS has a complicated cache system.  The cache you're most likely to want to fiddle with is the called Adaptive Replacement Cache, usually abbreviated ARC.  This is the first-level (fastest) of ZFS's caches.
ZFS has a complicated cache system.  The cache you're most likely to want to fiddle with is the called Adaptive Replacement Cache, usually abbreviated ARC.  This is the first-level (fastest) of ZFS's caches.
Line 112: Line 112:
To change the maximum size of the ARC cache to (for example) 12 GB, add this to your NixOS configuration:
To change the maximum size of the ARC cache to (for example) 12 GB, add this to your NixOS configuration:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
boot.kernelParams = ["zfs.zfs_arc_max=12884901888"];
boot.kernelParams = [ "zfs.zfs_arc_max=12884901888" ];
</syntaxhighlight>
</syntaxhighlight>


In some versions of ZFS, you can change the maximum size of the ARC on the fly, but in NixOS 18.03 this is not possible.  (Nor is it possible in other versions of ZFS on Linux yet, according to Stack Exchange.)
In some versions of ZFS, you can change the maximum size of the ARC on the fly, but in NixOS 18.03 this is not possible.  (Nor is it possible in other versions of ZFS on Linux yet, according to Stack Exchange.)


== Automatic Scrubbing ==
== Automatic scrubbing ==


Regular scrubbing of ZFS pools is recommended and can be enabled in your NixOS configuration via:
Regular scrubbing of ZFS pools is recommended and can be enabled in your NixOS configuration via:
Line 132: Line 132:
To reserve space create a new unused dataset that gets a guaranteed disk space of 1GB.
To reserve space create a new unused dataset that gets a guaranteed disk space of 1GB.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
zfs create -o refreservation=1G -o mountpoint=none zroot/reserved
# zfs create -o refreservation=1G -o mountpoint=none zroot/reserved
</syntaxhighlight>
</syntaxhighlight>


Line 139: Line 139:
The dataset itself should not be used. In case you would run out of space you can shrink the reservation to reclaim enough disk space to cleanup the other data from the pool:
The dataset itself should not be used. In case you would run out of space you can shrink the reservation to reclaim enough disk space to cleanup the other data from the pool:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
zfs set refreservation=none zroot/reserved
# zfs set refreservation=none zroot/reserved
</syntaxhighlight>
</syntaxhighlight>


Line 147: Line 147:
To auto-snapshot a ZFS filesystem or a ZVol, set its <code>com.sun:auto-snapshot</code> property to <code>true</code>, like this:
To auto-snapshot a ZFS filesystem or a ZVol, set its <code>com.sun:auto-snapshot</code> property to <code>true</code>, like this:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
zfs set com.sun:auto-snapshot=true <pool>/<fs>
# zfs set com.sun:auto-snapshot=true <pool>/<fs>
</syntaxhighlight>
</syntaxhighlight>


Line 173: Line 173:
You can also disable a given type of snapshots on a per-dataset basis by setting a ZFS property, like this:
You can also disable a given type of snapshots on a per-dataset basis by setting a ZFS property, like this:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
zfs set com.sun:auto-snapshot:weekly=false <pool>/<fs>
# zfs set com.sun:auto-snapshot:weekly=false <pool>/<fs>
</syntaxhighlight>
</syntaxhighlight>


Line 188: Line 188:
This guide is based on the above OpenZFS guide and the NixOS installation instructions in the [https://nixos.org/manual/nixos/stable/index.html#sec-installation NixOS manual].
This guide is based on the above OpenZFS guide and the NixOS installation instructions in the [https://nixos.org/manual/nixos/stable/index.html#sec-installation NixOS manual].


=== Pool Layout Considerations ===
=== Pool layout considerations ===


it is important to keep <code>/nix</code> and the rest of the filesystem in
it is important to keep <code>/nix</code> and the rest of the filesystem in
different sections of the dataset hierarchy, like this:
different sections of the dataset hierarchy, like this:


<syntaxhighlight lang="none">
<syntaxhighlight lang="text">
rpool/
rpool/
       nixos/
       nixos/
Line 203: Line 203:
</syntaxhighlight>
</syntaxhighlight>


the name of `nixos` and `userdata/` can change, but them being peers is important.
the name of <code>nixos</code> and <code>userdata/</code> can change, but them being peers is important.


ZFS can take consistent and atomic snapshots recursively down a dataset's hierarchy. Since Nix is good at being Nix, most users will want their server's ''data'' backed up, and don't mind reinstalling NixOS and then restoring data. If this is sufficient, only snapshot and back up the <code>userdata</code> hierarchy. Users who want to be able to restore a service with only ZFS snapshots will want to snapshot the entire tree, at the significant expense of snapshotting the Nix store.
ZFS can take consistent and atomic snapshots recursively down a dataset's hierarchy. Since Nix is good at being Nix, most users will want their server's ''data'' backed up, and don't mind reinstalling NixOS and then restoring data. If this is sufficient, only snapshot and back up the <code>userdata</code> hierarchy. Users who want to be able to restore a service with only ZFS snapshots will want to snapshot the entire tree, at the significant expense of snapshotting the Nix store.


=== Dataset Properties ===
=== Dataset properties ===


The following is a list of recommended dataset properties which have no drawbacks under regular uses:
The following is a list of recommended dataset properties which have no drawbacks under regular uses:
Line 227: Line 227:


<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ zpool create  -O xattr=sa -O acltype=posixacl rpool ...
# zpool create  -O xattr=sa -O acltype=posixacl rpool ...
</syntaxhighlight>
</syntaxhighlight>


or:
or:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ zfs create -o xattr=sa -o acltype=posixacl rpool/root
# zfs create -o xattr=sa -o acltype=posixacl rpool/root
</syntaxhighlight>
</syntaxhighlight>


Line 238: Line 238:


<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ zfs set xattr=sa acltype=posixacl rpool/root
# zfs set xattr=sa acltype=posixacl rpool/root
</syntaxhighlight>
</syntaxhighlight>


=== Environment Setup ===
=== Environment setup ===
For convenience set a shell variable with the paths to your disk(s):
For convenience set a shell variable with the paths to your disk(s):


Line 281: Line 281:
=== Laying out the filesystem hierarchy ===
=== Laying out the filesystem hierarchy ===
In this guide, we will be using a <code>tmpfs</code> for <code>/</code>, since no system state will be stored outside of the ZFS datasets we will create.
In this guide, we will be using a <code>tmpfs</code> for <code>/</code>, since no system state will be stored outside of the ZFS datasets we will create.
<syntaxhighlight lang="console">
<syntaxhighlight lang="bash">
$ sudo mount -t tmpfs none /mnt
sudo mount -t tmpfs none /mnt
</syntaxhighlight>
</syntaxhighlight>


==== Create the ZFS pool ====
==== Create the ZFS pool ====
<syntaxhighlight lang="console">
<syntaxhighlight lang="bash">
$ sudo zpool create \
sudo zpool create \
   -o ashift=12 \
   -o ashift=12 \
   -o autotrim=on \
   -o autotrim=on \
Line 313: Line 313:
==== Create the ZFS datasets ====
==== Create the ZFS datasets ====
Since zfs is a copy-on-write filesystem even for deleting files disk space is needed. Therefore it should be avoided to run out of disk space. Luckily it is possible to reserve disk space for datasets to prevent this.  
Since zfs is a copy-on-write filesystem even for deleting files disk space is needed. Therefore it should be avoided to run out of disk space. Luckily it is possible to reserve disk space for datasets to prevent this.  
<syntaxhighlight lang="bash>
<syntaxhighlight lang="console">
sudo zfs create -o refreservation=1G -o mountpoint=none rpool/reserved
# zfs create -o refreservation=1G -o mountpoint=none rpool/reserved
</syntaxhighlight>
</syntaxhighlight>


Create the datasets for the operating system.
Create the datasets for the operating system.
<syntaxhighlight lang="bash>
<syntaxhighlight lang="bash">
sudo zfs create -o canmount=off -o mountpoint=/ rpool/nixos
sudo zfs create -o canmount=off -o mountpoint=/ rpool/nixos
sudo zfs create -o canmount=on rpool/nixos/nix
sudo zfs create -o canmount=on rpool/nixos/nix
Line 329: Line 329:


Create datasets for user home directories.  If you opted to not encrypt the entire pool, you can encrypt just the userdata by specifying the same ZFS properties when creating rpool/userdata, and the child datasets will also be encrypted.
Create datasets for user home directories.  If you opted to not encrypt the entire pool, you can encrypt just the userdata by specifying the same ZFS properties when creating rpool/userdata, and the child datasets will also be encrypted.
<syntaxhighlight lang="bash>
<syntaxhighlight lang="bash">
sudo zfs create -o canmount=off -o mountpoint=/ rpool/userdata
sudo zfs create -o canmount=off -o mountpoint=/ rpool/userdata
sudo zfs create -o canmount=on rpool/userdata/home
sudo zfs create -o canmount=on rpool/userdata/home
Line 339: Line 339:
</syntaxhighlight>
</syntaxhighlight>


==== Mount /boot ====
==== Mount <code>/boot</code> ====
We are going to use the default NixOS bootloader systemd-boot, which can install to only one device.  You will want to periodically rsync <code>/mnt/boot</code> to <code>/mnt/boot2</code> so that you can always boot your system if either disk fails.
We are going to use the default NixOS bootloader systemd-boot, which can install to only one device.  You will want to periodically rsync <code>/mnt/boot</code> to <code>/mnt/boot2</code> so that you can always boot your system if either disk fails.
<syntaxhighlight lang="bash>
<syntaxhighlight lang="bash">
sudo mkdir /mnt/boot /mnt/boot2
sudo mkdir /mnt/boot /mnt/boot2
sudo mount "${disk[0]}-part3" /mnt/boot
sudo mount "${disk[0]}-part3" /mnt/boot
Line 348: Line 348:


Or for single-disk systems:
Or for single-disk systems:
<syntaxhighlight lang="bash>
<syntaxhighlight lang="bash">
sudo mkdir /mnt/boot
sudo mkdir /mnt/boot
sudo mount "${disk}-part3" /mnt/boot
sudo mount "${disk}-part3" /mnt/boot
Line 355: Line 355:
=== Configure the NixOS system ===
=== Configure the NixOS system ===
Generate the base NixOS configuration files.
Generate the base NixOS configuration files.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ nixos-generate-config --root /mnt
# nixos-generate-config --root /mnt
</syntaxhighlight>
</syntaxhighlight>


Line 363: Line 363:
Now Add the following block of code anywhere (how you organise your <code>configuration.nix</code> is up to you):
Now Add the following block of code anywhere (how you organise your <code>configuration.nix</code> is up to you):
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  # ZFS boot settings.
# ZFS boot settings.
  boot.supportedFilesystems = [ "zfs" ];
boot.supportedFilesystems = [ "zfs" ];
  boot.zfs.devNodes = "/dev/";
boot.zfs.devNodes = "/dev/";
</syntaxhighlight>
</syntaxhighlight>


Line 379: Line 379:
Now add some ZFS maintenance settings:
Now add some ZFS maintenance settings:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  # ZFS maintenance settings.
# ZFS maintenance settings.
  services.zfs.trim.enable = true;
services.zfs.trim.enable = true;
  services.zfs.autoScrub.enable = true;
services.zfs.autoScrub.enable = true;
  services.zfs.autoScrub.pools = [ "rpool" ];
services.zfs.autoScrub.pools = [ "rpool" ];
</syntaxhighlight>
</syntaxhighlight>


You may wish to also add <code>services.zfs.autoSnapshot.enable = true;</code> and set the ZFS property <code>com.sun:auto-snapshot</code> to <code>true</code> on <code>rpool/userdata</code> to have automatic snapshots.  (See [[#How to use the auto-snapshotting service]] earlier on this page.)
You may wish to also add <syntaxhighlight lang="nix" inline>services.zfs.autoSnapshot.enable = true;</syntaxhighlight> and set the ZFS property <code>com.sun:auto-snapshot</code> to <code>true</code> on <code>rpool/userdata</code> to have automatic snapshots.  (See [[#How to use the auto-snapshotting service]] earlier on this page.)


Now open <code>/mnt/etc/nixos/hardware-configuration-zfs.nix</code>.
Now open <code>/mnt/etc/nixos/hardware-configuration-zfs.nix</code>.


* Add <code>options = [ "zfsutil" ];</code> to every ZFS <code>fileSystems</code> block.
* Add <syntaxhighlight lang="nix" inline>options = [ "zfsutil" ];</syntaxhighlight> to every ZFS <code>fileSystems</code> block.
* Add <code>options = [ "X-mount.mkdir" ];</code> to <code>fileSystems."/boot"</code> and <code>fileSystems."/boot2"</code>.
* Add <syntaxhighlight lang="nix" inline>options = [ "X-mount.mkdir" ];</syntaxhighlight> to <syntaxhighlight lang="nix" inline>fileSystems."/boot"</syntaxhighlight> and <syntaxhighlight lang="nix" inline>fileSystems."/boot2"</syntaxhighlight>.
* Replace <code>swapDevices</code> with the following, replacing <code>DISK1</code> and <code>DISK2</code> with the names of your disks.
* Replace <code>swapDevices</code> with the following, replacing <code>DISK1</code> and <code>DISK2</code> with the names of your disks.


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  swapDevices = [
swapDevices = [
    { device = "/dev/disk/by-id/foo-part2";
  { device = "/dev/disk/by-id/foo-part2";
      randomEncryption = true;
    randomEncryption = true;
    }
  }
    { device = "/dev/disk/by-id/bar-part2";
  { device = "/dev/disk/by-id/bar-part2";
      randomEncryption = true;
    randomEncryption = true;
    }
  }
  ];
];
</syntaxhighlight>
</syntaxhighlight>
For single-disk installs, remove the second entry of this array.
For single-disk installs, remove the second entry of this array.
Line 419: Line 419:
     enable = true;
     enable = true;
     ssh = {
     ssh = {
      enable = true;
      enable = true;
      # To prevent ssh clients from freaking out because a different host key is used,
      # To prevent ssh clients from freaking out because a different host key is used,
      # a different port for ssh is useful (assuming the same host has also a regular sshd running)
      # a different port for ssh is useful (assuming the same host has also a regular sshd running)
      port = 2222;  
      port = 2222;  
      # hostKeys paths must be unquoted strings, otherwise you'll run into issues with boot.initrd.secrets
      # hostKeys paths must be unquoted strings, otherwise you'll run into issues with boot.initrd.secrets
      # the keys are copied to initrd from the path specified; multiple keys can be set
      # the keys are copied to initrd from the path specified; multiple keys can be set
      # you can generate any number of host keys using  
      # you can generate any number of host keys using  
      # `ssh-keygen -t ed25519 -N "" -f /path/to/ssh_host_ed25519_key`
      # `ssh-keygen -t ed25519 -N "" -f /path/to/ssh_host_ed25519_key`
      hostKeys = [ /path/to/ssh_host_rsa_key ];
      hostKeys = [ /path/to/ssh_host_rsa_key ];
      # public ssh key used for login
      # public ssh key used for login
      authorizedKeys = [ "ssh-rsa AAAA..." ];
      authorizedKeys = [ "ssh-rsa AAAA..." ];
     };
     };
     # this will automatically load the zfs password prompt on login
     # this will automatically load the zfs password prompt on login
Line 446: Line 446:
   };
   };
};
};
</syntaxHighlight>
</syntaxhighlight>
* In order to use DHCP in the initrd, network manager must not be enabled and <code>networking.useDHCP = true;</code> must be set.
* In order to use DHCP in the initrd, network manager must not be enabled and <syntaxhighlight lang="nix" inline>networking.useDHCP = true;</syntaxhighlight> must be set.
* If your network card isn't started, you'll need to add the according kernel module to the initrd as well, e.g. <code>boot.initrd.kernelModules = [ "r8169" ];</code>
* If your network card isn't started, you'll need to add the according kernel module to the initrd as well, e.g. <syntaxhighlight lang="nix" inline>boot.initrd.kernelModules = [ "r8169" ];</syntaxhighlight>


===== Import and unlock multiple encrypted pools/dataset at boot =====
===== Import and unlock multiple encrypted pools/dataset at boot =====
Line 458: Line 458:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
boot = {
boot = {
  initrd.network = {
  initrd.network = {
    enable = true;
    enable = true;
    ssh = {
    ssh = {
        enable = true;
      enable = true;
        port = 2222;  
      port = 2222;  
        hostKeys = [ /path/to/ssh_host_rsa_key ];
      hostKeys = [ /path/to/ssh_host_rsa_key ];
        authorizedKeys = [ "ssh-rsa AAAA..." ];
      authorizedKeys = [ "ssh-rsa AAAA..." ];
    };
    };
    postCommands = ''
    postCommands = ''
      zpool import tankXXX
      zpool import tankXXX
      echo "zfs load-key -a; killall zfs" >> /root/.profile
      echo "zfs load-key -a; killall zfs" >> /root/.profile
    '';
    '';
  };
  };
};
};
</syntaxHighlight>
</syntaxhighlight>


When you login by SSH into the box or when you have physical access to the machine itself, you will be prompted to supply the unlocking password for your zroot and tankXXX pools.
When you login by SSH into the box or when you have physical access to the machine itself, you will be prompted to supply the unlocking password for your zroot and tankXXX pools.
Line 479: Line 479:
=== Install NixOS ===
=== Install NixOS ===
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ nixos-install --show-trace --root /mnt
# nixos-install --show-trace --root /mnt
</syntaxhighlight>
</syntaxhighlight>
<code>--show-trace</code> will show you where exactly things went wrong if <code>nixos-install</code> fails.  To take advantage of all cores on your system, also specify <code>--max-jobs n</code> replacing <code>n</code> with the number of cores on your machine.
<code>--show-trace</code> will show you where exactly things went wrong if <code>nixos-install</code> fails.  To take advantage of all cores on your system, also specify <code>--max-jobs n</code> replacing <code>n</code> with the number of cores on your machine.


== ZFS Trim Support for SSDs ==
== ZFS trim support for SSDs ==


ZFS 0.8 now also features trim support for SSDs.
ZFS 0.8 now also features trim support for SSDs.
Line 492: Line 492:


To manually start trimming of a zpool run: <code>zpool trim tank</code>.
To manually start trimming of a zpool run: <code>zpool trim tank</code>.
Since [https://github.com/NixOS/nixpkgs/pull/65331 PR-65331] this can be also done periodically (by default once a week) by setting <code>services.zfs.trim.enable = true</code>.
Since [https://github.com/NixOS/nixpkgs/pull/65331 PR-65331] this can be also done periodically (by default once a week) by setting <syntaxhighlight lang="nix" inline>services.zfs.trim.enable = true;</syntaxhighlight>.


To set a pool for auto-trim run: <code>zpool set autotrim=on tank</code>
To set a pool for auto-trim run: <code>zpool set autotrim=on tank</code>
Line 512: Line 512:
* https://discourse.nixos.org/t/how-to-add-extra-build-input-to-linux-kernel/8208/3
* https://discourse.nixos.org/t/how-to-add-extra-build-input-to-linux-kernel/8208/3


==Mail notification for ZFS Event Daemon==
== Mail notification for ZFS Event Daemon ==


ZFS Event Daemon (zed) monitors events generated by the ZFS kernel module and runs configured tasks. It can be configured to send an email when a pool scrub is finished or a disk has failed.
ZFS Event Daemon (zed) monitors events generated by the ZFS kernel module and runs configured tasks. It can be configured to send an email when a pool scrub is finished or a disk has failed.


First, we need to configure a mail transfer agent, the program that sends email:
First, we need to configure a mail transfer agent, the program that sends email:
<pre>
<syntaxhighlight lang="nix">
{
{
    programs.msmtp = {
  programs.msmtp = {
      enable = true;
    enable = true;
      setSendmail = true;
    setSendmail = true;
      defaults = {
    defaults = {
        aliases = "/etc/aliases";
      aliases = "/etc/aliases";
        port = 465;
      port = 465;
        tls_trust_file = "/etc/ssl/certs/ca-certificates.crt";
      tls_trust_file = "/etc/ssl/certs/ca-certificates.crt";
        tls = "on";
      tls = "on";
        auth = "login";
      auth = "login";
        tls_starttls = "off";
      tls_starttls = "off";
      };
    };
      accounts = {
    accounts = {
        default = {
      default = {
          host = "mail.example.com";
        host = "mail.example.com";
          passwordeval = "cat /etc/emailpass.txt";
        passwordeval = "cat /etc/emailpass.txt";
          user = "user@example.com";
        user = "user@example.com";
          from = "user@example.com";
        from = "user@example.com";
        };
       };
       };
     };
     };
  };
}
}
</pre>
</syntaxhighlight>


Then, configure an alias for root account. With this alias configured, all mails sent to root, such as cron job results and failed sudo login events, will be redirected to the configured email account.
Then, configure an alias for root account. With this alias configured, all mails sent to root, such as cron job results and failed sudo login events, will be redirected to the configured email account.


<pre>
<syntaxhighlight lang="bash">
tee -a /etc/aliases <<EOF
tee -a /etc/aliases <<EOF
root: user@example.com
root: user@example.com
EOF
EOF
</pre>
</syntaxhighlight>


Finally, override default zed settings with a custom one:
Finally, override default zed settings with a custom one:
<pre>
<syntaxhighlight lang="nix">
{
  services.zfs.zed.settings =
{
{
  ZED_DEBUG_LOG = "/tmp/zed.debug.log";
  services.zfs.zed.settings = {
  ZED_EMAIL_ADDR = [ "root" ];
    ZED_DEBUG_LOG = "/tmp/zed.debug.log";
  ZED_EMAIL_PROG = "${ pkgs.msmtp }/bin/msmtp";
    ZED_EMAIL_ADDR = [ "root" ];
    ZED_EMAIL_PROG = "${pkgs.msmtp}/bin/msmtp";
     ZED_EMAIL_OPTS = "@ADDRESS@";
     ZED_EMAIL_OPTS = "@ADDRESS@";


Line 565: Line 564:
     ZED_USE_ENCLOSURE_LEDS = true;
     ZED_USE_ENCLOSURE_LEDS = true;
     ZED_SCRUB_AFTER_RESILVER = true;
     ZED_SCRUB_AFTER_RESILVER = true;
};
  };
   # this option does not work; will return error
   # this option does not work; will return error
   services.zfs.zed.enableMail = false;
   services.zfs.zed.enableMail = false;
}
}
</pre>
</syntaxhighlight>


You can now test this by performing a scrub
You can now test this by performing a scrub
<pre>
<syntaxhighlight lang="console">
zpool scrub $pool
# zpool scrub $pool
</pre>
</syntaxhighlight>
==Mount datasets without legacy mountpoint==
 
== Mount datasets without legacy mountpoint ==
Contrary to conventional wisdom, <code>mountpoint=legacy</code> is not required for mounting datasets. The trick is to use <code>mount -t zfs -o zfsutil path/to/dataset /path/to/mountpoint</code>.
Contrary to conventional wisdom, <code>mountpoint=legacy</code> is not required for mounting datasets. The trick is to use <code>mount -t zfs -o zfsutil path/to/dataset /path/to/mountpoint</code>.


Line 581: Line 581:


An example configuration of mounting non-legacy dataset is the following:
An example configuration of mounting non-legacy dataset is the following:
<pre>
<syntaxhighlight lang="nix">
{
{
   fileSystems."/tank" =
   fileSystems."/tank" =
Line 588: Line 588:
     };
     };
}
}
</pre>
</syntaxhighlight>


An alternative is to set <code>boot.zfs.extraPools=[pool_name]</code>, which is recommended by the documentation if you have many zfs filesystems.
An alternative is to set <syntaxhighlight lang="nix" inline>boot.zfs.extraPools = [ pool_name ];</syntaxhighlight>, which is recommended by the documentation if you have many zfs filesystems.


==NFS share==
== NFS share ==
With <code>sharenfs</code> property, ZFS has build-in support for generating <code>/etc/exports.d/zfs.exports</code> file, which in turn is processed by NFS service automatically.
With <code>sharenfs</code> property, ZFS has build-in support for generating <code>/etc/exports.d/zfs.exports</code> file, which in turn is processed by NFS service automatically.


Line 601: Line 601:
First, enable [[NFS|NFS service]]:
First, enable [[NFS|NFS service]]:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  services.nfs.server.enable = true;
services.nfs.server.enable = true;
</syntaxhighlight>
</syntaxhighlight>
Only this line is needed. Configure firewall if necessary, as described in [[NFS]] article.
Only this line is needed. Configure firewall if necessary, as described in [[NFS]] article.


Then, set <code>sharenfs</code> property:
Then, set <code>sharenfs</code> property:
<pre>
<syntaxhighlight lang="console">
zfs set sharenfs="ro=192.168.1.0/24,all_squash,anonuid=70,anongid=70" rpool/myData
# zfs set sharenfs="ro=192.168.1.0/24,all_squash,anonuid=70,anongid=70" rpool/myData
</pre>
</syntaxhighlight>
For more options, see <code>man 5 exports</code>.
For more options, see <code>man 5 exports</code>.


==See also==
== See also ==


This article on how to setup encrypted ZFS on Hetzner: <https://mazzo.li/posts/hetzner-zfs.html>.
This article on how to setup encrypted ZFS on Hetzner: <https://mazzo.li/posts/hetzner-zfs.html>.
Anonymous user
Retrieved from "https://wiki.nixos.org/wiki/Special:MobileDiff/450"