Official NixOS Wiki

ZFS: Difference between revisions

← Older edit
Kacper (talk | contribs)
4 edits
VisualWikitext
Tie-ling (talk | contribs)
92 edits
Kacper (talk | contribs)
4 edits
→Remote unlock: Remove a note to a closed issue, assuming this has been resolved already.
 
(6 intermediate revisions by 5 users not shown)
Line 5: Line 5:
ZFS integrates into NixOS via the {{nixos:option|boot.zfs}} and {{nixos:option|services.zfs}} options.
ZFS integrates into NixOS via the {{nixos:option|boot.zfs}} and {{nixos:option|services.zfs}} options.


Uninterruptible power supply (UPS), provides near-instantaneous
== Limitations ==
protection from input power interruptions by switching to energy
stored in battery packs, supercapacitors or flywheels. The on-battery
run-times of most UPSs are relatively short (only a few minutes) but
sufficient to "buy time" for initiating a standby power source or
properly shutting down the protected equipment. (source: Wikipedia)


Network UPS Tools (NUT) is a collection of software for managing power
==== Latest Kernel compatible with ZFS ====
devices, mainly UPS units. This article describes the configuration of
ZFS often does not support the latest Kernel versions. It is recommended to use an LTS Kernel version whenever possible; the NixOS default Kernel is generally suitable. See [[Linux kernel|Linux Kernel]] for more information about configuring a specific Kernel version.
NUT for a simple server with a single power supply, with no local
users and no additional equipment.


This article is mostly adapted from the excellent NUT ConfigExamples
If your config specifies a Kernel version that is not officially supported by upstream ZFS, the ZFS module will fail to evaluate with an error that the ZFS package is "broken". Upstream ZFS changed in 2.3 to refuse to build by default, regardless of Nixpkgs’ broken marking (or ignoring).  
book, version 3.0, by Roger
Price. [https://github.com/networkupstools/ConfigExamples/releases/latest/download/ConfigExamples.pdf]


= Compatible Hardware =
===== Selecting the latest ZFS-compatible Kernel =====
{{Warning|This will often result in the Kernel version going backwards as Kernel versions become end-of-life and are removed from Nixpkgs. If you need more control over the Kernel version due to hardware requirements, consider simply pinning a specific version rather than calculating it as below.}}
To use the latest ZFS-compatible Kernel currently available, the following configuration may be used.


Compatible hardware are listed at NUT website
[https://networkupstools.org/stable-hcl.html].  For best results,
choose a model with good driver support with battery replacement
notification.  Battery pack in UPS is designed to be replaced every
few years.  For example, Eaton Ellipse ECO 650
[https://www.eaton.com/content/dam/eaton/products/backup-power-ups-surge-it-power-distribution/backup-power-ups/eaton-ellipse-eco/eaton-ellipse-eco-userguides-en.pdf]
needs a new battery every four years, under optimal operating
conditions.
= Components of NUT Software =
One or more UPS's are attached to the <strong>attachment
daemon</strong> <code>upsd</code> via a UPS-specific <strong>driver
daemon</strong>.
The attachment daemon maintains an abstract image of the UPS in
memory.  The attachment daemon can be queried by the <code>upsc</code>
command. The driver daemon talks to the hardware and the attachment
daemon.  The driver daemon can be controlled by the
<code>upsdrvctl</code> command.
The <strong>management daemon</strong> <code>upsmon</code> is a client
of upsd.  It runs permanently, checks the status of UPS, and react to
status changes, such as initiating a shutdown.
= Configuration files in use =
In this simple standalone server setup, the following configuration
files are generated:
* ups.conf, declare UPS-specific driver information, power.ups.ups.* option
* upsd.conf, control access to upsd, power.ups.upsd option
* upsd.users, add user with access to upsd, power.ups.users option
* upsmon.conf, connect to upsd, power.ups.upsmon.monitor section;
* upsmon.conf, set how upsmon should react to status changes, power.ups.upsmon.settings section
* delayed UPS shutdown systemd unit, to make Restore Power on AC Return BIOS option functional, systemd.services.nut-delayed-ups-shutdown section
= Declare UPS units =
Corresponds to file ups.conf
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
  power.ups = {
{
    enable = true;
  config,
    mode = "standalone";
  lib,
    # section: The upsd UPS declarations: ups.conf
  pkgs,
    # this UPS device is named UPS-1.
  ...
    ups."UPS-1" = {
}:
      description = "Eaton Ellipse ECO 650 with 12V 7Ah Batt";


      # driver name from https://networkupstools.org/stable-hcl.html
let
      driver = "usbhid-ups";
  zfsCompatibleKernelPackages = lib.filterAttrs (
 
    name: kernelPackages:
      # usbhid-ups driver always use value "auto"
    (builtins.match "linux_[0-9]+_[0-9]+" name) != null
      port = "auto";
    && (builtins.tryEval kernelPackages).success
 
    && (!kernelPackages.${config.boot.zfs.package.kernelModuleAttribute}.meta.broken)
      directives = [
  ) pkgs.linuxKernel.packages;
        # "Restore power on AC" BIOS option needs power to be cut a few seconds to work;
  latestKernelPackage = lib.last (
        # this is achieved by the offdelay and ondelay directives.
    lib.sort (a: b: (lib.versionOlder a.kernel.version b.kernel.version)) (
 
      builtins.attrValues zfsCompatibleKernelPackages
        # in the last stages of system shutdown, "upsdrvctl shutdown" is called to tell UPS that
    )
        # after offdelay seconds, the UPS power must be cut, even if
  );
        # wall power returns.
in
        "offdelay = 60"
{
 
  # Note this might jump back and forth as kernels are added or removed.
        # UPS power is now cut regardless of wall power. After (ondelay minus offdelay) seconds,
  boot.kernelPackages = latestKernelPackage;
        # if wall power returns, turn on UPS power. The system has now been disconnected for a minimum of (ondelay minus offdelay) seconds,
}
        # "Restore power on AC" should now power on the system.
</syntaxhighlight>
        # For reasons described above, ondelay value must be larger than offdelay value.
        "ondelay = 70"


        # set value for battery.charge.low,
===== Using unstable, pre-release ZFS =====
        # upsmon initiate shutdown once this threshold is reached.
{{Warning|Pre-release ZFS versions may be less well-tested, and may have critical bugs that may cause data loss.}}{{Warning|Running ZFS with a Kernel unsupported by upstream “is considered EXPERIMENTAL by the OpenZFS project. Even if it appears to build and run correctly, there may be bugs that can cause SERIOUS DATA LOSS.”}}
        "lowbatt = 40"
In some cases, a pre-release version of ZFS may be available that supports a newer Kernel. Use it with <code>boot.zfs.package = pkgs.zfs_unstable;</code>. Using zfs_unstable may allow the use of an unsupported Kernel; as warned above, [https://github.com/openzfs/zfs/blob/6a2f7b38442b42f4bc9a848f8de10fc792ce8d76/config/kernel.m4#L473-L487 upstream considers this experimental].
      ];
    };
</syntaxhighlight>


= Declare upsd listening ports =
==== Partial support for swap on ZFS ====
Corresponds to file upsd.conf.  This file declares which ports the upsd daemon will listen to.


<syntaxhighlight lang="nix">
ZFS does not support swapfiles. swap devices can be used instead. Additionally, hibernation is disabled by default due to a [https://github.com/NixOS/nixpkgs/pull/208037 high risk] of data corruption. Note that even if that pull request is merged, it does not fully mitigate the risk. If you wish to enable hibernation regardless and made sure that swapfiles on ZFS are not used, set <code>boot.zfs.allowHibernation = true</code>.
  power.ups = {
    # section: The upsd daemon access control; upsd.conf
    upsd = {
      listen = [
        {
          address = "127.0.0.1";
          port = 3493;
        }
        {
          address = "::1";
          port = 3493;
        }
      ];
    };
  };
</syntaxhighlight>


= Declare users with access to UPS =
==== Zpool not found ====
Corresponds to file upsd.users.  This file declares a virtual user (not related to /etc/passwd users) with write access to UPS.  A password is also declared.


<syntaxhighlight lang="nix">
If NixOS fails to import the zpool on reboot, you may need to add <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-path";</syntaxhighlight> or <syntaxhighlight lang="nix" inline>boot.zfs.devNodes = "/dev/disk/by-partuuid";</syntaxhighlight> to your configuration.nix file.
  power.ups = {
    # section: Users that can access upsd. The upsd daemon user
    # declarations. upsd.users
    users."nut-admin" = {
      passwordFile = ../resources/ups-passwd.txt;
      upsmon = "primary";
    };
  };
</syntaxhighlight>


= Connect upsmon to upsd =
The differences can be tested by running <code>zpool import -d /dev/disk/by-id</code> when none of the pools are discovered, eg. a live iso.
Corresponds to upsmon.conf.  This file declares how upsmon should connect to upsd
<syntaxhighlight lang="nix">
  power.ups = {
    # section: The upsmon daemon configuration: upsmon.conf
    upsmon.monitor."UPS-1" = {
      system = "UPS-1@localhost";
      powerValue = 1;
      user = "nut-admin";
      passwordFile = ../resources/ups-passwd.txt;
      type = "primary";
    };
  };
</syntaxhighlight>


= Declare how upsmon should react to status changes =
==== ZFS conflicting with systemd ====
Corresponds to upsmon.conf.  This file declares how upsmon is to handle NOTIFY events.


<syntaxhighlight lang="nix">
ZFS will manage mounting non-legacy ZFS filesystems, but NixOS tries to manage mounting with systemd. ZFS native mountpoints are not managed as part of the system configuration (but better support hibernation with a separate swap partition). This can lead to conflicts if the ZFS mount service is also enabled for the same datasets.  
  power.ups = {
    upsmon.settings = {
      # This configuration file declares how upsmon is to handle
      # NOTIFY events.


      # POWERDOWNFLAG and SHUTDOWNCMD is provided by NixOS default
Disable the mount service with <code>systemd.services.zfs-mount.enable = false;</code> or remove the <code>fileSystems</code> entries in hardware-configuration.nix. Otherwise, use legacy mountpoints (created with e.g. <code>zfs create -o mountpoint=legacy</code>). Mountpoints must be specified with <code>fileSystems."/mount/point" = {};</code> or with <code>nixos-generate-config</code>.
      # values


      # values provided by ConfigExamples 3.0 book
==== Nix builds and ZFS properties like normalization or utf8only ====
      NOTIFYMSG = [
These options are often suggested in guides to setting up ZFS. <code>normalization</code> makes filenames compare the same in cases where there exists more than one UTF8 bytestring that represents the same characters. <code>utf8only</code> prevents the creation of files with non-UTF8 filenames, e.g. filenames using a Latin1 character set. These are non-POSIX and will make the tests for certain packages fail, which may interfere with builds. After nix 2.30, builds no longer happen in /tmp by default, instead they happen in <code>/nix/var/nix/builds</code>. On any system where you plan to run nix builds, you should ensure that this filesystem is POSIX-compliant. Either mounting a tmpfs in that directory (if you have lots of RAM + swap) or creating a zfs dataset there which does not have these or other non-POSIX settings like <code>noatime</code>, <code>snapdir=visible</code>, <code>acltype=nfsv4</code>, or <code>caseinsensitivity=insensitive</code>. Many of these cannot be changed after dataset creation so if this is your root filesystem, you will need to restore from a backup in order to recreate them.
        [ "ONLINE" ''"UPS %s: On line power."'' ]
        [ "ONBATT" ''"UPS %s: On battery."'' ]
        [ "LOWBATT" ''"UPS %s: Battery is low."'' ]
        [ "REPLBATT" ''"UPS %s: Battery needs to be replaced."'' ]
        [ "FSD" ''"UPS %s: Forced shutdown in progress."'' ]
        [ "SHUTDOWN" ''"Auto logout and shutdown proceeding."'' ]
        [ "COMMOK" ''"UPS %s: Communications (re-)established."'' ]
        [ "COMMBAD" ''"UPS %s: Communications lost."'' ]
        [ "NOCOMM" ''"UPS %s: Not available."'' ]
        [ "NOPARENT" ''"upsmon parent dead, shutdown impossible."'' ]
      ];
      NOTIFYFLAG = [
        [ "ONLINE" "SYSLOG+WALL" ]
        [ "ONBATT" "SYSLOG+WALL" ]
        [ "LOWBATT" "SYSLOG+WALL" ]
        [ "REPLBATT" "SYSLOG+WALL" ]
        [ "FSD" "SYSLOG+WALL" ]
        [ "SHUTDOWN" "SYSLOG+WALL" ]
        [ "COMMOK" "SYSLOG+WALL" ]
        [ "COMMBAD" "SYSLOG+WALL" ]
        [ "NOCOMM" "SYSLOG+WALL" ]
        [ "NOPARENT" "SYSLOG+WALL" ]
      ];
      # every RBWARNTIME seconds, upsmon will generate a replace
      # battery NOTIFY event
      RBWARNTIME = 216000;
      # every NOCOMMWARNTIME seconds, upsmon will generate a UPS
      # unreachable NOTIFY event
      NOCOMMWARNTIME = 300;
      # after sending SHUTDOWN NOTIFY event to warn users, upsmon
      # waits FINALDELAY seconds long before executing SHUTDOWNCMD
      # Some UPS's don't give much warning for low battery and will
      # require a value of 0 here for aq safe shutdown.
      FINALDELAY = 0;
    };
  };
</syntaxhighlight>


== Guides ==
== Guides ==
Line 461: Line 326:
</syntaxhighlight>
</syntaxhighlight>


You can confirm whether any specified configuration/tuning got applied via commands like <code>arc_summary</code> and <code>arcstat -a -s " "</code>.
You can confirm whether any specified configuration/tuning got applied via commands like <code>zarcsummary</code> and <code>zarcstat -a -s " "</code>.


== Automatic scrubbing ==
== Automatic scrubbing ==
Line 474: Line 339:
=== Unlock encrypted ZFS via SSH on boot ===
=== Unlock encrypted ZFS via SSH on boot ===


{{note|As of 22.05, rebuilding your config with the below directions may result in a situation where, if you want to revert the changes, you may need to do some pretty hairy nix-store manipulation to be able to successfully rebuild, see https://github.com/NixOS/nixpkgs/issues/101462#issuecomment-1172926129}}
{{merge|Remote_disk_unlocking}}In case you want unlock a machine remotely (after an update), having an ssh service in initrd for the password prompt is handy:
 
In case you want unlock a machine remotely (after an update), having an ssh service in initrd for the password prompt is handy:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
Line 557: Line 420:
== Take snapshots automatically ==
== Take snapshots automatically ==


See {{nixos:option|services.sanoid}} section in <code>man configuration.nix</code>.
See {{nixos:option|services.zfs.autoSnapshot}} or {{nixos:option|services.sanoid}} section in <code>man configuration.nix</code>.


== NFS share ==
== NFS share ==
Line 634: Line 497:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{
{
   services.zfs.zed. = {
   services.zfs.zed = {
     enableMail = true;
     enableMail = true;
     settings = {
     settings = {
Retrieved from "https://wiki.nixos.org/wiki/ZFS"