Distributed build

Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If you have ssh access to a machine where Nix (not necessarily NixOS) is installed, then you can offload building to this machine.

There is a dedicated chapter in the Nix Manual.

This is a step by step guide to setting up distributed builds.

Prerequisites

First, log-in as the user which runs builds locally. If you are using a single user install, this means yourself, and if this is a multi-user install, this means root, so on the SSH server, you must set PermitRootLogin yes (in NixOS, set services.openssh.settings.permitRootLogin = "yes";)

You must ensure you can run nix* commands on the remote without user interaction and without any option on the ssh command line:

$ ssh builder nix-store --version

Tip

When testing this as another user, make sure your environment ($HOME, ssh-agent) does not leak to this account.

Here is a way to achieve this: First we configure how ssh should connect to our builder.

~/.ssh/config

Host builder
        HostName 192.168.42.42
        Port 1234
        User foo

        # any other fancy option needed to log in
        # ProxyJump foo ...

        # Prevent using ssh-agent or another keyfile, useful for testing
        IdentitiesOnly yes
        IdentityFile /root/.ssh/nix_remote

Tip

When connecting to a new remote builder, ssh will ask you whether you trust the identity of the builder. Nix needs fully unattended connection, so you may want to accept all fingerprints by default. To achieve this you can use StrictHostKeyChecking=accept-new. Note that this has security implications if you usually really check the fingerprints when prompted to do so.

SSH connection must be non-interactive so we use a public key without a passphrase.

$ ssh-keygen -f ~/.ssh/nix_remote
# do not add a passphrase to the ssh key!
$ ssh-copy-id -i ~/.ssh/nix_remote builder

.

Make sure that your local user is added to nix.trustedUsers

When you are done, you can test your setup like this:

$ nix store ping --store ssh://builder --extra-experimental-features nix-command && echo ok

If you get an error like serialised integer ... is too big for type j this means that something (for example /etc/profile or environment.shellInit) outputs bytes to stdout before launching the command specified on the ssh command line. Either disable this behavior or have the output be sent to stderr instead.

Non standard Nix installations

If you are not root on the remote builder and have used nix-user-chroot or PRoot to install nix there (see Nix Installation Guide) then nix is not available on the PATH of the remote builder. We describe a solution for nix-user-chroot which is easily adapted to PRoot.

• Create a script ~/bin/nix_wrapper.sh as follows:

#!/bin/sh
exec ~/bin/nix-user-chroot ~/.nix bash -c '
. ~/.nix-profile/etc/profile.d/nix.sh
exec $SSH_ORIGINAL_COMMAND
'

Of course, adapt this script to the location of the store and nix-user-chroot. Make the script executable.

• In ~/.ssh/authorized_keys, locate the line corresponding to ~/.ssh/nix_remote.pub and prepend this: command="/home/something/bin/nix_wrapper.sh".

Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key.

Single user install

This article or section needs expansion.

Reason: untested (Discuss in Talk:Distributed build#)
Please consult the pedia article metapage for guidelines on contributing.

See the Nix Manual and the option --builders.

Multi-User install

We must configure the nix-daemon to use our builder. Options like --builders on the command line is ignored unless your user is in the trusted user list.

NixOS

There are a few NixOS options we can use:

/etc/nixos/configuration.nix

{ config, pkgs, ... }:

{
	nix.buildMachines = [ {
	 hostName = "builder";
	 system = "x86_64-linux";
	 # if the builder supports building for multiple architectures, 
	 # replace the previous line by, e.g.,
	 # systems = ["x86_64-linux" "aarch64-linux"];
	 maxJobs = 1;
	 speedFactor = 2;
	 supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
	 mandatoryFeatures = [ ];
	}] ;
	nix.distributedBuilds = true;
	# optional, useful when the builder has a faster internet connection than yours
	nix.extraOptions = ''
		builders-use-substitutes = true
	'';
}

Tip

In order to affect your NixOS system by your nix-language-specific changes you must first evaluate it:

$ nixos-rebuild switch --use-remote-sudo

See the Nix Manual for the exact signification of each option.

Non NixOS

This article or section needs expansion.

Reason: untested (Discuss in Talk:Distributed build#)
Please consult the pedia article metapage for guidelines on contributing.

The previous method should be rather easily adaptable: replace adding NixOS options by editing /etc/nix/nix.conf.

Using remote builders

Local builder

Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally. To never use the local machine set the max-jobs nix option to 0

$ nix-build -j0 blah

Features

Each builder is declared with a set of supportedFeatures. When a builder lacks one of the requiredSystemFeatures of a derivation, it will be ignored. Here are some features used in nixpkgs:

Feature	Derivations requiring it
kvm	Everything which builds inside a vm, like NixOS tests
nixos-test	Machine can run NixOS tests
big-parallel	kernel config, libreoffice, evolution, llvm and chromium.
benchmark	Machine can generate metrics (Means the builds usually takes the same amount of time)

To know what features a derivation needs, you can run

$ nix show-derivation /nix/store/hash-foo.drv 

Using remote builders as substituters

If you have two remote builders A and B (where A has higher speed than B), that a derivation foo.drv is already built on B, and that your local machine needs to build foo.drv, then it will:

• build (possibly remotely) all the build dependencies of foo.drv
• build foo.drv on A

Even if foo.drv is 'also' on A, you will still have to build the build dependencies of foo.drv before sending the build to A which will build it instantly since it is in cache.

To solve this problem, you can set up your remote builders as substituters. Every time (the local machine's) nix considers building a derivation, it will connect to the remote builders to check whether it is already available there. Here is how to set this up via ssh. See also Binary Cache for an alternative using http and nix-serve.

1. On the remote builder, create a binary cache key:

$ nix-store --generate-binary-cache-key builder-name cache-priv-key.pem cache-pub-key.pem

The private key must be readable only by the user running the build: ??? on multi-user installs, and the owner of /nix on single-user installs. builder-name is only here for your convenience to distinguish several public keys, it has no functional meaning.

2. On the remote builder, set up nix to sign all store paths it builds: in the nix configuration (/etc/nix/nix.conf on multi-user installs and ~/.config/nix/nix.conf on single user installs), add the following line

secret-key-files = /path/to/cache-priv-key.pem

If necessary, restart the nix daemon.

3. The previous point does not retroactively sign existing paths in the store of the builder. To do so, run

$ nix sign-paths --all -k /path/to/cache-priv-key.pem

4. In the nix configuration of the local machine, append the content of cache-pub-key.pem to the option trusted-public-keys. Also append ssh-ng://builder to the option substituters. If you only want to use the remote builder occasionally as a substituter, use trusted-substituters instead of substituters. Then, when you want to use the builder, pass --option extra-substituters ssh-ng://builder to the nix command you run.

Troubleshooting

• How do I know if I'm distributing my build at all?
  • Run nix build with --max-jobs 0.
• How do I know why my builds aren't being distributed?
  • Run nix build -vvvvvvvvv 2>&1 | less and search for decline.
• I can nix ping-store but the build doesn't distribute.
  • If on NixOS, Check that nix ping-store command works when run as root.
  • If you configured builders on the command line (with --builders), make sure your account is in nix.trustedUsers in /etc/nixos/configuration.nix. Only /etc/nix/nix.conf is taken into account otherwise.
• I can ping the store as root, but I'm getting "broken pipe" errors when trying to distribute.
  • You may have hit bug #46038. Add nix.distributedBuilds = true; to configuration.nix and nixos-rebuild switch.

See also

See also:

• The NixOS Remote Builds Test Case
• The Mail to nixos-dev about setting up remote builds by Russell O'Connor
• A step-by-step guide on remote Firefox building through bastion host
• Offloading NixOS builds to a faster machine
• Run a qemu Linux builder on macOS