Distributed build

Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If you have ssh access to a machine where Nix (not necessarily NixOS) is installed, then you can offload building to this machine.

There is a dedicated chapter in the Nix Manual.

This is a step by step guide to setting up distributed builds.

Prerequisites

First, log-in as the user which runs builds locally. If you are using a single user install, this means yourself, and if this is a multi-user install, this means root.

You must ensure you can run nix* commands on the remote without user interaction and without any option on the ssh command line:

$ ssh builder nix-store --version

Tip

When testing this as another user, make sure your environment ($HOME, ssh-agent) does not leak to this account.

Here is a way to achieve this: First we configure how ssh should connect to our builder.

~/.ssh/config

Host builder
        HostName 192.168.42.42
        Port 1234
        User foo

        # any other fancy option needed to log in
        # ProxyJump foo ...

        # Prevent using ssh-agent or another keyfile, useful for testing
        IdentitiesOnly yes
        IdentityFile /root/.ssh/nix_remote

Tip

You can add the StrictHostKeyChecking no and UserKnownHostsFile /dev/null options to disable host authentication for the remote builder, which can be convenient for instance when working with frequently replaced local VMs. Do not use this for machines accessed via any network you do not trust, especially the internet!

SSH connection must be non-interactive so we use a public key without a passphrase.

$ ssh-keygen -f ~/.ssh/nix_remote
# do not add a passphrase to the ssh key!
$ ssh-copy-id -i ~/.ssh/nix_remote builder

When you are done, you can test your setup like this:

$ nix ping-store --store ssh://builder

If you get an error like serialised integer ... is too big for type j this means that something (/etc/profile for example) outputs bytes to stdout before launching the command specified on the ssh command line. Either disable this behavior or have the output be sent to stderr instead.

Single user install

This article or section needs expansion.

Reason: untested (Discuss in Talk:Distributed build#)
Please consult the pedia article metapage for guidelines on contributing.

See the Nix Manual and the option --builders.

Multi-User install

We must configure the nix-daemon to use our builder. Options like --builders on the command line is ignored unless your user is in the trusted user list.

NixOS

There are a few NixOS options we can use:

/etc/nixos/configuration.nix

{ config, pkgs, ... }:

{
	nix.buildMachines = [ {
	 hostName = "builder";
	 system = "x86_64-linux";
	 maxJobs = 1;
	 speedFactor = 2;
	 supportedFeatures = [ ];
	 mandatoryFeatures = [ ];
	}] ;
	nix.distributedBuilds = true;
	# optional, useful when the builder has a faster internet connection than yours
	nix.extraOptions = ''
		builders-use-substitutes = true
	'';
}

Tip

In order to affect your NixOS system by your nix-language-specific changes you must first evaluate it:

$ nixos-rebuild switch --use-remote-sudo

See the Nix Manual for the exact signification of each option.

Non NixOS

This article or section needs expansion.

Reason: untested (Discuss in Talk:Distributed build#)
Please consult the pedia article metapage for guidelines on contributing.

The previous method should be rather easily adaptable: replace adding NixOS options by editing /etc/nix/nix.conf.

Note

Because of bug #46038, you need to set nix.distributedBuilds = true; in configuration.nix even if you are triggering a distributed build via the command line.

Using remote builders

Local builder

Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally. To never use the local machine set the max-jobs nix option to 0

$ nix-build -j0 blah

Features

Each builder is declared with a set of supportedFeatures. When a builder lacks one of the requiredSystemFeatures of a derivation, it will be ignored. Here are some features used in nixpkgs:

Feature	Derivations requiring it
kvm	Everything which builds inside a vm, like NixOS tests
nixos-test	Machine can run NixOS tests
big-parallel	kernel config, libreoffice, evolution and chromium.
benchmark	Machine can generate metrics (Means the builds usually takes the same amount of time)

To know what features a derivation needs, you can run

$ nix show-derivation /nix/store/hash-foo.drv 

Troubleshooting

• How do I know if I'm distributing my build at all?
  • Run nix build with --max-jobs 0.
• How do I know why my builds aren't being distributed?
  • Run nix build -vvvvvvvvv 2>&1 | less and search for decline.
• I can nix ping-store but the build doesn't distribute.
  • If on NixOS, Check that nix ping-store command works when run as root.
  • If on NixOS, make sure your account is in nix.trustedUsers in /etc/nixos/configuration.nix.
• I can ping the store as root, but I'm getting "broken pipe" errors when trying to distribute.
  • You may have hit bug #46038. Add nix.distributedBuilds = true; to configuration.nix and nixos-rebuild switch.

See also

See also:

• The NixOS Remote Builds Test Case
• The Mail to nixos-dev about setting up remote builds by Russell O'Connor