Distributed build

From NixOS Wiki
Revision as of 12:44, 26 May 2018 by imported>Symphorien (completely rewrite for nix2; NixOS only.)

Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If you have ssh access to a machine where Nix (not necessarily NixOS) is installed, then you can offload building to this machine.

This is a step by step guide to setting up distributed builds.

Prerequisites

First, log-in as the user which runs builds locally. If you are using a single user install, this means yourself, and if this is a multi-user install, this means root.

You must ensure you can run nix* commands on the remote without user interaction and without any option on the ssh command line:

$ ssh builder nix-store --version

Here is a way to achieve this: First we configure how ssh should connect to our builder.

~/.ssh/ssh_config
Host builder
        HostName 192.168.42.42
        Port 1234
        User foo

        # any other fancy option needed to log in
        # ProxyJump foo ...

        # Prevent using ssh-agent or another keyfile, useful for testing
        IdentitiesOnly=yes
        IdentityFile /root/.ssh/nix_remote
        # There must not be any user interaction for logging in
        # Disable the annoying prompt when ssh-ing for the first time
        StrictHostKeyChecking=no
        UserKnownHostsFile=/dev/null

SSH connection must be non-interactive so we use a public key without a passphrase.

$ ssh-keygen -f ~/.ssh/nix_remote
# do not add a passphrase to the ssh key!
$ ssh-copy-id -i ~/.ssh/nix_remote builder

When you are done, you can test your setup like this:

$ nix ping-store --store ssh://builder

If you get an error like serialised integer ... is too big for type j this means that something (/etc/profile for example) outputs bytes to stdout before launching the command specified on the ssh command line. Either disable this behavior or have the output be sent to stderr instead.

Single user install

See the Nix Manual and the option --builders.

Multi-User install

We must configure the nix-daemon to use our builder. Options like --builders on the command line seem to be ignored.

NixOS

There are a few NixOS options we can use:

/etc/nixos/configuration.nix
{ config, pkgs, ... }:

{
	nix.buildMachines = [ {
	 hostName = "builder";
	 system = "x86_64-linux";
	 maxJobs = 1;
	 speedFactor = 2;
	 supportedFeatures = [ ];
	 mandatoryFeatures = [ ];
	}] ;
	nix.distributedBuilds = true;
	# optional, useful when the builder has a faster internet connection than yours
	nix.extraOptions = ''
		builders-use-substitutes = true
	'';
}

See the Nix Manual for the exact signification of each option.

Non NixOS

The previous method should be rather easily adaptable: replace adding NixOS options by editing /etc/nix/nix.conf.

Using remote builders

Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally. To never use the local machine set the max-jobs nix option to 0

$ nix-build -j0 blah

See also

See also: