Distributed build

Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If both of them run NixOS you can follow this little HOWTO to make it happen.

Using nix.buildMachines

Even after enabling nix.distributedBuilds and setting nix.buildMachines there are some catches:

• If you have max builds > 0 then nix will try to build locally when connecting to remote fails - keep this in mind when testing.
• Your local root must have the slave in ~/.ssh/known_hosts

Prerequisites

This article or section is out of date.

Reason: look at nix.buildMachines in NixOS Manual (Discuss in Talk:Distributed build)
Once the page has been updated this notice can be removed.

You'll need to setup public key based login from the slower machine to root account on the one that'll do the actual building, and this article won't cover setting this up (there are many that do, so you shouldn't have any problem googling this).

Preparing

First you have to prepare the remote-systems.conf file on the slower machine, for example in /etc/nixos/remote-systems.conf, with contents similar to:

root@builder.example.com i686-linux /etc/nixos/id_rsa 4

Where:

1. root@builder.example.com is and addres of the machine that'll do the build. If you want to use port other than 22, you have to set up an alias in /root/.ssh/config or pass a command line switch to ssh using NIX_SSHOPTS environment variable

export NIX_SSHOPTS="-p10022"

2. i686-linux is a platform of that machine

3. /etc/nixos/id_rsa is a path to the private part of a key used to log in to the builder

4. 4 is a number of jobs that should be run in parallel on that machine

As a convenience, we'll make a little script. Put this into /etc/nixos/remote-build-env:

mkdir /tmp/build-remote-load/
chmod a+rwX /tmp/build-remote-load/

# First find our build hook script
STORE_PATH_FOR_NIX=$(ls -l `which nix-env` | awk '{ sub( /bin\/nix-env/, "", $NF ); print $NF }')
BUILD_REMOTE=`find "${STORE_PATH_FOR_NIX}" -name build-remote.pl`

# now set up environment for nix-worker                    
export NIX_BUILD_HOOK="${BUILD_REMOTE}"
export NIX_REMOTE_SYSTEMS="/etc/nixos/remote-systems.conf"
export NIX_CURRENT_LOAD="/tmp/build-remote-load"

Running the build

This is the part that you'll have to do every time you want to build something remotely. First, as root:

stop nix-daemon
. /etc/nixos/remote-build-env
nix-worker --daemon &

Then, as a user you want to do the build as:

. /etc/nixos/remote-build-env

after that you can run nix-env normally, and the work should be distributed among machines in your remote-systems.conf

using remote builds on NixOS

See configuration option nix.distributedbuilds and nix.buildhosts to set this up in your /etc/configuration.nix file. See also:

• The NixOS Remote Builds Test Case
• The Mail to nixos-dev about setting up remote builds by Russell O'Connor