Official NixOS Wiki

Distributed build: Difference between revisions

← Older editNewer edit →
VisualWikitext
imported>Makefu
No edit summary
imported>Symphorien
completely rewrite for nix2; NixOS only.
Line 1: Line 1:
Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If both of them run NixOS you can follow this little HOWTO to make it happen.
Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If you have ssh
access to a machine where Nix (not necessarily NixOS) is installed, then you can offload building to this machine.


== Using nix.buildMachines ==
This is a step by step guide to setting up distributed builds.
 
Even after enabling nix.distributedBuilds and setting nix.buildMachines there are some catches:
 
* If you have max builds > 0 then nix will try to build locally when connecting to remote fails - keep this in mind when testing.
* Your local root must have the slave in ~/.ssh/known_hosts


== Prerequisites ==
First, log-in as the user which runs builds locally. If you are using a single user install, this means yourself, and if this is a
multi-user install, this means <code>root</code>.


== Prerequisites ==
You must ensure you can run <code>nix*</code> commands on the remote without user interaction and without any option on the ssh command line:
{{outdated|look at nix.buildMachines in [https://nixos.org/nix/manual/#chap-distributed-builds NixOS Manual]}}
{{Commands|$ ssh builder nix-store --version}}


You'll need to setup public key based login from the slower machine to root account on the one that'll do the actual building, and this article won't cover setting this up (there are many that do, so you shouldn't have any problem googling this).
{{Tip|When testing this as another user, make sure your environment (<code>$HOME</code>, <code>ssh-agent</code>) does not leak to this account.}}


== Preparing ==
Here is a way to achieve this:
First we configure how ssh should connect to our builder.
{{file|~/.ssh/ssh_config|text|
<nowiki>
Host builder
        HostName 192.168.42.42
        Port 1234
        User foo


First you have to prepare the remote-systems.conf file on the slower machine, for example in /etc/nixos/remote-systems.conf, with contents similar to:
        # any other fancy option needed to log in
<pre>
        # ProxyJump foo ...
root@builder.example.com i686-linux /etc/nixos/id_rsa 4
</pre>


Where:
        # Prevent using ssh-agent or another keyfile, useful for testing
        IdentitiesOnly=yes
        IdentityFile /root/.ssh/nix_remote
        # There must not be any user interaction for logging in
        # Disable the annoying prompt when ssh-ing for the first time
        StrictHostKeyChecking=no
        UserKnownHostsFile=/dev/null
</nowiki>
}}
{{Tip|Some options presented here like <code>StrictHostKeyChecking no</code> weaken security and are not strictly speaking necessary  but make things easier. Adapt to your taste.}}


1. root@builder.example.com is and addres of the machine that'll do the build. If you want to use port other than 22, you have to set up an alias in /root/.ssh/config or pass a command line switch to ssh using NIX_SSHOPTS environment variable
SSH connection must be non-interactive so we use a public key '''without a passphrase'''.
<pre>
{{Commands|
export NIX_SSHOPTS="-p10022"
$ ssh-keygen -f ~/.ssh/nix_remote
</pre>
# do not add a passphrase to the ssh key!
$ ssh-copy-id -i ~/.ssh/nix_remote builder
}}


2. i686-linux is a platform of that machine
When you are done, you can test your setup like this:
{{Commands|$ nix ping-store --store ssh://builder}}


3. /etc/nixos/id_rsa is a path to the private part of a key used to log in to the builder
If you get an error like <code>serialised integer ... is too big for type j</code> this means that something (<code>/etc/profile</code> for example) outputs bytes to <code>stdout</code> before launching the command specified on the ssh
command line. Either disable this behavior or have the output be sent to <code>stderr</code> instead.
== Single user install ==
{{Expansion|untested}}
See the [https://nixos.org/nix/manual/#chap-distributed-builds Nix Manual] and the option <code>--builders</code>.


4. 4 is a number of jobs that should be run in parallel on that machine
== Multi-User install ==
We must configure the <code>nix-daemon</code> to use our builder. Options like <code>--builders</code> on the command line seem to be ignored.
=== NixOS ===
There are a few NixOS options we can use:
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{ config, pkgs, ... }:


As a convenience, we'll make a little script. Put this into /etc/nixos/remote-build-env:
{
<syntaxhighlight lang="bash">
nix.buildMachines = [ {
mkdir /tmp/build-remote-load/
hostName = "builder";
chmod a+rwX /tmp/build-remote-load/
system = "x86_64-linux";
maxJobs = 1;
speedFactor = 2;
supportedFeatures = [ ];
mandatoryFeatures = [ ];
}] ;
nix.distributedBuilds = true;
# optional, useful when the builder has a faster internet connection than yours
nix.extraOptions = ''
builders-use-substitutes = true
'';
}
</nowiki>}}
{{Evaluate}}
See the [https://nixos.org/nix/manual/#chap-distributed-builds Nix Manual] for the exact signification of each option.


# First find our build hook script
=== Non NixOS ===
STORE_PATH_FOR_NIX=$(ls -l `which nix-env` | awk '{ sub( /bin\/nix-env/, "", $NF ); print $NF }')
{{Expansion|untested}}
BUILD_REMOTE=`find "${STORE_PATH_FOR_NIX}" -name build-remote.pl`
The previous method should be rather easily adaptable: replace adding NixOS options by editing <code>/etc/nix/nix.conf</code>.


# now set up environment for nix-worker                   
== Using remote builders ==
export NIX_BUILD_HOOK="${BUILD_REMOTE}"
Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally.
export NIX_REMOTE_SYSTEMS="/etc/nixos/remote-systems.conf"
To never use the local machine set the <code>max-jobs</code> nix option to 0
export NIX_CURRENT_LOAD="/tmp/build-remote-load"
{{Commands|$ nix-build -j0 blah}}
</syntaxhighlight>
== Running the build ==
This is the part that you'll have to do every time you want to build something remotely.
First, as root:
<syntaxhighlight lang="bash">
stop nix-daemon
. /etc/nixos/remote-build-env
nix-worker --daemon &
</syntaxhighlight>
Then, as a user you want to do the build as:
<syntaxhighlight lang="bash">
. /etc/nixos/remote-build-env
</syntaxhighlight>
after that you can run nix-env normally, and the work should be distributed among machines in your remote-systems.conf


== using remote builds on NixOS ==
== See also ==
See configuration option [https://nixos.org/nixos/options.html#nix.distributedbuilds nix.distributedbuilds and nix.buildhosts] to set this up in your <code>/etc/configuration.nix</code> file.
See also:
See also:
* [https://github.com/NixOS/nix/blob/master/tests/remote-builds.nix#L46-L58 The NixOS Remote Builds Test Case]
* [https://github.com/NixOS/nix/blob/master/tests/remote-builds.nix#L46-L58 The NixOS Remote Builds Test Case]
* [https://nixos.org/nix-dev/2015-September/018255.html The Mail to nixos-dev about setting up remote builds by Russell O'Connor]
* [https://nixos.org/nix-dev/2015-September/018255.html The Mail to nixos-dev about setting up remote builds by Russell O'Connor]
Retrieved from "https://wiki.nixos.org/wiki/Distributed_build"