Distributed build: Difference between revisions

From NixOS Wiki
imported>Srid
→‎NixOS: Use ssh-ng (more reliable)
imported>Doronbehar
Rewrite most of the guide, mostly with better specification of remote vs local machine, with additional usecases.
Line 4: Line 4:
There is a dedicated chapter in the [https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html Nix Manual].
There is a dedicated chapter in the [https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html Nix Manual].


This is a step by step guide to setting up distributed builds.
This is a step by step guide to setting up distributed builds, that includes some SSH tips that are out of scope for Nix Manual's chapter.


== Prerequisites ==
== Prerequisites ==
First, log-in as the user which runs builds locally. If you are using a single user install, this means yourself, and if this is a
multi-user install, this means <code>root</code>, so on the SSH server, you must set <code>PermitRootLogin yes</code> (in NixOS, set <code>services.openssh.settings.permitRootLogin = "yes";</code>)


You must ensure you can run <code>nix*</code> commands on the remote without user interaction and without any option on the ssh command line:
SSH access to the remote builder must be setup for being able to distribute builds to it. The table below lists SSH access requirements for (hopefully) all the scenarios you might find yourself:
{{Commands|$ ssh builder nix-store --version}}


{{Tip|When testing this as another user, make sure your environment (<code>$HOME</code>, <code>ssh-agent</code>) does not leak to this account.}}
{| class="wikitable" style="margin:auto"
|-
! Local Machine !! Remote Builder !! Requirements
|-
| NixOS / System-wide Nix installation  || NixOS / System-wide Nix installation || '''Local''' machine's {{ic|root}} user needs SSH access to ''a'' user on the '''remote''' machine.
|-
| Single-user Nix installation || NixOS / System-wide Nix installation || ''Your'' user on your '''Local''' machine needs SSH access to ''a'' user on the '''remote''' machine.
|-
| NixOS / System-wide Nix installation || Single-user Nix installation || '''Local''' machine's {{ic|root}} user needs SSH access to the user on the '''remote''' machine with Nix installed with their UID (see [https://nixos.org/manual/nix/stable/installation/single-user.html Nix manual page]).
|-
| Single-user Nix installation || NixOS / System-wide Nix installation || ''Your'' user on your '''Local''' machine needs SSH access to the user on the '''remote''' machine with Nix installed with their UID (see [https://nixos.org/manual/nix/stable/installation/single-user.html Nix manual page]).
|}
 
In any case, the definitive test for SSH access for Nix is:
 
{{Commands|nix store ping --store ssh://<REMOTE-BUILDER>}}
 
Where {{ic|<REMOTE-BUILDER>}} is the remote builder's IP address, host address or whatever name you configure in {{ic|~/.ssh/config}} or {{ic|/root/.ssh/config}}, including the {{ic|user@}} prefix.
 
An alternative check, is the following:
 
{{Commands|ssh <REMOTE-BUILDER> 'type nix-store'}}
 
The following sections guide you how to setup such authentication, with security in mind, and maximal comfort, assuming basic knowledge about SSH authentication keys.
 
===General Best practices===
 
It is recommended to not allow {{ic|root}} access to the remote machine, even if only via an SSH public/private key pair! Especially because it's not required in any of the 4 scenarios described in the table above.
 
In all of the cases above it is recommended to create an SSH public / private key pair without a passphrase, so that you won't have to run {{ic|ssh-add}} along with {{ic|ssh-agent}} prior to using the remote builder. In cases the local machine has NixOS / System-wide installation of Nix you'd probably need to spawn {{ic|eval $(ssh-agent)}} while you are logged in as {{ic|root}}. Not using a passphrase for the SSH key allows other users to enjoy the remote builder.
 
Since the access to the remote machine doesn't have to be privileged, you can choose to login to a weakly privileged, and password-locked user on the remote machine, which may help you feel comfortable with the fact {{ic|root}} can access it without a passphrase.
 
{{Tip|The above assumes you have root access on the '''remote''' machine, that allows you to create such a weakly priviledged user, or that you have good relationship with your system administrator - which is the expected scenario if they have installed Nix system wide.}}
 
In case the remote machine doesn't have NixOS / System-wide Nix installation, you probably don't have a choice, but to allow access without passphrase and with an SSH key to the user with Nix installed for them.
 
===Recommended System-wide Nix -> System-wide Nix setup===
 
For the common case where your local Nix is installed system-wide. It is recommended to create a user on the '''remote''' machine, that will have an unwriteable home directory, with a {{ic|~/.ssh/authorized_keys}} in it, that will allow SSH access to that user without passphrase.


Here is a way to achieve this:
Here are steps that will help you to achieve that:
First we configure how ssh should connect to our builder.
 
{{file|~/.ssh/config|text|
- {{ic|ssh}} to the remote builder.
<nowiki>
- Run (requires priveledges) {{ic|useradd -m -L nixremote}} where {{ic|-L}} locks the user such that no body will be able to {{ic|su}} to it, and {{ic|-m}} makes sure a home directory is create for the {{ic|nixremote}} user.
Host builder
- Run (requires priveledges) {{ic|mkdir ~nixremote/.ssh}}.
        HostName 192.168.42.42
        Port 1234
        User foo


        # any other fancy option needed to log in
In case your '''remote builder''' has Nix installed system-wide, but without NixOS, you may need to add something like the following to your {{ic|/etc/ssh/sshd_config}}:
        # ProxyJump foo ...


        # Prevent using ssh-agent or another keyfile, useful for testing
{{file|/etc/ssh/sshd_config|text|
        IdentitiesOnly yes
<nowiki>
        IdentityFile /root/.ssh/nix_remote
SetEnv PATH=/nix/var/nix/profiles/default/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
</nowiki>
</nowiki>
}}
}}


{{Tip|When connecting to a new remote builder, ssh will ask you whether you trust the identity of the builder. Nix needs fully unattended connection, so you may want to accept all fingerprints by default. To achieve this you can use <code><nowiki>StrictHostKeyChecking=accept-new</nowiki></code>. Note that this has security implications if you usually really check the fingerprints when prompted to do so.}}
Such that straight away when logging in, your {{ic|$PATH}} will include nix' executables' directory.
 
Then, '''on your local machine''', create the private / public key pair without a passphrase, as root, run:


SSH connection must be non-interactive so we use a public key '''without a passphrase'''.
{{Commands|
{{Commands|
$ ssh-keygen -f ~/.ssh/nix_remote
# ssh-keygen -f /root/.ssh/nixremote
# do not add a passphrase to the ssh key!
}}
$ ssh-copy-id -i ~/.ssh/nix_remote builder
}}.


Make sure that your local user is added to <code>nix.trustedUsers</code>
Copy the contents of {{ic|/root/.ssh/nixremote.pub}} '''from your local machine''' to '''the remote builder''' {{ic|~nixremote/.ssh/authorized_keys}}.


When you are done, you can test your setup like this:
Then to further harden the setup, remove write permissions from everyone '''on the remote host''''s {{ic|nixremote}} home directory with:
{{Commands|<nowiki>$ nix store ping --store ssh://builder --extra-experimental-features nix-command && echo ok</nowiki>}}


If you get an error like <code>serialised integer ... is too big for type j</code> this means that something (for example <code>/etc/profile</code> or <code>environment.shellInit</code>) outputs bytes to <code>stdout</code> before launching the command specified on the ssh
{{Commands|
command line. Either disable this behavior or have the output be sent to <code>stderr</code> instead.
# chmod -R a-w ~nixremote
}}.


===== Non standard Nix installations =====
Now you'd want to make it easy for {{ic|root}} on '''your local machine''' to connect to {{ic|nixremote@builder}}. You can do that by creating the following {{ic|/root/.ssh/config}} on the '''Local''' machine:
If you are not root on the remote builder and have used '''nix-user-chroot''' or '''PRoot''' to install nix there (see [[Nix Installation Guide]]) then nix is not available on the PATH of the remote builder. We describe a solution for nix-user-chroot which is easily adapted to PRoot.
 
{{file|/root/.ssh/config|text|
* Create a script <code>~/bin/nix_wrapper.sh</code> as follows:
<nowiki>
<syntaxHighlight lang="sh">
Host builder # Replace by IP address, or add a ProxyCommand, see `man ssh_config` for full docs.
#!/bin/sh
        # Prevent using ssh-agent or another keyfile, useful for testing
exec ~/bin/nix-user-chroot ~/.nix bash -c '
        IdentitiesOnly yes
. ~/.nix-profile/etc/profile.d/nix.sh
        IdentityFile /root/.ssh/nixremote
exec $SSH_ORIGINAL_COMMAND
        # The weakly privileged user on the remote builder - if not set, `root` is used - which will hopefully fail
'
        User nixremote
</syntaxHighlight>
</nowiki>
Of course, adapt this script to the location of the store and nix-user-chroot. Make the script executable.
}}


* In <code>~/.ssh/authorized_keys</code>, locate the line corresponding to <code>~/.ssh/nix_remote.pub</code> and prepend this: <code>command="/home/something/bin/nix_wrapper.sh"</code>.
{{Tip|When connecting to a new remote builder, ssh will ask you whether you trust the identity of the builder. Nix needs fully unattended connection, so you may want to accept all fingerprints by default. To achieve this you can use <code><nowiki>StrictHostKeyChecking=accept-new</nowiki></code> in that config section. Note that this has security implications if you usually really check the fingerprints when prompted to do so. If you anticipate the host key won't change in the future, you can add it manually to '''the local machines'''' {{ic|/root/.ssh/known_hosts}}.}}


Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key.
You may also want to make nix on '''the remote machine''' trust that new user by adding it to {{ic|nix.trustedUsers}} if it's using NixOS, or by manually adding <code><nowiki>trusted-users = nixremote</nowiki></code> to {{ic|/etc/nix/nix.conf}}.


== Single user install ==
You may also want to tell Nix '''on your local machine''' that the remote builder is available, and tell it what are it's ''supported features'' (see [https://github.com/NixOS/nix/issues/7380 missing nix documentation issue]). If your '''local machine''', uses NixOS, you can use the documented [https://search.nixos.org/options?channel=unstable&from=0&size=15&sort=relevance&type=packages&query=nix.buildmachine {{ic|nix.buildMachines}}] NixOS option.
{{Expansion|untested}}
See the [https://nixos.org/nix/manual/#chap-distributed-builds Nix Manual] and the option <code>--builders</code>.


== Multi-User install ==
Here's an example NixOS configuration that may be a useful inspiration:
We must configure the <code>nix-daemon</code> to use our builder. Options like <code>--builders</code> on the command line is ignored unless your user is in the trusted user list.


=== NixOS ===
There are a few NixOS options we can use:
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{ config, pkgs, ... }:
{ config, pkgs, ... }:
Line 101: Line 127:
{{Evaluate}}
{{Evaluate}}


See the [https://nixos.org/nix/manual/#chap-distributed-builds Nix Manual] for the exact signification of each option.
==== Remote Builders' Features ====


=== Non NixOS ===
{{Expansion|untested}}
The previous method should be rather easily adaptable: replace adding NixOS options by editing <code>/etc/nix/nix.conf</code>.
== Using remote builders ==
==== Local builder ====
Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally.
To never use the local machine set the <code>max-jobs</code> nix option to 0
{{Commands|$ nix-build -j0 blah}}
==== Features ====
Each builder is declared with a set of <code>supportedFeatures</code>.
Each builder is declared with a set of <code>supportedFeatures</code>.
When a builder lacks one of the <code>requiredSystemFeatures</code> of a derivation, it will be ignored. Here are some features used in nixpkgs:
When a builder lacks one of the <code>requiredSystemFeatures</code> of a derivation, it will be ignored. Here are some features used in nixpkgs:
Line 134: Line 150:
|}
|}


To know what features a derivation needs, you can run {{Commands|$ nix show-derivation /nix/store/hash-foo.drv | grep requiredSystemFeatures}}
=== Non standard Nix installations ===
 
If you are not root on the remote builder and have used '''nix-user-chroot''' or '''PRoot''' to install nix there (see [[Nix Installation Guide]]) then nix is not available on the PATH of the remote builder. We describe a solution for nix-user-chroot which is easily adapted to PRoot.
 
* Create a script <code>~/bin/nix_wrapper.sh</code> as follows:
<syntaxHighlight lang="sh">
#!/bin/sh
exec ~/bin/nix-user-chroot ~/.nix bash -c '
. ~/.nix-profile/etc/profile.d/nix.sh
exec $SSH_ORIGINAL_COMMAND
'
</syntaxHighlight>
Of course, adapt this script to the location of the store and nix-user-chroot. Make the script executable.
 
* In <code>~/.ssh/authorized_keys</code>, locate the line corresponding to <code>~/.ssh/nixremote.pub</code> and prepend this: <code>command="/home/something/bin/nix_wrapper.sh"</code>.
 
Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key.
 
== Using remote builders ==
 
==== Local builder ====
 
Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally.
To never use the local machine set the <code>max-jobs</code> nix option to 0
{{Commands|$ nix-build -j0 blah}}


====Using remote builders as substituters====
====Using remote builders as substituters====
Line 170: Line 210:
* How do I know why my builds aren't being distributed?
* How do I know why my builds aren't being distributed?
** Run <code>nix build -vvvvvvvvv 2>&1 | less</code> and search for <code>decline</code>.
** Run <code>nix build -vvvvvvvvv 2>&1 | less</code> and search for <code>decline</code>.
* I can <code>nix ping-store</code> but the build doesn't distribute.
* I can <code>nix store ping</code> but the build doesn't distribute.
** If on NixOS, Check that <code>nix ping-store</code> command works when run as root.
** If on NixOS, Check that <code>nix store ping</code> command works when run as root.
** If you configured builders on the command line (with <code>--builders</code>), make sure your account is in <code>nix.trustedUsers</code> in <code>/etc/nixos/configuration.nix</code>. Only <code>/etc/nix/nix.conf</code> is taken into account otherwise.
** If you configured builders on the command line (with <code>--builders</code>), make sure your account is in <code>nix.trustedUsers</code> in <code>/etc/nixos/configuration.nix</code>. Only <code>/etc/nix/nix.conf</code> is taken into account otherwise.
* I can ping the store as root, but I'm getting "broken pipe" errors when trying to distribute.
* I can ping the store as root, but I'm getting "broken pipe" errors when trying to distribute.

Revision as of 18:27, 29 May 2023

Sometimes you want to use a faster machine for building a nix derivation you want to use on a slower one. If you have ssh access to a machine where Nix (not necessarily NixOS) is installed, then you can offload building to this machine.

There is a dedicated chapter in the Nix Manual.

This is a step by step guide to setting up distributed builds, that includes some SSH tips that are out of scope for Nix Manual's chapter.

Prerequisites

SSH access to the remote builder must be setup for being able to distribute builds to it. The table below lists SSH access requirements for (hopefully) all the scenarios you might find yourself:

Local Machine Remote Builder Requirements
NixOS / System-wide Nix installation NixOS / System-wide Nix installation Local machine's root user needs SSH access to a user on the remote machine.
Single-user Nix installation NixOS / System-wide Nix installation Your user on your Local machine needs SSH access to a user on the remote machine.
NixOS / System-wide Nix installation Single-user Nix installation Local machine's root user needs SSH access to the user on the remote machine with Nix installed with their UID (see Nix manual page).
Single-user Nix installation NixOS / System-wide Nix installation Your user on your Local machine needs SSH access to the user on the remote machine with Nix installed with their UID (see Nix manual page).

In any case, the definitive test for SSH access for Nix is:

nix store ping --store ssh://<REMOTE-BUILDER>

Where <REMOTE-BUILDER> is the remote builder's IP address, host address or whatever name you configure in ~/.ssh/config or /root/.ssh/config, including the user@ prefix.

An alternative check, is the following:

ssh <REMOTE-BUILDER> 'type nix-store'

The following sections guide you how to setup such authentication, with security in mind, and maximal comfort, assuming basic knowledge about SSH authentication keys.

General Best practices

It is recommended to not allow root access to the remote machine, even if only via an SSH public/private key pair! Especially because it's not required in any of the 4 scenarios described in the table above.

In all of the cases above it is recommended to create an SSH public / private key pair without a passphrase, so that you won't have to run ssh-add along with ssh-agent prior to using the remote builder. In cases the local machine has NixOS / System-wide installation of Nix you'd probably need to spawn eval $(ssh-agent) while you are logged in as root. Not using a passphrase for the SSH key allows other users to enjoy the remote builder.

Since the access to the remote machine doesn't have to be privileged, you can choose to login to a weakly privileged, and password-locked user on the remote machine, which may help you feel comfortable with the fact root can access it without a passphrase.

In case the remote machine doesn't have NixOS / System-wide Nix installation, you probably don't have a choice, but to allow access without passphrase and with an SSH key to the user with Nix installed for them.

Recommended System-wide Nix -> System-wide Nix setup

For the common case where your local Nix is installed system-wide. It is recommended to create a user on the remote machine, that will have an unwriteable home directory, with a ~/.ssh/authorized_keys in it, that will allow SSH access to that user without passphrase.

Here are steps that will help you to achieve that:

- ssh to the remote builder. - Run (requires priveledges) useradd -m -L nixremote where -L locks the user such that no body will be able to su to it, and -m makes sure a home directory is create for the nixremote user. - Run (requires priveledges) mkdir ~nixremote/.ssh.

In case your remote builder has Nix installed system-wide, but without NixOS, you may need to add something like the following to your /etc/ssh/sshd_config:

/etc/ssh/sshd_config
SetEnv PATH=/nix/var/nix/profiles/default/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Such that straight away when logging in, your $PATH will include nix' executables' directory.

Then, on your local machine, create the private / public key pair without a passphrase, as root, run:

# ssh-keygen -f /root/.ssh/nixremote

Copy the contents of /root/.ssh/nixremote.pub from your local machine to the remote builder ~nixremote/.ssh/authorized_keys.

Then to further harden the setup, remove write permissions from everyone on the remote host's nixremote home directory with:

# chmod -R a-w ~nixremote

.

Now you'd want to make it easy for root on your local machine to connect to nixremote@builder. You can do that by creating the following /root/.ssh/config on the Local machine:

/root/.ssh/config
Host builder # Replace by IP address, or add a ProxyCommand, see `man ssh_config` for full docs.
        # Prevent using ssh-agent or another keyfile, useful for testing
        IdentitiesOnly yes
        IdentityFile /root/.ssh/nixremote
        # The weakly privileged user on the remote builder - if not set, `root` is used - which will hopefully fail
        User nixremote

You may also want to make nix on the remote machine trust that new user by adding it to nix.trustedUsers if it's using NixOS, or by manually adding trusted-users = nixremote to /etc/nix/nix.conf.

You may also want to tell Nix on your local machine that the remote builder is available, and tell it what are it's supported features (see missing nix documentation issue). If your local machine, uses NixOS, you can use the documented nix.buildMachines NixOS option.

Here's an example NixOS configuration that may be a useful inspiration:

/etc/nixos/configuration.nix
{ config, pkgs, ... }:

{
	nix.buildMachines = [ {
	 hostName = "builder";
	 system = "x86_64-linux";
         protocol = "ssh-ng";
	 # if the builder supports building for multiple architectures, 
	 # replace the previous line by, e.g.,
	 # systems = ["x86_64-linux" "aarch64-linux"];
	 maxJobs = 1;
	 speedFactor = 2;
	 supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
	 mandatoryFeatures = [ ];
	}] ;
	nix.distributedBuilds = true;
	# optional, useful when the builder has a faster internet connection than yours
	nix.extraOptions = ''
		builders-use-substitutes = true
	'';
}

Remote Builders' Features

Each builder is declared with a set of supportedFeatures. When a builder lacks one of the requiredSystemFeatures of a derivation, it will be ignored. Here are some features used in nixpkgs:

Feature Derivations requiring it
kvm Everything which builds inside a vm, like NixOS tests
nixos-test Machine can run NixOS tests
big-parallel kernel config, libreoffice, evolution, llvm and chromium.
benchmark Machine can generate metrics (Means the builds usually takes the same amount of time)

Non standard Nix installations

If you are not root on the remote builder and have used nix-user-chroot or PRoot to install nix there (see Nix Installation Guide) then nix is not available on the PATH of the remote builder. We describe a solution for nix-user-chroot which is easily adapted to PRoot.

  • Create a script ~/bin/nix_wrapper.sh as follows:
#!/bin/sh
exec ~/bin/nix-user-chroot ~/.nix bash -c '
. ~/.nix-profile/etc/profile.d/nix.sh
exec $SSH_ORIGINAL_COMMAND
'

Of course, adapt this script to the location of the store and nix-user-chroot. Make the script executable.

  • In ~/.ssh/authorized_keys, locate the line corresponding to ~/.ssh/nixremote.pub and prepend this: command="/home/something/bin/nix_wrapper.sh".

Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key.

Using remote builders

Local builder

Your local machine is still a builder, notably when connecting to remote builders fails, nix will fallback to building locally. To never use the local machine set the max-jobs nix option to 0

$ nix-build -j0 blah

Using remote builders as substituters

If you have two remote builders A and B (where A has higher speed than B), that a derivation foo.drv is already built on B, and that your local machine needs to build foo.drv, then it will:

  • build (possibly remotely) all the build dependencies of foo.drv
  • build foo.drv on A

Even if foo.drv is 'also' on A, you will still have to build the build dependencies of foo.drv before sending the build to A which will build it instantly since it is in cache.

To solve this problem, you can set up your remote builders as substituters. Every time (the local machine's) nix considers building a derivation, it will connect to the remote builders to check whether it is already available there. Here is how to set this up via ssh. See also Binary Cache for an alternative using http and nix-serve.

1. On the remote builder, create a binary cache key:

$ nix-store --generate-binary-cache-key builder-name cache-priv-key.pem cache-pub-key.pem

The private key must be readable only by the user running the build: ??? on multi-user installs, and the owner of /nix on single-user installs. builder-name is only here for your convenience to distinguish several public keys, it has no functional meaning.

2. On the remote builder, set up nix to sign all store paths it builds: in the nix configuration (/etc/nix/nix.conf on multi-user installs and ~/.config/nix/nix.conf on single user installs), add the following line

secret-key-files = /path/to/cache-priv-key.pem

If necessary, restart the nix daemon.

3. The previous point does not retroactively sign existing paths in the store of the builder. To do so, run

$ nix sign-paths --all -k /path/to/cache-priv-key.pem

4. In the nix configuration of the local machine, append the content of cache-pub-key.pem to the option trusted-public-keys. Also append ssh-ng://builder to the option substituters. If you only want to use the remote builder occasionally as a substituter, use trusted-substituters instead of substituters. Then, when you want to use the builder, pass --option extra-substituters ssh-ng://builder to the nix command you run.

Troubleshooting

  • How do I know if I'm distributing my build at all?
    • Run nix build with --max-jobs 0.
  • How do I know why my builds aren't being distributed?
    • Run nix build -vvvvvvvvv 2>&1 | less and search for decline.
  • I can nix store ping but the build doesn't distribute.
    • If on NixOS, Check that nix store ping command works when run as root.
    • If you configured builders on the command line (with --builders), make sure your account is in nix.trustedUsers in /etc/nixos/configuration.nix. Only /etc/nix/nix.conf is taken into account otherwise.
  • I can ping the store as root, but I'm getting "broken pipe" errors when trying to distribute.
    • You may have hit bug #46038. Add nix.distributedBuilds = true; to configuration.nix and nixos-rebuild switch.

See also

See also: