Distributed build: Difference between revisions
Malteneuss (talk | contribs) m Fix sentence |
m →Modify the local machine's Nix config to know about the remote machine.: Remove erroneous bold header |
||
| (15 intermediate revisions by 6 users not shown) | |||
| Line 1: | Line 1: | ||
When your '''local machine''' is too slow or doesn't have the right CPU architecture or operating system for the Nix derivation you want to build, you can delegate the build to some other '''remote machine'''. | When your '''local machine''' is too slow or doesn't have the right CPU architecture or operating system for the Nix derivation you want to build, you can delegate the build to some other '''remote machine'''. For this you need | ||
# the '''Nix package manager installed on both machines'''; just follow the [https://nixos.org/download/ official installation instructions] and prefer the normal "multi-user" install. You don't need to run NixOS; any operating system like Debian, Ubuntu, Arch, MacOS or others where the Nix package manager can be installed, should work. | |||
# '''SSH access from the local to the remote machine'''. | |||
# '''modify the local machine's Nix config to know about the remote machine'''. | |||
There is a dedicated chapter in the [https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html Nix Manual] but it may be difficult to follow for beginners. | There is a dedicated chapter in the [https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html Nix Manual] but it may be difficult to follow for beginners. | ||
| Line 5: | Line 9: | ||
This is an easier, step-by-step guide to setting up a "'''remote builder'''" machine to create distributed builds, and includes some SSH tips that are out of scope for the Nix Manual chapter. | This is an easier, step-by-step guide to setting up a "'''remote builder'''" machine to create distributed builds, and includes some SSH tips that are out of scope for the Nix Manual chapter. | ||
== | == Setting up SSH == | ||
The main tool to connect to a remote builder, exchange files and trigger builds is SSH. | The main tool to connect to a remote builder, exchange files and trigger builds is SSH. | ||
| Line 23: | Line 27: | ||
! SSH Connection Requirements | ! SSH Connection Requirements | ||
|- | |- | ||
| '''Multi-user''' || '''Multi-user''' || '''Local''' | | '''Multi-user''' || '''Multi-user''' || '''Local:''' {{ic|root}} user ------------SSH----> '''Remote''': ''any'' user '''(most frequent case)''' | ||
|- | |- | ||
| Single-user || Multi-user || '' | | Single-user || Multi-user || '''Local:''' ''Your'' single-user -----SSH----> '''Remote''': ''any'' user | ||
|- | |- | ||
| Multi-user || Single-user || '''Local''' | | Multi-user || Single-user || '''Local:''' {{ic|root}} user ------------SSH----> '''Remote''': ''your'' single-user for which Nix is installed with their UID (see [https://nixos.org/manual/nix/stable/installation/single-user.html Nix manual page]). | ||
|- | |- | ||
| Single-user || | | Single-user || Single-user || '''Local:''' ''Your'' single-user -----SSH----> '''Remote''': ''your single-user'' each of which Nix is installed for with their UID (see [https://nixos.org/manual/nix/stable/installation/single-user.html Nix manual page]). | ||
|} | |} | ||
The thing to know about the '''"Multi-user"''' installation is that '''Nix is installed with a "nix-daemon" background process that runs as root''' and actually manages the builds on your behalf. So when you call '''"nix build ..." as a non-root user, this is delegated to the nix-daemon''' process, which runs as root. And this process can further delegate the build to a remote builder; that's why the '''local machine's root user''' needs the SSH access. | The thing to know about the '''"Multi-user"''' installation is that '''Nix is installed with a "nix-daemon" background process that runs as root''' and actually manages the builds on your behalf. So when you call '''"nix build ..." as a non-root user, this is delegated to the nix-daemon''' process, which runs as root. And this process can further delegate the build to a remote builder; that's why the '''local machine's root user''' needs the SSH access. | ||
{{Tip|The best test to check that the SSH access works for Nix is to run on your local machine: | |||
<code>nix store ping --store ssh://<REMOTE-BUILDER></code> | |||
Where | Where <code><REMOTE-BUILDER></code> is the remote builder's IP address, host address or whatever name you configure in ~/.ssh/config or /root/.ssh/config, including the user@ prefix.}} | ||
An alternative check is: | An alternative check is: | ||
| Line 58: | Line 62: | ||
When the remote machine doesn't have NixOS / System-wide Nix installation, the only option is to allow access without passphrase and with an SSH key to the user with Nix installed for them. | When the remote machine doesn't have NixOS / System-wide Nix installation, the only option is to allow access without passphrase and with an SSH key to the user with Nix installed for them. | ||
===Recommended | ===Recommended setup: multi-user Nix local –> multi-user Nix remote=== | ||
For the common case where your local Nix is installed system-wide, create a user on the '''remote''' machine that will have an unwriteable home directory, with a {{ic|~/.ssh/authorized_keys}} in it, that will allow SSH access to that user without a passphrase. The steps are: | For the common case where your local Nix is installed system-wide in multi-user mode, create a user on the '''remote''' machine that will have an unwriteable home directory, with a {{ic|~/.ssh/authorized_keys}} in it, that will allow SSH access to that user without a passphrase. The steps are: | ||
* {{ic|ssh}} to the remote builder. | |||
* Run (requires privileges) {{ic|useradd -m nixremote}}; {{ic|-m}} makes sure a home directory is created for the {{ic|nixremote}} user. | |||
* Run (requires privileges) {{ic|usermod nixremote -L}}; {{ic|-L}} locks the user such that nobody will be able to {{ic|su}} to it | |||
* Run (requires privileges) {{ic|mkdir ~nixremote/.ssh}}. Make sure to run this command as {{ic|nixremote}} user or {{ic|chown}} it afterwards | |||
If your '''remote builder''' has Nix installed system-wide, but | If your '''remote builder''' has Nix installed system-wide in multi-user mode, but you're not running NixOS, '''you may need to add something like the following to your''' {{ic|/etc/ssh/sshd_config}} on this remote machine: | ||
{{file|/etc/ssh/sshd_config|text| | {{file|/etc/ssh/sshd_config|text| | ||
| Line 74: | Line 79: | ||
}} | }} | ||
Explanation: This extends the {{ic|$PATH}} variable on your remote builder for your ssh connection such that the installed Nix tools like {{ic|/var/nix/var/nix/profiles/default/bin/nix-store}} can be found on this remote builder when connecting through ssh from your local machine. Otherwise you will get an error on your local machine like "ssh.. nix-store: command not found". The reason is that the Nix ssh connection uses an "non-interactive" shell on the remote builder that doesn't load any {{ic|.bashrc}} files like a normal "interactive" shell would do, when connect manually. | |||
Then, '''on your local machine''', create the private / public key pair without a passphrase, as root: | Then, '''on your local machine''', create the private / public key pair without a passphrase, as root: | ||
| Line 107: | Line 112: | ||
You may also want to make nix on '''the remote machine''' trust that new user by adding it to {{ic|nix.settings.trusted-users}} if it's using NixOS, or by manually adding <code><nowiki>trusted-users = nixremote</nowiki></code> to {{ic|/etc/nix/nix.conf}}. | You may also want to make nix on '''the remote machine''' trust that new user by adding it to {{ic|nix.settings.trusted-users}} if it's using NixOS, or by manually adding <code><nowiki>trusted-users = nixremote</nowiki></code> to {{ic|/etc/nix/nix.conf}}. | ||
== Modify the local machine's Nix config to know about the remote machine. == | |||
The Nix package manager '''on your local machine''' '''needs to know that the remote builder exists''' and what its ''supported features'' are. See [https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-system-features official supportedFeatures documentation]. | |||
If your '''local machine''' uses NixOS, you can mention the remote builder within a NixOS [https://search.nixos.org/options?channel=unstable&from=0&size=15&sort=relevance&type=packages&query=nix.buildmachine {{ic|nix.buildMachines}}] section. For example: | |||
{{file|/etc/nixos/configuration.nix|nix|<nowiki> | {{file|/etc/nixos/configuration.nix|nix|<nowiki> | ||
| Line 129: | Line 135: | ||
# - M1, M2, M3 ARM Macs use "aarch64-darwin" | # - M1, M2, M3 ARM Macs use "aarch64-darwin" | ||
# - Newer RISCV computers use "riscv64-linux" | # - Newer RISCV computers use "riscv64-linux" | ||
# See https://github.com/NixOS/nixpkgs/blob/nixos-unstable/lib/systems/flake-systems.nix | |||
# If your builder supports multiple architectures | # If your builder supports multiple architectures | ||
# (e.g. search for "binfmt" for emulation), | # (e.g. search for "binfmt" for emulation), | ||
| Line 134: | Line 141: | ||
# systems = ["x86_64-linux" "aarch64-linux" "riscv64-linux"]; | # systems = ["x86_64-linux" "aarch64-linux" "riscv64-linux"]; | ||
system = "x86_64-linux"; | system = "x86_64-linux"; | ||
# Nix custom ssh-variant that avoids lots of "trusted-users" settings pain | |||
protocol = "ssh-ng"; | protocol = "ssh-ng"; | ||
# default is 1 but may keep the builder idle in between builds | # default is 1 but may keep the builder idle in between builds | ||
| Line 145: | Line 153: | ||
nix.distributedBuilds = true; | nix.distributedBuilds = true; | ||
# optional, useful when the builder has a faster internet connection than yours | # optional, useful when the builder has a faster internet connection than yours | ||
nix. | nix.settings = { | ||
builders-use-substitutes = true | builders-use-substitutes = true; | ||
}; | |||
} | } | ||
</nowiki>}} | </nowiki>}} | ||
| Line 193: | Line 201: | ||
Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key. | Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key. | ||
== | == Further use of remote builders == | ||
==== | ==== Force builds on remote builder ==== | ||
Your local machine is | Your local machine is also a builder, so when connecting to remote builders fails, Nix will fall back to building locally. | ||
To never use the local machine, set the <code>max-jobs</code> | To never use the local machine, set the <code>--max-jobs <n>/-j<n></code> Nix option to 0 as follows: | ||
{{Commands|$ nix-build -j0 blah}} | {{Commands|$ nix-build -j0 blah}} | ||
| Line 243: | Line 251: | ||
== See also == | == See also == | ||
* [https://github.com/NixOS/nix/blob/ | * [https://github.com/NixOS/nix/blob/a6e6da3b0c579fc540acb00748fe3fd1858b9d99/tests/nixos/remote-builds.nix#L11-L21 The NixOS Remote Builds Test Case] | ||
* [https://nixos.org/nix-dev/2015-September/018255.html Mail to nixos-dev about setting up remote builds by Russell O'Connor] | * [https://nixos.org/nix-dev/2015-September/018255.html Mail to nixos-dev about setting up remote builds by Russell O'Connor] | ||
* [https://gist.github.com/danbst/09c3f6cd235ae11ccd03215d4542f7e7 A step-by-step guide on remote Firefox building through bastion host] | * [https://gist.github.com/danbst/09c3f6cd235ae11ccd03215d4542f7e7 A step-by-step guide on remote Firefox building through bastion host] | ||
* [https://sgt.hootr.club/molten-matter/nix-distributed-builds/ Offloading NixOS builds to a faster machine] | * [https://sgt.hootr.club/molten-matter/nix-distributed-builds/ Offloading NixOS builds to a faster machine] | ||
* [https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder Run a qemu Linux builder on macOS] | * [https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder Run a qemu Linux builder on macOS] | ||
[[Category:Nix]] | |||
[[Category:Guide]] | |||