Distributed build: Difference between revisions
Malteneuss (talk | contribs) m Make ssh connection path setting easier to understand |
Malteneuss (talk | contribs) m Make overall structure clearer with explicit steps |
||
| Line 1: | Line 1: | ||
When your '''local machine''' is too slow or doesn't have the right CPU architecture or operating system for the Nix derivation you want to build, you can delegate the build to some other '''remote machine'''. | When your '''local machine''' is too slow or doesn't have the right CPU architecture or operating system for the Nix derivation you want to build, you can delegate the build to some other '''remote machine'''. For this you need | ||
# the '''Nix package manager installed on both machines'''; just follow the [https://nixos.org/download/ official installation instructions] (prefer the normal "multi-user" install. You don't need to run NixOS; any operating system like Debian, Ubuntu, Arch, MacOS or others where the Nix package manager can be installed, should work. | |||
# '''SSH access from the local to the remote machine'''. | |||
# '''modify the local machine's Nix config to know about the remote machine'''. | |||
There is a dedicated chapter in the [https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html Nix Manual] but it may be difficult to follow for beginners. | There is a dedicated chapter in the [https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html Nix Manual] but it may be difficult to follow for beginners. | ||
| Line 5: | Line 9: | ||
This is an easier, step-by-step guide to setting up a "'''remote builder'''" machine to create distributed builds, and includes some SSH tips that are out of scope for the Nix Manual chapter. | This is an easier, step-by-step guide to setting up a "'''remote builder'''" machine to create distributed builds, and includes some SSH tips that are out of scope for the Nix Manual chapter. | ||
== | == Setting up SSH == | ||
The main tool to connect to a remote builder, exchange files and trigger builds is SSH. | The main tool to connect to a remote builder, exchange files and trigger builds is SSH. | ||
| Line 34: | Line 38: | ||
The thing to know about the '''"Multi-user"''' installation is that '''Nix is installed with a "nix-daemon" background process that runs as root''' and actually manages the builds on your behalf. So when you call '''"nix build ..." as a non-root user, this is delegated to the nix-daemon''' process, which runs as root. And this process can further delegate the build to a remote builder; that's why the '''local machine's root user''' needs the SSH access. | The thing to know about the '''"Multi-user"''' installation is that '''Nix is installed with a "nix-daemon" background process that runs as root''' and actually manages the builds on your behalf. So when you call '''"nix build ..." as a non-root user, this is delegated to the nix-daemon''' process, which runs as root. And this process can further delegate the build to a remote builder; that's why the '''local machine's root user''' needs the SSH access. | ||
{{Tip|The best litmus test to ensure that the SSH access works for remote Nix builds is: | |||
nix store ping --store ssh://<REMOTE-BUILDER> | |||
Where | Where <REMOTE-BUILDER> is the remote builder's IP address, host address or whatever name you configure in ~/.ssh/config or /root/.ssh/config, including the user@ prefix.}} | ||
An alternative check is: | An alternative check is: | ||
| Line 66: | Line 70: | ||
- Run (requires privileges) {{ic|mkdir ~nixremote/.ssh}}. | - Run (requires privileges) {{ic|mkdir ~nixremote/.ssh}}. | ||
If your '''remote builder''' has Nix installed system-wide in multi-user mode, but you're not running NixOS, '''you may need to add something like the following to your''' {{ic|/etc/ssh/sshd_config}}: | If your '''remote builder''' has Nix installed system-wide in multi-user mode, but you're not running NixOS, '''you may need to add something like the following to your''' {{ic|/etc/ssh/sshd_config}} on this remote machine: | ||
{{file|/etc/ssh/sshd_config|text| | {{file|/etc/ssh/sshd_config|text| | ||
| Line 107: | Line 111: | ||
You may also want to make nix on '''the remote machine''' trust that new user by adding it to {{ic|nix.settings.trusted-users}} if it's using NixOS, or by manually adding <code><nowiki>trusted-users = nixremote</nowiki></code> to {{ic|/etc/nix/nix.conf}}. | You may also want to make nix on '''the remote machine''' trust that new user by adding it to {{ic|nix.settings.trusted-users}} if it's using NixOS, or by manually adding <code><nowiki>trusted-users = nixremote</nowiki></code> to {{ic|/etc/nix/nix.conf}}. | ||
== '''Modify the local machine's Nix config to know about the remote machine'''. == | |||
The Nix package manager '''on your local machine''' '''needs to know that the remote builder exists''' and what its ''supported features'' are. See [https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-system-features official supportedFeatures documentation]. | |||
If your '''local machine''' uses NixOS, you can mention the remote builder within a NixOS [https://search.nixos.org/options?channel=unstable&from=0&size=15&sort=relevance&type=packages&query=nix.buildmachine {{ic|nix.buildMachines}}] section. For example: | |||
{{file|/etc/nixos/configuration.nix|nix|<nowiki> | {{file|/etc/nixos/configuration.nix|nix|<nowiki> | ||
| Line 194: | Line 199: | ||
Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key. | Now ssh will transparently run nix-user-chroot when you connect to the remote builder with the specified ssh key. | ||
== | == Further use of remote builders == | ||
==== | ==== Force builds on remote builder ==== | ||
Your local machine is | Your local machine is also a builder, so when connecting to remote builders fails, Nix will fall back to building locally. | ||
To never use the local machine, set the <code>max-jobs</code> | To never use the local machine, set the <code>--max-jobs <n>/-j<n></code> Nix option to 0 as follows: | ||
{{Commands|$ nix-build -j0 blah}} | {{Commands|$ nix-build -j0 blah}} | ||