View source for NixOS virtual machines on macOS

NixOS is based on Linux, but a lot of developers are using MacBooks. The [https://github.com/LnL7/nix-darwin nix-darwin] project aims to bring the convenience of a declarative system approach to macOS and it is built around Nixpkgs (much like NixOS). After installing Nix, <code>nix-darwin</code> will help us with the following goals:

# Build a complete NixOS (Linux) system.
# Build this system as a virtual machine image.
# Create a script that runs such an image locally using QEMU-KVM.
# Build integration tests based on running virtual machines.

For point 1, the challenge is to build Linux binaries from MacOS. This can be
solved by using the <code>linux-builder</code> feature provided by the <code>nix-darwin</code>
project. (A similar solution is possible without <code>nix-darwin</code>, but requires
additional manual setup.) Once we have access to a Linux builder, we need to
use the <code>system</code> configuration option natively supported by Nix.

Point 2 is similar to point 1.

Point 3 is similar to point 2, except that it requires a script and a local
QEMU KVM. This means that we need to configure the <code>system</code> option to be
<code>aarch64-linux</code> for the system we are building, but leave it as
<code>aarch64-darwin</code> for the script + QEMU-KVM part.

Once done, we can run the resulting Linux virtual machine on a MacOS M1. Note
that the Linux builder is actually such a virtual machine.

Point 4 is similar to point 3, but since the tests are '''built''' using virtual
machines, it means that these virtual machines are running inside the Linux
builder. This is not a problem if our environment supports nested
virtualization.

Conversely, it's a problem if it doesn't support nested virtualization. As it
happens, M1 doesn't support nested virtualization. So it can run a Linux
builder, or any other NixOS virtual machine, but it cannot do so inside e.g.
the Linux builder.

Another situation where this is a problem is with the M1 runners provided by
GitHub: the runner is itself a virtual machine running on M1, so we can't even
start a Linux builder in a GitHub workflow.

GitHub has announced support for <code>macos-15-xlarge</code> in Q4 2024.

Before understanding the above limitations, I experimented on an OakHost M2.S
machine. The notes that follow are the result of this experiment.

== Nix on M2.S ==

OakHost is a company that specializes in providing MacOS machines. The machines
are usually rented by the month, but they also have a 1-week offer:
[https://www.oakhost.net/try-macos "try-macos"] M2.S. After paying, you get an
IP address and a password to SSH into the machine. The username is "customer".

<syntaxhighlight lang="console">
$ ssh customer@xxx.xxx.xxx.xxx
</syntaxhighlight>

I haven't done this process enough times to write a script, but the steps to
get Nix and <code>nix-darwin</code> installed, and then configured to get a Linux builder,
are as follows.

Allow our user to use <code>sudo</code> without retyping the password each time:

<syntaxhighlight lang="console">
% export NIX_USER="customer"
% echo "%admin ALL = NOPASSWD: ALL" | sudo tee /etc/sudoers.d/passwordless
</syntaxhighlight>

Install Nix by downloading the install script, making it executable, and
running it (we cat <code>/dev/null</code> so the install script doesn't run
interactively):

<syntaxhighlight lang="console">
% curl -sL https://nixos.org/releases/nix/nix-2.9.2/install >/Users/${NIX_USER}/install-nix
% chmod +x /Users/$NIX_USER/install-nix
% cat /dev/null | sudo -i -H -u $NIX_USER -- sh /Users/${NIX_USER}/install-nix --daemon
% rm /Users/${NIX_USER}/install-nix
</syntaxhighlight>

Once Nix is installed, we need a new shell to actually get access to the <code>nix</code>
commands. An alternative is to source the correct file:

<syntaxhighlight lang="console">
% . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
% nix --version
</syntaxhighlight>

== nix-darwin ==

[https://github.com/LnL7/nix-darwin <code>nix-darwin</code>] is a project to configure a
macOS system using Nix modules, similar to what is done in NixOS.

First configure the <code>nixpkgs</code> and <code>nix-darwin</code> channels, ...

<syntaxhighlight lang="console">
% sudo -i -H -u $NIX_USER -- nix-channel --add https://github.com/LnL7/nix-darwin/archive/master.tar.gz darwin
% sudo -i -H -u $NIX_USER -- nix-channel --add https://nixos.org/channels/nixpkgs-24.05-darwin nixpkgs
% sudo -i -H -u $NIX_USER -- nix-channel --update
</syntaxhighlight>

... then install <code>nix-darwin</code>:

<syntaxhighlight lang="console">
% installer=$(nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer --no-out-link)
% echo N | sudo -i -H -u $NIX_USER -- "$installer/bin/darwin-installer"
</syntaxhighlight>

The <code>nix-darwin</code> configuration is then located in
<code>/Users/customer/.nixpkgs/darwin-configuration.nix</code>.

== <code>linux-builder</code> ==

We can change the configuration file to enable <code>linux-builder</code>. Don't configure
it any further at this point; we want to download a pre-built <code>linux-builder</code>
from the official nix binary cache. Otherwise, we'll have to build it ourselves
(and we won't be able to, since it would require a <code>linux-builder</code>, which we
don't have yet).

<syntaxhighlight lang="nix">
    nix.package = pkgs.nix;
    nix.linux-builder.enable = true;
    nix.settings.trusted-users = [ "@admin" ];
</syntaxhighlight>

In my case, using the channel configured above didn't allow me to get a
pre-built <code>linux-builder</code> from the cache.

Instead I got this error; it means we're trying to build something that
requires a Linux builder and we don't have it (yet).

<syntaxhighlight lang="console">
% . /etc/static/zshrc
> darwin-rebuild switch
...
error: a 'aarch64-linux' with features {} is required to build '/nix/store/2931ap3xi18184kzw7ms9lyfm9ngxaag-X-Restart-Triggers-sshd.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}
</syntaxhighlight>

We can force a specific <code>nixpkgs</code> (and hope that the corresponding
<code>linux-builder</code> is in the cache). This worked at the time of this writing:

<syntaxhighlight lang="nix">
{ config, pkgs, ... }:
let
  system = "aarch64-darwin";
  pkgs-darwin = import (builtins.fetchTarball {
    # nixpkgs-24.05-darwin
    url = "https://github.com/nixos/nixpkgs/archive/bf32c404263862fdbeb6e5f87a4bcbc6a01af565.tar.gz";
    sha256 = "132bd16a1wp145wz4m16w2maz0md6y2hp0qn5x1102wkyr9gkk0n";
  }) { inherit system; };
in
{
...

  nix.linux-builder.package = pkgs-darwin.darwin.linux-builder;
...
}
</syntaxhighlight>

We can confirm that we can start the <code>linux-builder</code> with something like:

<syntaxhighlight lang="console">
> darwin-rebuild switch
> nix-build \
  --impure \
  --expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "uname" {} "uname -a > $out")'
> cat result
Linux localhost 6.6.45 #1-NixOS SMP Sun Aug 11 10:47:28 UTC 2024 aarch64 GNU/Linux
</syntaxhighlight>

Note: If you run the <code>nix-build</code> command above, you'll see logs similar to
these, indicating that the <code>linux-builder</code> is being used:

<syntaxhighlight lang="console">
building '/nix/store/hlvznfa3iwjsrxq3jpn4j4li2pnqbfki-uname.drv' on 'ssh-ng://builder@linux-builder'...
...
copying path '/nix/store/wl2b6fy25y11kl5fbg928j18c7mj26b5-uname' from 'ssh-ng://builder@linux-builder'...
</syntaxhighlight>

Note: I think the <code>--option sandbox false</code> is not passed to the Linux builder
when using <code>nix-build</code>. So we set it in the above configuration file.

== <code>linux-builder</code>, further configuration ==

Remote builders can be configured with additional features. Here, we enable the
"kvm", and "nixos-test" features, and disable the sandbox. We also suggest
additional tweaks and exposing the builder's logs:

<syntaxhighlight lang="nix">
  nix.linux-builder = {
    enable = true;
    config = {
      nix.settings.sandbox = false;
    };
    ephemeral = true;
    package = pkgs-darwin.darwin.linux-builder;
    maxJobs = 4;
    supportedFeatures = [ "kvm" "benchmark" "big-parallel" "nixos-test" ];
  };
  nix.package = pkgs.nix;
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
  nix.settings.system-features = [ "nixos-test" "apple-virt" ];
  nix.settings.trusted-users = [ "@admin" ];

  launchd.daemons.linux-builder = {
    serviceConfig = {
      StandardOutPath = "/var/log/darwin-builder.log";
      StandardErrorPath = "/var/log/darwin-builder.log";
    };
  };
</syntaxhighlight>

Note: Notice how further configuring the Linux builder uses the running Linux
builder to build the new one.

== <code>direnv</code> ==

Optionally, we might want to use <code>direnv</code>. <code>nix-darwin</code> also makes it easy to
get it installed:

<syntaxhighlight lang="nix">
  programs.direnv.enable = true;
  programs.direnv.nix-direnv.enable = true;
</syntaxhighlight>

(Don't forget to run again <code>darwin-rebuild switch</code> after changing the
configuration file.)

== Resetting an OakHost machine ==

It is a manual process to reset the OakHost machine to a state similar to when
you received it.

At first, I tried to follow these steps, but it was a dead end:

We shut down the machine:

<syntaxhighlight lang="console">
> sudo shutdown -h now
</syntaxhighlight>

The OakHost documentation recommands waiting 1 minute before proceeding with
the next step.

In the OakHost web interface, click the "Force Power Off" button, then the
"Open KVM Screen" button (located in the "Remote Access" tab).

You will be greeted with two main icons: "Macintosh HD" and "Options".

I choose "Options", then select "Continue", then a disk icon on the right to
boot the machine, then "Reinstall macOS Sonoma", then. "Continue" and
"Continue".

After a while, you'll get an "OakHost Customer" login prompt. At this point,
SSH access is available again (with the same initial password).

But the <code>/etc/nix/</code> directory was still populated from my previous attempt !

What we want instead is to keep the machine running, then use the KVM to follow
these steps: From the Apple menu (the Apple logo), click "System settings...",
then "General", then "Transfer or Reset", and finally "Erase all contents and
settings...".

The machine reboots, and we can install the system manually. I skip everything
I can, including using an Apple ID. As a full name, the original was OakHost
Customer and the account name was "customer".

Once you're logged into the machine, don't forget to enable "Remote login" for
SSH in "System settings...", "General", "Sharing". At this point we disconnect
the KVM screen.

Someone at OakHost suggested disabling sleep mode.

<syntaxhighlight lang="console">
% sudo pmset -a displaysleep 0
% sudo systemsetup -setcomputersleep Off
% sudo pmset -a hibernatemode 0
</syntaxhighlight>

This is not necessary for us, but they also added this:

<blockquote>
One additional detail: If you want to sign in to the Mac using your iCloud
account, please first run the following line to disable the "Find My Mac”
module. This won’t impact any iCloud services. Otherwise resetting the device
might prove problematic later on due to theft protection:
<syntaxhighlight lang="console">
> sudo defaults write /Library/Preferences/com.apple.icloud.managed.plist DisableFMMiCloudSetting -bool true
</syntaxhighlight>
</blockquote>

The original machine had also Screen Sharing enabled.

== References ==

* "Build and Deploy Linux Systems from macOS" on the [https://nixcademy.com/posts/macos-linux-builder/ Nixcademy Blog].

[[Category:macOS]]
[[Category:nix-darwin]]
[[Category:Tutorial]]