NixOS Wiki

NixOS on ARM/Raspberry Pi 5: Difference between revisions

← Older edit
Visual
m Add deploy and update section
No edit summary
 
(21 intermediate revisions by 6 users not shown)
Line 29: Line 29:
The Raspberry Pi family of devices is a series of single-board computers made by the Raspberry Pi Foundation. They are all based on Broadcom System-on-a-chip (SoCs).  
The Raspberry Pi family of devices is a series of single-board computers made by the Raspberry Pi Foundation. They are all based on Broadcom System-on-a-chip (SoCs).  


== Status ==
== Current Status ==
NixOS is not officially supported on the Raspberry Pi 5, and the community efforts to bring the operating system to the development board have been less successful than the ones targeting the previous generation, the [[NixOS on ARM/Raspberry Pi 4|Raspberry Pi 4]]. It is advisable, for the time being, that all critical projects continue using and deploying the older model, for which the support is significantly more robust.


'''NixOS works on Raspberry Pi 5 but is currently experimental'''.  
For the adventurous, there are multiple community efforts enabling NixOS on the board, with various degrees of user friendliness, support availability or robustness, some of which can be found in the ''Other solutions'' section below. Amongst them, the community has flagged one repository that seems to have achieved a reasonable trade-off between all the necessary requirements for such a project; this is described in the ''Proposed Solution'' section below. While by no means official, and its level of support still under development, it is a good enough starting ground for people wanting to take advantage of their Raspberry Pi 5.
NixOS doesn't run out-of-box, but relies on several tweaks on the boot process that are maintained by different individuals and spread over multiple repositories.


The Raspberry Pi 5's boot process follows the [https://youtu.be/UvFG76qM6co?t=308 typical boot stages on embedded devices], and has the following boot loader steps by default:
However, this solution reuses proprietary software distributed by the Raspberry Pi Foundation; which is highly criticized by the NixOS community, and the Linux community at large, for reasons including reproducibility, transparency, security, predictibility, etc. There are efforts in the community to address these issues as well, and the following sub-section is an overview into these efforts.
  1. ROM -> 2. EEPROM -> Pi-custom Firmware/Kernel -> PiOS (custom Debian)
'''The pain points for NixOS support''' are the Pi's [https://github.com/raspberrypi/rpi-eeprom custom EEPROM boot bootloader], its  [https://github.com/raspberrypi/firmware proprietary, closed-source firmware] (code to use to hardware components) and [https://github.com/raspberrypi/linux its separately maintained Linux kernel], all of which we would need to '''update, build and test constantly and separately''' from the other NixOS Linux kernel variations, '''which is a large, unmaintainable burden for the NixOS community''' when the Pi 5 is not the only supported SoC; so we don't want that.


Instead our goal is to '''migrate towards the standard, generic boot process''' with UEFI:
=== Generic UEFI boot support ===
   1. ROM -> 2. EEPROM -> 3. UEFI boot loader -> 4. systemd-boot boot loader -> Generic Kernel -> NixOS
NixOS doesn't run out-of-box, but relies on several tweaks on the boot process that are maintained by different individuals and spread over multiple repositories. The Raspberry Pi 5's boot process follows the [https://youtu.be/UvFG76qM6co?t=308 typical boot stages on embedded devices], and has the following boot loader steps by default:{{caption|align=center|{{mermaid|flowchart LR|
We can't affect the ROM and EEPROM boot loaders as they come built-in into the hardware. However, [https://de.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface UEFI] is also used for booting normal Intel/AMD computers, and the [https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html systemd-boot] boot loader is THE software that allows us to have and select from multiple NixOS generations on boot (and perform rollbacks if we messed up). '''The currently required manual steps are roughly listed further down below''' and require lots of Linux and NixOS understanding.
   ROM --> EEPROM
  EEPROM --> FK[Firmware / Kernel]
  FK --> OS[Raspberry Pi OS]
}}|The normal boot process of a Raspberry Pi 5|alt=A flowchart showing 4 boxes with text in each, one leading to another. In the first box, ROM. In the second, EEPROM. In the third, Firmware / Kernel. In the fourth, Raspberry Pi OS.}}


Alternatively, '''if you want to get up-and-running quickly in the mean-time''' there's a setup that '''reuses lots of the pi-custom software''' (non-desired approach but works)
The pain points for NixOS support are the Pi's [https://github.com/raspberrypi/rpi-eeprom custom EEPROM boot bootloader], its [https://github.com/raspberrypi/firmware proprietary, closed-source firmware] (code to use to hardware components) and [https://github.com/raspberrypi/linux its separately maintained Linux kernel], all of which we would need to update, build and test constantly and separately from the other NixOS Linux kernel variations, which is a large, unmaintainable burden for the NixOS community when the Pi 5 is not the only supported SoC.
  1. ROM -> 2. EEPROM -> Pi-custom Firmware/Kernel -> NixOS
and creates a hard disk image that you can flash onto an SD card or NVMe SSD. The heavy lifting has been done and is maintained at https://github.com/tstat/raspberry-pi-nix (just follow the example and hints at https://github.com/tstat/raspberry-pi-nix/issues/13) and has the best out-of-the-box experience. You have to remote-build many Nix packages, probably the kernel as well, yourself (e.g. using the [https://wiki.nixos.org/wiki/Distributed_build Pi without NixOS as an intermediate remote builder]) and that can take several hours though.


It will also likely get [https://github.com/u-boot/u-boot U-Boot] support soon (likely only interesting for people that need more exotic hardware support like hats):
The more sustainable goal would be to move towards UEFI support, which would mimic how desktop computers boot into the NixOS operating system, thus minimising the amount of bespoke maintenance needed for the development board. This is what this process would look like:
   1. ROM -> 2. EEPROM -> Pi-custom Firmware/Kernel -> U-Boot boot loader -> NixOS
{{caption|align=center|{{mermaid|flowchart LR|
   ROM --> EEPROM
  EEPROM --> UEFI[UEFI Bootloader]
  UEFI --> SD[Systemd Bootloader]
  SD --> LK[Generic Linux Kernel]
  LK --> OS[Raspberry Pi OS]
}}|The ideal Raspberry Pi 5 boot process|alt=A flowchart containing 5 boxes with text. In this order, each box contains the following: ROM, EEPROM, UEFI Bootloader, Systemd Bootloader, Generic Linux Kernel, and NixOS.}}


== Raspberry Pi Boot stages ==
The ROM and EEPROM can't be modified, as they are built into the hardware on the development board. However, [https://de.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface UEFI] is also used for booting normal Intel/AMD computers, and the [https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html systemd-boot] boot loader is THE software that allows us to have and select from multiple NixOS generations on boot (and perform rollbacks if we messed up). Further details about the steps involved in achieving this can be found in the ''Other Solutions'' section, under ''Generic UEFI Boot.''
 
Finally, there should also be [https://github.com/u-boot/u-boot U-Boot] support soon, as most development boards are widely supported by the project. In that case, the boot process would look like:
{{caption|align=center|{{mermaid|flowchart LR|
  ROM --> EEPROM
  EEPROM --> FK[Firmware / Kernel]
  FK --> U[U-Boot Bootloader]
  U --> OS[Raspberry Pi OS]
}}|Theoretical Raspberry Pi 5 U-Boot boot process|alt=A flowchart showing 5 boxes containing text. In this order, each box contains the following text: ROM, EEPROM, Firmware or Kernel, U-Boot, and NixOS.}}
 
 
== Proposed Solution ==
There is a repository aiming to develop a declarative way of defining Raspberry Pi setups, including, but not limited to, the Raspberry Pi 5. The project is called [https://github.com/nvmd/nixos-raspberrypi?tab=readme-ov-file nixos-raspberrypi], and so far seems to be successful at achieving functioning system builds. It also provides a binary cache, speeding up iteration and deployment.
 
Take note that this solution reuses the proprietary firmware distributed by The Raspberry Pi Foundation. If that's not desired, one of the ''Other Solutions'' may be preferable.
 
{{Note|The author of the following section acknowledges that this may not be the best (or even intended) process. These instructions are provided on a best effort basis. If you have a better set of instructions, please contribute them to this section, replacing this content, and removing this notice.}}Setting up a NixOS system on your Raspberry Pi 5 consists of a number of steps. First, it's important to get a SD Image installer onto the device. If you're not currently running on an ARM system (almost definitely the case), you have three options:
 
# Cross compile the image (Note: an editor should add how-to information);
# Hope that the compiled image is available in the cache;
# Bootstrap a build machine onto the Raspberry Pi 5.
 
=== Step 1: Building the SD Image ===
{{Note|It might be possible to do the following steps via the official SD installer as well. If you manage to do it, please mark this section as optional and remove this notice.}}The project provides build images as well, which are modified versions of the nix-hardware ones. Before getting into building one, in order to make use of the community binary caches, and potentially avoid rebuilding the image, make sure you are added to the trusted users in your own nix configuration:
{{File|||3=trusted-users = username|name=nix.conf|lang=conf}}
Or, alternatively, in your nixos configuration:
{{File|||3={
  nix.settings.trusted-users = [ "your-username" ];
<nowiki>}</nowiki>|name=configuration.nix|lang=nix}}
Afterwards, you can build one of the SD images:<syntaxhighlight lang="console">
$ nix build github:nvmd/nixos-raspberrypi#installerImages.rpi5
</syntaxhighlight>You can either [[Cross Compiling|cross-compile]] the image on your system, or alternatively, you can install the "normal" Raspberry Pi OS on your Raspberry Pi 5, then install Nix standalone (multi-user), and set it up as a [[distributed build]] machine.
 
Finally, simply copy the image to the current directory from the <code>result</code> directory, extract it and write it to a USB drive, and boot to this drive on the raspberry pi (by default the raspberry pi will first try to boot the SD card, then the USB drive, so just don't put any SD card when booting and plug it later. Note that since the installer will format the SD card, you cannot put the installer on the SD card itself). You can find more information on the [[NixOS on ARM/Installation]] page.
 
=== Step 2: Installing NixOS ===
 
Once you're running an installer image, you can create a flake referencing the repository. For example:{{File|3={
  description = "Example Raspberry Pi 5 configuration flake";
    inputs = {
      nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
      nixos-raspberrypi.url = "github:nvmd/nixos-raspberrypi/main";
    };
 
  nixConfig = {
    extra-substituters = [
      "https://nixos-raspberrypi.cachix.org"
    ];
    extra-trusted-public-keys = [
      "nixos-raspberrypi.cachix.org-1:4iMO9LXa8BqhU+Rpg6LQKiGa2lsNh/j2oiYLNOQ5sPI="
    ];
  };
 
  outputs = { self, nixpkgs, nixos-raspberrypi }@inputs:
    {
      nixosConfigurations = {
        yourHostname = nixos-raspberrypi.lib.nixosSystem {
          specialArgs = inputs;
          modules = [
            ({...}: {
              imports = with nixos-raspberrypi.nixosModules; [
                raspberry-pi-5.base
                raspberry-pi-5.bluetooth
              ];
            })
            ({ ... }: {
              networking.hostName = "yourHostName";
              users.users.yourUserName = {
                initialPassword = "yourInitialPassword";
                isNormalUser = true;
                extraGroups = [
                  "wheel"
                ];
              };
 
              services.openssh.enable = true;
            })
 
            ({ ... }: {
              fileSystems = {
                "/boot/firmware" = {
                  device = "/dev/disk/by-uuid/2175-794E";
                  fsType = "vfat";
                  options = [
                    "noatime"
                    "noauto"
                    "x-systemd.automount"
                    "x-systemd.idle-timeout=1min"
                  ];
                };
                "/" = {
                  device = "/dev/disk/by-uuid/44444444-4444-4444-8888-888888888888";
                  fsType = "ext4";
                  options = [ "noatime" ];
                };
              };
            })
          ];
        };
      };
    };
<nowiki>}</nowiki>|name=flake.nix|lang=nix}}Finally, simply <code>nixos-rebuild switch --flake .#yourHostname</code> the flake.
 
== Other Solutions ==
In general, as show in in the first section, the closed source and proprietary software powering part of the Raspberry Pi development boards is a big source of contention. Therefore, there are largely two kinds of solutions going forward: '''moving towards a generic boot process''', and '''using proprietary software''', each with advantages and disadvantages.
 
=== Using proprietary software ===
Besides the current ''Proposed Solution'', there has been one more attempt at https://github.com/tstat/raspberry-pi-nix (just follow the example and hints at https://github.com/tstat/raspberry-pi-nix/issues/13) and has the best out-of-the-box experience. You have to remote-build many Nix packages, probably the kernel as well, yourself (e.g. using the [https://wiki.nixos.org/wiki/Distributed_build Pi without NixOS as an intermediate remote builder]) and that can take several hours though.
 
=== Generic UEFI Boot ===
This solution is technically more complex than the others, and the overview is separated into sections.
 
==== Raspberry Pi Boot stages ====
To understand the adaptions for NixOS better, it's helpful to understand more about the stages:
To understand the adaptions for NixOS better, it's helpful to understand more about the stages:


Line 101: Line 218:
[https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#differences-on-raspberry-pi-5 official documentation].
[https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#differences-on-raspberry-pi-5 official documentation].


== Setting up a generic UEFI NixOS ==
==== Setting up a generic UEFI NixOS ====
The task to get a generic NixOS setup requires a
The task to get a generic NixOS setup requires a


# UEFI boot loader for Pi 5: There exists a WIP [https://github.com/worproject/rpi5-uefi EDK2 for Pi 5 Github project] but with several limitations like no boot from NVMe SSDs.
# UEFI boot loader for Pi 5: There exists a WIP [https://github.com/worproject/rpi5-uefi EDK2 for Pi 5 Github project] but with a few limitations see the project for details.
# systemd-boot boot loader: Works
# systemd-boot boot loader: Works
# generic Linux kernel that works for the Pi 5's ARM v8 processor and hardware: An almost generic Pi 5 compatible kernel exists [https://github.com/NixOS/nixos-hardware/blob/master/raspberry-pi/5/default.nix at the NixOS-hardware repository]; it's an adaption from a kernel for the Pi 4)  
# generic Linux kernel that works for the Pi 5's ARM v8 processor and hardware: An almost generic Pi 5 compatible kernel exists [https://github.com/NixOS/nixos-hardware/blob/master/raspberry-pi/5/default.nix at the NixOS-hardware repository]; it's an adaption from a kernel for the Pi 4)  
Line 131: Line 248:
and the rest of the NixOS system into your second partition.
and the rest of the NixOS system into your second partition.


== Alternative board-specific installation notes ==
==== Alternative board-specific installation notes ====
First, install EDK2, following the [https://github.com/worproject/rpi5-uefi#getting-started instructions from the port README]. With EDK2 installed as the Platform Firmware, you can follow the [[NixOS_on_ARM/UEFI|standard instructions for UEFI on ARM]].
First, install EDK2, following the [https://github.com/worproject/rpi5-uefi#getting-started instructions from the port README]. With EDK2 installed as the Platform Firmware, you can follow the [[NixOS_on_ARM/UEFI|standard instructions for UEFI on ARM]].


EDK2 enables booting a mainline kernel, but hardware support will be very limited. Notably, you'll need to perform the installation using Wi-Fi, as Ethernet is unsupported. Once the system is installed, you can switch to the vendor's modified kernel. This is not (yet?) available in Nixpkgs, so you'll need to get it from [https://gitlab.com/vriska/nix-rpi5 a flake]. If you're not using flakes, you can simply add this to your configuration:
EDK2 enables booting a mainline kernel, but hardware support will be very limited. Notably, you'll need to perform the installation using Wi-Fi, as Ethernet is unsupported. Once the system is installed, you can switch to the vendor's modified kernel. This is not (yet?) available in Nixpkgs, so you'll need to get it from [https://gitlab.com/vriska/nix-rpi5 a flake]. If you're not using flakes, you can simply add this to your configuration:


{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|3=<nowiki>
{
{
   boot.kernelPackages = (import (builtins.fetchTarball https://gitlab.com/vriska/nix-rpi5/-/archive/main.tar.gz)).legacyPackages.aarch64-linux.linuxPackages_rpi5;
   boot.kernelPackages = (import (builtins.fetchTarball https://gitlab.com/vriska/nix-rpi5/-/archive/main.tar.gz)).legacyPackages.aarch64-linux.linuxPackages_rpi5;
}
}
</nowiki>}}
</nowiki>|name=configuration.nix|lang=nix}}


For the vendor kernel to boot properly, you must switch from ACPI to Device Tree in the UEFI settings (at Device Manager → Raspberry Pi Configuration → ACPI / Device Tree → System Table Mode). When using the vendor kernel (which provides full power management support), you may additionally wish to remove <code>force_turbo=1</code> from <code>/boot/config.txt</code>.
For the vendor kernel to boot properly, you must switch from ACPI to Device Tree in the UEFI settings (at Device Manager → Raspberry Pi Configuration → ACPI / Device Tree → System Table Mode). When using the vendor kernel (which provides full power management support), you may additionally wish to remove <code>force_turbo=1</code> from <code>/boot/config.txt</code>.


If you are using nixos-unstable, then you can also use the rpi4 kernel (which is a generic aarch64 kernel for Pi 3 and later models). Although, due to a smaller page size, this will have slightly worse performance:
If you are using nixos-unstable, then you can also use the rpi4 kernel (which is a generic aarch64 kernel for Pi 3 and later models). Although, due to a smaller page size, this will have slightly worse performance:
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|||<nowiki>
{
{
   boot.kernelPackages = pkgs.linuxPackages_rpi4;
   boot.kernelPackages = pkgs.linuxPackages_rpi4;
}
}
</nowiki>}}
</nowiki>|name=configuration.nix|lang=nix}}


== Troubleshooting ==
==== Troubleshooting ====


=== GPU ===
===== GPU =====
For the GPU drivers to work, <code>dtoverlay=vc4-kms-v3d-pi5</code> must be added to <code>/boot/config.txt</code>, and the vendor kernel must currently be used. Only Wayland-based compositors are supported without additional configuration (see the nixos-hardware PR linked previously). Note that Xwayland applications may produce broken graphics on KDE; the root cause of this issue has not yet been evaluated.
For the GPU drivers to work, <code>dtoverlay=vc4-kms-v3d-pi5</code> must be added to <code>/boot/config.txt</code>, and the vendor kernel must currently be used. Only Wayland-based compositors are supported without additional configuration (see the nixos-hardware PR linked previously). Note that Xwayland applications may produce broken graphics on KDE; the root cause of this issue has not yet been evaluated.


The rpi5-uefi download does not include overlays. You can get them by copying the <code>boot/overlays</code> folder from [https://github.com/raspberrypi/firmware the firmware repository] to <code>/boot</code> (so that <code>/boot/overlays/vc4-kms-v3d-pi5.dtbo</code> is available).
The rpi5-uefi download does not include overlays. You can get them by copying the <code>boot/overlays</code> folder from [https://github.com/raspberrypi/firmware the firmware repository] to <code>/boot</code> (so that <code>/boot/overlays/vc4-kms-v3d-pi5.dtbo</code> is available).


=== Bluetooth ===
===== Bluetooth =====
If your Bluetooth doesn't show up, and you are getting errors in dmesg regarding the serial port at 107050c00, add the following to your NixOS configuration:{{file|/etc/nixos/configuration.nix|nix|<nowiki>
 
If your Bluetooth doesn't show up, and you are getting errors in dmesg regarding the serial port at 107050c00, add the following to your NixOS configuration:{{file|||<nowiki>
{
{
   boot.kernelParams = [ "8250.nr_uarts=11" "console=ttyAMA10,9600" "console=tty0"];
   boot.kernelParams = [ "8250.nr_uarts=11" "console=ttyAMA10,9600" "console=tty0"];
}
}
</nowiki>}}
</nowiki>|name=configuration.nix|lang=nix}}


== Using the Pi 5 as a remote builder to build native ARM packages for the Pi 5 ==
==== Using the Pi 5 as a remote builder to build native ARM packages for the Pi 5 ====
Building an NixOS system image that can be flashed to an SD card or NVMe SSD requires to '''build ARM binaries''', more specifically for the <code>"aarch64-linux"</code>platform. From a typical Intel/AMD computer we can either
Building an NixOS system image that can be flashed to an SD card or NVMe SSD requires to '''build ARM binaries''', more specifically for the <code>"aarch64-linux"</code>platform. From a typical Intel/AMD computer we can either


Line 181: Line 299:
Missing:
Missing:


# How to do cross-compilation.
# How to do cross-compilation. ( and/or explain why the above emulation is required at all)


== Deploy and Update the Pi 5 NixOS system once it's running NixOS ==
==== Deploy and Update the Pi 5 NixOS system once it's running NixOS ====
Once the Pi 5 is running NixOS, you can update it with newer NixOS system configurations using e.g. the usual  <code>nix-rebuild</code>   
Once the Pi 5 is running NixOS, you can update it with newer NixOS system configurations using e.g. the usual  <code>nix-rebuild</code>   


Line 193: Line 311:


See [https://nixcademy.com/2023/08/10/nixos-rebuild-remote-deployment/ this guide] for a good explanation of this terminal call.
See [https://nixcademy.com/2023/08/10/nixos-rebuild-remote-deployment/ this guide] for a good explanation of this terminal call.
[[Category:NixOS on ARM]]
Retrieved from "https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5"