NixOS on ARM/Raspberry Pi 5: Difference between revisions

Line 31:

== Status ==

Support for the Raspberry Pi 5 is currently downstream and experimental. It will not be fully supported until mature support exists in upstream Linux and U-Boot, but using UEFI already provides a workable solution.

Support for the Raspberry Pi 5 is currently downstream and experimental.

It will not be fully supported until mature support exists in the upstream Linux kernel and

the [https://github.com/u-boot/u-boot U-Boot] boot loader, but using an UEFI bootloader already provides a workable solution.

The Raspberry Pi 5's boot process follows the [https://youtu.be/UvFG76qM6co?si=e9O1s9mS3wpIpEl9&t=308 typical boot stages on embedded devices],

some of which have to be adapted to work with NixOS:

1. ROM boot loader: The first-stage boot loader comes "burned in" on the Pi in a tiny

[https://de.wikipedia.org/wiki/One_Time_Programmable One-Time-Programmable memory (OTP)]

so it cannot be changed anymore. It's only able to load the next second-stage boot loader below, and reset it in case you have messed up.

See the [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#first-stage-bootloader official documentation].

Nothing to adapt here.

2. EEPROM boot loader: The second-stage boot loader comes built-in on the Pi in a bit larger, rewriteable

[https://en.wikipedia.org/wiki/EEPROM EEPROM] memory

This loader is also very limited is only able to search for and start yet another, third-stage boot loader from other

storage hardware like an SD card, an NVME SSD, an USB disk, or from the network.

This loader (as many other second-stage boot loaders of other devices) is so size-constraint that it only contains

the bare mimum code to be able to read from an FAT formatted partition. That's why you see and want a separate small

<code>/boot</code> partition on your SD card or SSD that is formatted "FAT" or "VFAT", while your main data is stored on a second

"rootfs" or <code>/</code> partition with fancy, newer partition types like "ext4", "ZFS" or "btrfs".

See the [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#second-stage-bootloader official documentation].

See [https://github.com/raspberrypi/rpi-eeprom/releases EEPROM image releases] for improved and wider hardware support.

This boot loader can be updated via the <code>rpi-eeprom-update</code> terminal tool

(also [https://search.nixos.org/packages?channel=unstable&type=packages&query=raspberrypi-eeprom available in Nixpkgs])

and loads the binary images (the <code>firmware-2712/pieeprom-*.bin</code> files) from the

[https://github.com/raspberrypi/rpi-eeprom/tree/master/firmware-2712 rpi-eeprom Github project].

Nothing to adapt here yet. However, there's a

[https://github.com/raspberrypi/firmware/issues/1857 feature request to support smaller

third-stage boot loaders in this second-stage].

3. Firmware boot loader: The third-stage boot loader is loaded from the first partition (usually called <code>/boot</code>)

of an SD card, NVME SSD or other storage hardware described above.

Because size is usually not an issue here anymore you can have large, fully-fledged boot loaders like

[https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html systemd-boot] (default with NixOS; requires UEFI), or full

[https://github.com/u-boot/u-boot U-Boot] (popular with embedded devices like the Pi) or

[https://www.gnu.org/software/grub/ GRUB] (generally popular with Linux Distros).

However, the standard Pi 5 setup has no third-stage boot loader.

The second stage EEPROM boot loader loads the firmware (code to control other hardware on the Pi 5; [https://en.wikipedia.org/wiki/Devicetree device tree

files] in compact binary format <code>*.dtb</code>), some settings (<code>cmdline.txt</code> on which partition to find the rest of the system;

<code>config.txt</code> for general boot settings),

and directly the Linux kernel from a <code>/boot/firmware/</code> folder. On the Pi 5's default Debian image this is

the <code>kernel2712.img</code> (specialized, more-performant kernel

named after the Pi 5's Brodcom BCM2712 ARMv8 SoC chip) or as a fallback the <code>kernel8.img</code> (generic, slower ARMv8 kernel for the Pi 4 that

also works for Pi 5) that you find on the [https://github.com/raspberrypi/firmware/tree/master/boot Pi Firmware Github project].

See the [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#differences-on-raspberry-pi-5 official documentation].

Adaptions for NixOS:

In order to boot NixOS with it's default boot loader systemd-boot we need UEFI support so replace the

{

1. ROM -> 2. EEPROM -> Kernel -> NixOS

}

workflow with third-stage and fourth-stage boot loaders (both as firmware on an SD card or NVME SSD etc.)

{

1. ROM -> 2. EEPROM -> 3. EDK2 (UEFI boot loader) -> 4. systemd-boot -> Kernel -> NixOS

}

as follows:

1. Install EDK2:

We need the first partition of the SD card (or NVME SSD, etc.) again to be formatted as FAT

but labelled <code>ESP</code> (EFI System Partition) to conform to (U)EFI standards.

Into this partition we need to place the EDK2 boot loader file <code>RPI_EFI.fd</code>

and a <code>config.txt</code> file with a line <code>armstub=RPI_EFI.fd</code>

which instructs the EEPROM boot loader to load our EDK2 boot loader instead.

See the [https://github.com/worproject/rpi5-uefi EDK2 for Pi 5 Github project]; the releases already

contain both these files.

See a [https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1908664693 guide on how to setup partitions and these files].

2. Install systemd-boot, kernel and NixOS:

The rest is usual NixOS installation on a second partition with the caveat to select a Linux kernel that supports the Pi 5;

a suitable kernel can be created using [https://gitlab.com/vriska/nix-rpi5/-/blob/main/linux-rpi.nix leo60228's flake] until

that [https://github.com/NixOS/nixpkgs/pull/284391 Pi 5 compatible Linux kernel is available in Nixos-hardware].

Follow [https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1936211154 this guide] to build

a NixOS system closure that you can install manually onto the Pi with a <code>nixos-install</code> call.

That install tool will install the systemd-boot loader at <code>/boot/EFI/systemd/systemd-bootaa64.efi</code>

and the kernel files at <code>/boot/EFI/nixos/*.efi</code> onto your first ESP partition

and the rest of the NixOS system into your second partition.

Note that building that NixOS system closure has to happen natively, e.g. on a similar aarch64/Arm64 system like newer Macs (fast), or with

the [https://docs.kernel.org/admin-guide/binfmt-misc.html binfmt] kernel feature

(enabled with setting <code>boot.binfmt.emulatedSystems = [ "aarch64-linux" ];</code>) using QEMU virtualization

on any NixOS computer(veeeeerry slow)

Missing:

- How to do faster cross-compilation without binfmt.

- How to create a convenient SD card image to not install anything manually.

- How to install U-Boot instead of EDK2 and systemd-boot.

=== GPU ===

@@ Line 31: / Line 31: @@
 == Status ==
-Support for the Raspberry Pi 5 is currently downstream and experimental. It will not be fully supported until mature support exists in upstream Linux and U-Boot, but using UEFI already provides a workable solution.
+Support for the Raspberry Pi 5 is currently downstream and experimental.
+It will not be fully supported until mature support exists in the upstream Linux kernel and
+the [https://github.com/u-boot/u-boot U-Boot] boot loader, but using an UEFI bootloader already provides a workable solution.
+The Raspberry Pi 5's boot process follows the [https://youtu.be/UvFG76qM6co?si=e9O1s9mS3wpIpEl9&t=308 typical boot stages on embedded devices],
+some of which have to be adapted to work with NixOS:
+. ROM boot loader: The first-stage boot loader comes "burned in" on the Pi in a tiny
+[https://de.wikipedia.org/wiki/One_Time_Programmable One-Time-Programmable memory (OTP)]
+so it cannot be changed anymore. It's only able to load the next second-stage boot loader below, and reset it in case you have messed up.
+See the [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#first-stage-bootloader official documentation].
+Nothing to adapt here.
+. EEPROM boot loader: The second-stage boot loader comes built-in on the Pi in a bit larger, rewriteable
+[https://en.wikipedia.org/wiki/EEPROM EEPROM] memory
+This loader is also very limited is only able to search for and start yet another, third-stage boot loader from other
+storage hardware like an SD card, an NVME SSD, an USB disk, or from the network.
+This loader (as many other second-stage boot loaders of other devices) is so size-constraint that it only contains
+the bare mimum code to be able to read from an FAT formatted partition. That's why you see and want a separate small
+<code>/boot</code> partition on your SD card or SSD that is formatted "FAT" or "VFAT", while your main data is stored on a second
+"rootfs" or <code>/</code> partition with fancy, newer partition types like "ext4", "ZFS" or "btrfs".
+See the [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#second-stage-bootloader official documentation].
+See [https://github.com/raspberrypi/rpi-eeprom/releases EEPROM image releases] for improved and wider hardware support.
+This boot loader can be updated via the <code>rpi-eeprom-update</code> terminal tool
+(also [https://search.nixos.org/packages?channel=unstable&type=packages&query=raspberrypi-eeprom available in Nixpkgs])
+and loads the binary images (the <code>firmware-2712/pieeprom-*.bin</code> files) from the
+[https://github.com/raspberrypi/rpi-eeprom/tree/master/firmware-2712 rpi-eeprom Github project].
+Nothing to adapt here yet. However, there's a
+[https://github.com/raspberrypi/firmware/issues/1857 feature request to support smaller
+third-stage boot loaders in this second-stage].
+. Firmware boot loader: The third-stage boot loader is loaded from the first partition (usually called <code>/boot</code>)
+of an SD card, NVME SSD or other storage hardware described above.
+Because size is usually not an issue here anymore you can have large, fully-fledged boot loaders like
+[https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html systemd-boot] (default with NixOS; requires UEFI), or full
+[https://github.com/u-boot/u-boot U-Boot] (popular with embedded devices like the Pi) or
+[https://www.gnu.org/software/grub/ GRUB] (generally popular with Linux Distros).
+However, the standard Pi 5 setup has no third-stage boot loader.
+The second stage EEPROM boot loader loads the firmware (code to control other hardware on the Pi 5; [https://en.wikipedia.org/wiki/Devicetree device tree
+files] in compact binary format <code>*.dtb</code>), some settings (<code>cmdline.txt</code> on which partition to find the rest of the system;
+<code>config.txt</code> for general boot settings),
+and directly the Linux kernel from a <code>/boot/firmware/</code> folder. On the Pi 5's default Debian image this is
+the <code>kernel2712.img</code> (specialized, more-performant kernel
+named after the Pi 5's Brodcom BCM2712 ARMv8 SoC chip) or as a fallback the <code>kernel8.img</code> (generic, slower ARMv8 kernel for the Pi 4 that
+also works for Pi 5) that you find on the [https://github.com/raspberrypi/firmware/tree/master/boot Pi Firmware Github project].
+See the [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#differences-on-raspberry-pi-5 official documentation].
+Adaptions for NixOS:
+In order to boot NixOS with it's default boot loader systemd-boot we need UEFI support so replace the
+{
+. ROM -> 2. EEPROM -> Kernel -> NixOS
+}
+workflow with third-stage and fourth-stage boot loaders (both as firmware on an SD card or NVME SSD etc.)
+{
+. ROM -> 2. EEPROM -> 3. EDK2 (UEFI boot loader) -> 4. systemd-boot -> Kernel -> NixOS
+}
+as follows:
+. Install EDK2:
+We need the first partition of the SD card (or NVME SSD, etc.) again to be formatted as FAT
+but labelled <code>ESP</code> (EFI System Partition) to conform to (U)EFI standards.
+Into this partition we need to place the EDK2 boot loader file <code>RPI_EFI.fd</code>
+and a <code>config.txt</code> file with a line <code>armstub=RPI_EFI.fd</code>
+which instructs the EEPROM boot loader to load our EDK2 boot loader instead.
+See the [https://github.com/worproject/rpi5-uefi EDK2 for Pi 5 Github project]; the releases already
+contain both these files.
+See a [https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1908664693 guide on how to setup partitions and these files].
+. Install systemd-boot, kernel and NixOS:
+The rest is usual NixOS installation on a second partition with the caveat to select a Linux kernel that supports the Pi 5;
+a suitable kernel can be created using [https://gitlab.com/vriska/nix-rpi5/-/blob/main/linux-rpi.nix leo60228's flake] until
+that [https://github.com/NixOS/nixpkgs/pull/284391 Pi 5 compatible Linux kernel is available in Nixos-hardware].
+Follow [https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1936211154 this guide] to build
+a NixOS system closure that you can install manually onto the Pi with a <code>nixos-install</code> call.
+That install tool will install the systemd-boot loader at <code>/boot/EFI/systemd/systemd-bootaa64.efi</code>
+and the kernel files at <code>/boot/EFI/nixos/*.efi</code> onto your first ESP partition
+and the rest of the NixOS system into your second partition.
+Note that building that NixOS system closure has to happen natively, e.g. on a similar aarch64/Arm64 system like newer Macs (fast), or with
+the [https://docs.kernel.org/admin-guide/binfmt-misc.html binfmt] kernel feature
+(enabled with setting <code>boot.binfmt.emulatedSystems = [ "aarch64-linux" ];</code>) using QEMU virtualization
+on any NixOS computer(veeeeerry slow)
+Missing:
+- How to do faster cross-compilation without binfmt.
+- How to create a convenient SD card image to not install anything manually.
+- How to install U-Boot instead of EDK2 and systemd-boot.
 === GPU ===