NixOS on ARM/Raspberry Pi 5: Difference between revisions
Malteneuss (talk | contribs) Give overall page more structure and mention working undesirable but already working NixOS setup |
Malteneuss (talk | contribs) Add remote builder section |
||
| Line 42: | Line 42: | ||
We can't affect the ROM and EEPROM boot loaders as they come built-in into the hardware. However, [https://de.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface UEFI] is also used for booting normal Intel/AMD computers, and the [https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html systemd-boot] boot loader is THE software that allows us to have and select from multiple NixOS generations on boot (and perform rollbacks if we messed up). '''The currently required manual steps are roughly listed further down below''' and require lots of Linux and NixOS understanding. | We can't affect the ROM and EEPROM boot loaders as they come built-in into the hardware. However, [https://de.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface UEFI] is also used for booting normal Intel/AMD computers, and the [https://www.freedesktop.org/software/systemd/man/latest/systemd-boot.html systemd-boot] boot loader is THE software that allows us to have and select from multiple NixOS generations on boot (and perform rollbacks if we messed up). '''The currently required manual steps are roughly listed further down below''' and require lots of Linux and NixOS understanding. | ||
Alternatively, '''if you want to get up-and-running quickly in the mean-time''' there's a setup that '''reuses | Alternatively, '''if you want to get up-and-running quickly in the mean-time''' there's a setup that '''reuses lots of the pi-custom software''' (non-desired approach but works) | ||
1. ROM -> 2. EEPROM -> Pi-custom Firmware/Kernel -> NixOS | 1. ROM -> 2. EEPROM -> Pi-custom Firmware/Kernel -> NixOS | ||
and creates a hard disk image that you can flash onto an SD card or NVMe SSD. The heavy lifting has been done and is maintained at https://github.com/tstat/raspberry-pi-nix (just follow the example and hints at https://github.com/tstat/raspberry-pi-nix/issues/13) and has the best out-of-the-box experience. You have to remote-build many Nix packages, probably the kernel as well, yourself (e.g. using the [https://wiki.nixos.org/wiki/Distributed_build Pi without NixOS as an intermediate remote builder]) and that can take several hours though. | and creates a hard disk image that you can flash onto an SD card or NVMe SSD. The heavy lifting has been done and is maintained at https://github.com/tstat/raspberry-pi-nix (just follow the example and hints at https://github.com/tstat/raspberry-pi-nix/issues/13) and has the best out-of-the-box experience. You have to remote-build many Nix packages, probably the kernel as well, yourself (e.g. using the [https://wiki.nixos.org/wiki/Distributed_build Pi without NixOS as an intermediate remote builder]) and that can take several hours though. | ||
| Line 101: | Line 101: | ||
[https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#differences-on-raspberry-pi-5 official documentation]. | [https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#differences-on-raspberry-pi-5 official documentation]. | ||
== | == Setting up a generic UEFI NixOS == | ||
1. ROM -> 2. EEPROM -> 3. UEFI boot loader (EDK2) -> 4. systemd-boot boot loader -> Pi 5-adapted NixOS Kernel -> NixOS | The task to get a generic NixOS setup requires a | ||
# UEFI boot loader for Pi 5: There exists a WIP [https://github.com/worproject/rpi5-uefi EDK2 for Pi 5 Github project] but with several limitations like no boot from NVMe SSDs. | |||
# systemd-boot boot loader: Works | |||
# generic Linux kernel that works for the Pi 5's ARM v8 processor and hardware: An almost generic Pi 5 compatible kernel exists [https://github.com/NixOS/nixos-hardware/blob/master/raspberry-pi/5/default.nix at the NixOS-hardware repository]; it's an adaption from a kernel for the Pi 4) | |||
1. ROM -> 2. EEPROM -> 3. UEFI boot loader (EDK2) -> 4. systemd-boot boot loader -> Pi 4/5-adapted NixOS Kernel -> NixOS | |||
1. '''Install EDK2''' (UEFI firmware implementation): | 1. '''Install EDK2''' (UEFI firmware implementation): | ||
| Line 117: | Line 123: | ||
2. '''Install systemd-boot, kernel and NixOS''': | 2. '''Install systemd-boot, kernel and NixOS''': | ||
The rest is a usual NixOS installation on a second partition with the caveat to '''select a Linux kernel that supports the Pi 5''' | The rest is a usual NixOS installation on a second partition with the caveat to '''select a Linux kernel that supports the Pi 5''' like the [https://github.com/NixOS/nixos-hardware/pull/927 Pi 5 compatible Linux kernel is available in nixos-hardware]. | ||
Follow [https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1936211154 this guide] to build | Follow [https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1936211154 this guide] to build | ||
| Line 127: | Line 131: | ||
and the rest of the NixOS system into your second partition. | and the rest of the NixOS system into your second partition. | ||
== Alternative board-specific installation notes == | |||
= | |||
First, install EDK2, following the [https://github.com/worproject/rpi5-uefi#getting-started instructions from the port README]. With EDK2 installed as the Platform Firmware, you can follow the [[NixOS_on_ARM/UEFI|standard instructions for UEFI on ARM]]. | First, install EDK2, following the [https://github.com/worproject/rpi5-uefi#getting-started instructions from the port README]. With EDK2 installed as the Platform Firmware, you can follow the [[NixOS_on_ARM/UEFI|standard instructions for UEFI on ARM]]. | ||
| Line 162: | Line 150: | ||
} | } | ||
</nowiki>}} | </nowiki>}} | ||
== Troubleshooting == | |||
=== GPU === | |||
For the GPU drivers to work, <code>dtoverlay=vc4-kms-v3d-pi5</code> must be added to <code>/boot/config.txt</code>, and the vendor kernel must currently be used. Only Wayland-based compositors are supported without additional configuration (see the nixos-hardware PR linked previously). Note that Xwayland applications may produce broken graphics on KDE; the root cause of this issue has not yet been evaluated. | |||
The rpi5-uefi download does not include overlays. You can get them by copying the <code>boot/overlays</code> folder from [https://github.com/raspberrypi/firmware the firmware repository] to <code>/boot</code> (so that <code>/boot/overlays/vc4-kms-v3d-pi5.dtbo</code> is available). | |||
=== Bluetooth === | === Bluetooth === | ||
| Line 169: | Line 164: | ||
} | } | ||
</nowiki>}} | </nowiki>}} | ||
== Using the Pi 5 as a remote builder to build native ARM packages for the Pi 5 == | |||
Building an NixOS system image that can be flashed to an SD card or NVMe SSD requires to '''build ARM binaries''', more specifically for the <code>"aarch64-linux"</code>platform. From a typical Intel/AMD computer we can either | |||
* '''emulated native compile using QEMU''' virtualization by enabling the [https://docs.kernel.org/admin-guide/binfmt-misc.html binfmt] kernel feature on NixOS configuration setting <code>boot.binfmt.emulatedSystems = [ "aarch64-linux" ];</code>). This can be fast if everything is downloaded pre-compiled from the cache.nixos.org cache and only few packages really need local compilation. In reality it can be extremely slow, e.g. compiling a Linux kernel alone can take days. | |||
* '''cross-compile''' to ARM using as to happen natively, but nothing will be cached from cache.nixos.org as this is not pre-build. So the compile itself is fast but there will be a lot more to compile locally. In practice it's quite fragile, because you may encounter packages that don't really support cross-compilation get stuck. | |||
* '''native compile on an remote builder''' like the Pi 5 itself running its custom Debian Linux at the beginning or later NixOS. This is reasonably fast as most packages are pre-build and cached on cache.nixos.org, and building a remaining Linux kernel only takes 2-3h on the Pi 5. | |||
Missing: | |||
# How to do cross-compilation. | |||
# How to create a convenient SD card image to not install anything manually. | |||
# How to install U-Boot instead of EDK2 and systemd-boot. | |||