NixOS on ARM: Difference between revisions

Revision as of 19:30, 26 September 2021

ARM support for NixOS is a work-in-progress, but is progressing quickly.

The support varies depending on the architecture and the specific boards. The way the ARM integration is built into NixOS is by making generic builds the first-class citizens; as soon as there is upstream support for the board in the kernel and the bootloader, NixOS should work once updated to these versions. It is still possible, when needed, to build and use a customised bootloader and kernel for specific boards^{[reference needed]}. At this moment in time (late 2021) only AArch64 has full support upstream. Though, neither armv6l or armv7l are being ignored, fixes are worked on and approved as needed; what's missing is support and upstream builds being maintained in binary form. At the time of writing, no publicly available caches for armv6l or armv7l are available.

Supported devices

Upstream (NixOS) supported devices

NixOS has support for these boards using AArch64 architecture on the nixpkgs-unstable and stable channel.

Support for those board assumes as much is supported as Mainline Linux supports.

Manufacturer	Board	SoC	ISA	CPU	RAM	Storage
Raspberry Pi Foundation	Raspberry Pi 3	Broadcom BCM2837	AArch64 / ARMv7	4× Cortex-A53 @ 1.2 GHz	1 GB	SD/microSD
Raspberry Pi Foundation	Raspberry Pi 4	Broadcom BCM2711	AArch64 / ARMv7	4× Cortex-A53 @ 1.5 GHz	1-8 GB	microSD

Community supported devices

Manufacturer	Board	SoC	ISA	CPU	RAM	Storage
ASUS	Tinker Board	Rockchip RK3288	ARMv7	4× Cortex-A17	2 GB	microSD
Banana Pi	Banana Pi	Allwinner A20	ARMv7	2× Cortex-A7	1 GB	SD, SATA
Banana Pi M64	Banana Pi M64	Allwinner A64	ARMv8	4× Cortex-A53	2 GB	microSD, 8GB eMMc
BeagleBoard.org	BeagleBone Black	TI AM335x	ARMv7	1× Cortex-A8 @ 1 GHz	512 MB	4 GB eMMC, microSD
Firefly	AIO-3399C	Rockchip RK3399	AArch64	2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz	2/4 GB	8/16 GB eMMC, microSD
FriendlyElec	NanoPC-T4	Rockchip RK3399	AArch64	2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz	4 GB	16 GB eMMC, microSD, NVMe
Hardkernel	ODROID-HC1 & ODROID-HC2	Samsung Exynos 5422	ARMv7	4 × Cortex-A15 @ 2GHz, 4 × Cortex-A7 @ 1.4GHz	2 GB	microSD
Hardkernel	ODROID-C2	Amlogic S905	AArch64	4 × Cortex-A53 @ 1.5GHz	2 GB	eMMC, microSD
Hardkernel	ODROID-HC4	Amlogic S905X3	AArch64	4 × Cortex-A55 @ 1.8GHz	4 GB	microSD
Libre Computer	ROC-RK3399-PC	Rockchip RK3399	AArch64	2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz	4 GB	eMMC, microSD, NVMe
Linksprite	pcDuino3 Nano	Allwinner A20	ARMv7	2× Cortex-A7 @ 1 GHz	1 GB	4 GB NAND, microSD, SATA
NVIDIA	Jetson TK1	Tegra K1/T124	ARMv7	4× Cortex-A15 @ 2.3 GHz	2 GB	16 GB eMMC, SD, SATA
Orange Pi	Orange Pi One	Allwinner H3	ARMv7	4× Cortex-A7 @ 1.2 GHz	512 MB	microSD
Orange Pi	Orange Pi PC	Allwinner H3	ARMv7	4× Cortex-A7 @ 1.6 GHz	1 GB	SD/microSD
Orange Pi	Orange Pi Zero Plus2 (H5)	Allwinner H5	AArch64	4× Cortex-A53 @ 1.2 GHz	1 GB	SD/microSD + 8GB eMMC
PINE64	PINE A64-LTS	Allwinner R18	AArch64	4× Cortex-A53 @ ? GHz	2 GB	microSD & eMMC
PINE64	Pinebook	Allwinner A64	AArch64	4× Cortex-A53 @ ? GHz	2 GB	microSD & eMMC
PINE64	Pinebook Pro	Rockchip RK3399	AArch64	2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz	4 GB	microSD & eMMC
PINE64	ROCK64	Rockchip RK3328	AArch64	4x Cortex-A53 @ 1.5 GHz	1/2/4 GB	microSD/eMMC
PINE64	ROCKPro64	Rockchip RK3399	AArch64	2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz	2/4 GB	microSD/eMMC
Raspberry Pi Foundation	Raspberry Pi	Broadcom BCM2835	ARMv6	1× ARM1176 @ 700 MHz	256 MB / 512 MB	SD/microSD
Raspberry Pi Foundation	Raspberry Pi 2	Broadcom BCM2836	ARMv7	4× Cortex-A7 @ 900 MHz	1 GB	SD/microSD
Raspberry Pi Foundation	Raspberry Pi 3	Broadcom BCM2837	AArch64 / ARMv7	4× Cortex-A53 @ 1.2 GHz	1 GB	SD/microSD
Raspberry Pi Foundation	Raspberry Pi 4	Broadcom BCM2711	AArch64 / ARMv7	4× Cortex-A53 @ 1.5 GHz	1-8 GB	microSD
Toshiba	AC100 (mini laptop)	Tegra 2 250 (T20)	ARMv7	2× Cortex-A9 @ 1 GHz	512 MB	8–32 GB eMMC, SD
Wandboard	Wandboard Solo/Dual/Quad	Freescale i.MX6	ARMv7	1×/2×/4× Cortex-A9 @ 1000 MHz	512 MB / 1 GB / 2 GB	microSD, SATA

Special Devices

It is possible to emulate an ARM platform with QEMU.

Manufacturer	Board	SoC	ISA	CPU	RAM	Storage
QEMU	QEMU	—	ARMv7	up to 8	up to 2 GB	Anything QEMU supports

Installation

Getting the installer

For AArch64 it is possible to download images from Hydra.

On the page click on the latest successful build to get a download link under build products. If the image has the extension .zst, it will need to be decompressed before writing to installation device. Use nix-shell -p zstd --run "unzstd <img-name>.img.zst" to decompress the image.

Installation steps

The .img files can be directly written to a microSD/SD card (minimal recommended size: 4 GB) using dd, once uncompressed from the ZSTD container. The SD card needs to be unmounted first.

The base images are configured to boot up with a serial TTY ( RX/TX UART ) @ 115200 Baud. That way you not necessarily have to have a HDMI Display and keyboard.

Note: For some platforms, manually editing and adding kernel command-line arguments to /boot/extlinux/extlinux.conf may be needed for serial to work, and is "as" supported as would be editing the command-line manually during boot.

Continue with #NixOS installation & configuration.

Binary cache

AArch64

The official NixOS Hydra instance builds a full set of binaries (available on https://cache.nixos.org) for the AArch64 architecture on the nixpkgs-unstable and stable channels.

armv6l and armv7l

Some users are providing best effort caches for 32 bit ARM.

@thefloweringash's

A binary cache for armv7l-linux, containing a subset of several channels, is hosted at https://app.cachix.org/cache/thefloweringash-armv7 .

Build your own image

You can customize image by using the following snippet.

# save as sd-image.nix somewhere
{ ... }: {
  imports = [
    <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix>
  ];
  # put your own configuration here, for example ssh keys:
  users.extraUsers.root.openssh.authorizedKeys.keys = [
     "ssh-ed25519 AAAAC3NzaC1lZDI1.... username@tld"
  ];
}

Then build with:

$ nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config=./sd-image.nix

Note that this requires a machine with aarch64. You can however also build it from your laptop using an aarch64 remote builder as described in Distributed build or ask for access on the community aarch64 builder.

Compiling through QEMU

It is also possible to compile for aarch64 on your non-aarch64 local machine, or a remote builder, by registering QEMU as a binfmt wrapper for the aarch64 architecture. This wrapper uses emulation and will therefore be slower than comparable native machines.

To enable the binfmt wrapper on NixOS, add the following to configuration.nix

{
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
}

If you are building on non-NixOS machine with QEMU binfmt wrapper configured, you will want to configure nix daemon to let it know that it can build for aarch64. Add the following line to /etc/nix/nix.conf: extra-platforms = aarch64-linux arm-linux

If you want to build just one specific package, use this:

nix-build '<nixpkgs/nixos>' -A pkgs.theRequiredPackage --argstr system aarch64-linux -I nixos-config=/path/to/target/machine/nixos/config/copy

(the last option should not be required on NixOS machines)

Installer image with custom U-Boot

The Mic92/nixos-aarch64-images repository provides a mechanism to modify the official NixOS installer to embed the board-specific U-Boot firmware required for different boards. This method does not require QEMU or native ARM builds since the existing Hydra-built U-Boot binaries are used.

Board-specific installation notes

Depending on the board, some additional preparation steps might be needed to make the SD card bootable on your device. All of the board-specific installation notes are now found on their respective pages.

Enable UART

If you try to use UART to log on NixOS, you might hang on the line "Starting kernel ...". To enable UART, you will need to add at the end of the line that contains loglevel4 in the file /extlinux/extlinux.conf the text:

/extlinux/extlinux.conf

    console=ttyAMA0,115200n8

/extlinux/extlinux.conf

    console=ttyS0,115200n8

The actual device (ttyAMA0, ttyS0, ttyS1) will depend on the hardware.

NixOS installation & configuration

This article or section is out of date.

Reason: The kernel version recommendations of this section are severely outdated. This section should be rewritten to be generic and refer people to the board-specific page. Only the board specific page should make recommendations about the kernel. (Discuss in Talk:NixOS on ARM)
Once the page has been updated this notice can be removed.

The installation image is actually a MBR partition table plus two partitions; a FAT16 /boot and a ext4 root filesystem. The image is designed such that it's possible to directly reuse the SD image's partition layout and "install" NixOS on the very same SD card by simply replacing the default configuration.nix and running nixos-rebuild. Using this installation method is strongly recommended, though if you know exactly what you're doing and how U-Boot on your board works, you can use nixos-install as usual. To help with the SD card installation method, the boot scripts on the image automatically resize the rootfs partition to fit the SD card on the first boot.

Use this as a template:

/etc/nixos/configuration.nix

{ config, pkgs, lib, ... }:
{
  # NixOS wants to enable GRUB by default
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;
 
  # !!! If your board is a Raspberry Pi 1, select this:
  boot.kernelPackages = pkgs.linuxPackages_rpi;
  # !!! Otherwise, pick this:
  boot.kernelPackages = pkgs.linuxPackages_latest;
  
  # !!! This is only for ARMv6 / ARMv7. Don't enable this on AArch64, cache.nixos.org works there.
  nix.binaryCaches = lib.mkForce [ "https://app.cachix.org/cache/thefloweringash-armv7" ];
  nix.binaryCachePublicKeys = [ "thefloweringash-armv7.cachix.org-1:v+5yzBD2odFKeXbmC+OPWVqx4WVoIVO6UXgnSAWFtso=" ];
    
  # File systems configuration for using the installer's partition layout
  fileSystems = {
    # Prior to 19.09, the boot partition was hosted on the smaller first partition
    # Starting with 19.09, the /boot folder is on the main bigger partition.
    # The following is to be used only with older images.
    /*
    "/boot" = {
      device = "/dev/disk/by-label/NIXOS_BOOT";
      fsType = "vfat";
    };
    */
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
    
  # !!! Adding a swap file is optional, but strongly recommended!
  # swapDevices = [ { device = "/swapfile"; size = 1024; } ];
}

Note: the default configuration.nix will contain something like imports = [ <nixos/modules/installer/sd-card/sd-image-armv7l-multiplatform.nix> ]; do not include that in your final installation or you will experience interesting problems. It is only for building the installation image!

First rebuild on ARMv6 and ARMv7

To make the unsupported ARM experience slightly less painful, the config template adds nixos-arm.dezgeg.me as a binary cache, which contains a small subset of packages on the unstable channel (though a caution for US users: the server hosting them is physically located in Finland). Note that the binary cache isn't enabled on the prebuilt images, so enable it via the command line when building for the first time:

nixos-rebuild switch --fast --option binary-caches http://nixos-arm.dezgeg.me/channel --option binary-cache-public-keys nixos-arm.dezgeg.me-1:xBaUKS3n17BZPKeyxL4JfbTqECsT+ysbDJz29kLFRW0=%

Details about the boot process

On NixOS, all ARM boards are expected to use U-Boot as the firmware and bootloader. NixOS uses U-Boot's Generic Distro Configuration Concept as the mechanism to communicate boot information (such as path to kernel zImage, initrd, DTB, command line arguments). For a quick TL;DR about the generic distro configuration support: U-Boot is scripted to scan all attached storage devices & partitions and look for a file named /extlinux/extlinux.conf or /boot/extlinux/extlinux.conf (which will be generated by NixOS, just like /boot/grub/grub.cfg is generated on PCs). The partition needs to have its "bootable" flag set.

U-Boot also provides an interactive shell and the generation selection menu (just like GRUB). However, support for input or display devices varies greatly, depending on the board. Details for what the boards support in relationship to the boot process are detailed in their respective pages.

Porting NixOS to new boards

The easiest way

Assuming upstream U-Boot supports the board through a defconfig, it is possible possible to build U-Boot using the cross-compiling architecture from an x86_64 host. Here's a sample use.

# Assuming you're in a recent nixpkgs checkout
$ nix-shell \
    -I "nixpkgs=$PWD" \
    -p 'let plat = pkgsCross.aarch64-multiplatform; in plat.buildUBoot{defconfig = "orangepi_zero_plus2_defconfig"; extraMeta.platforms = ["aarch64-linux"]; BL31 = "${plat.armTrustedFirmwareAllwinner}/bl31.bin"; filesToInstall = ["u-boot-sunxi-with-spl.bin"];}'

For armv7 and armv6 pkgsCross.arm-embedded should work, this is available in the unstable channel (19.03 and following) by setting -I "nixpkgs=/path/to/new-nixpkgs-checkout.

This should build whatever is needed for, and then build U-Boot for the desired defconfig, then open a shell with the build in $buildInputs. Do note that this particular invocation may need more changes than only the defconfig if built for other than allwinner boards.

Here's an example command, for allwinner boards, on how to write to an SD card.

$ sudo dd if=$buildInputs/u-boot-sunxi-with-spl.bin of=/dev/sdX bs=1024 seek=8

The easy way

(if you're lucky)

If your board is an ARMv7 board supported by multi_v7_defconfig and you have access to U-Boot on the board, getting sd-image-armv7l-linux.img to boot is the easiest option:

If you're lucky and your U-Boot build comes with the extlinux.conf support built in, the image boots out-of-the-box. This is the case for all (upstream) Allwinner and Tegra U-Boots, for instance.
Otherwise, you can get the boot information (path to kernel zImage, initrd, DTB, command line arguments) by extracting extlinux.conf from the boot partition of the image, and then attempt to boot it via the U-Boot shell, or some other mechanism that your board's distro uses (e.g. uEnv.txt).

Building U-Boot from your NixOS PC

Assuming

Your board is supported upstream by U-Boot or there is a recent enough fork with extlinux.conf support.
You do not have nix setup on an ARM device
Your nix isn't setup for cross-compilation

It is still possible to build U-Boot using tools provided by NixOS.

In the following terminal session, replace orangepi_pc_defconfig with the appropriate board from the configs folder of U-Boot.

$ nix-shell -E 'with import <nixpkgs> {}; stdenv.mkDerivation { name = "arm-shell"; buildInputs = [git gnumake gcc gcc-arm-embedded dtc]; }'
$ git clone git://git.denx.de/u-boot.git
$ cd u-boot
# We're checking out a version from before the use of `binman`.
# The dtc package is 1.4.2, which does not include `pylibftd`.
# Furthermore, I do not know how to package the library so it would be
# available in the python interpreter, making binman happy.
$ git checkout v2017.03
$ make -j4 CROSS_COMPILE=arm-none-eabi- orangepi_pc_defconfig
$ make -j4 CROSS_COMPILE=arm-none-eabi-

The name of the final file will change depending on the board. For this specific build, and most Allwinner builds, the file will be named u-boot-sunxi-with-spl.bin.

You can flash this file to boot device with

 dd if=u-boot-sunxi-with-spl.bin of=/dev/sdX bs=1024 seek=8

Note: This mailing list contains a patch which may help some builds: https://lists.denx.de/pipermail/u-boot/2016-December/275664.html

The hard way

Alternatively/if all else fails, you can do it the hard way and bootstrap NixOS from an existing ARM Linux installation.

Contributing new boards to nixpkgs

Add a new derivation for your board's U-Boot configuration, see for example ubootJetsonTK1 in all-packages.nix.
If your board's U-Boot configuration doesn't use the extlinux.conf format by default, create a patch to enable it. Some C hacking skills & U-Boot knowledge might be required. For some pointers, see this patch to enable it on the Versatile Express.
Make a pull request, also containing the board-specific instructions. Ping @dezgeg for review and for building & hosting the U-Boots at http://nixos-arm.dezgeg.me/installer.

Support

All ARM platforms are experimental. Only AArch64 platforms are currently being worked on for eventual support from NixOS.

There is a dedicated channel for the upstream effort on Matrix, #nixos-on-arm:nixos.org.

Resources

Subpages

The following is a list of all sub-pages of the NixOS on ARM topic.

@@ Line 329: / Line 329: @@
 A binary cache for <code>armv7l-linux</code>, containing a [https://github.com/thefloweringash/thefloweringash-armv7/blob/master/release.nix subset] of [https://hydra.cons.org.nz/project/thefloweringash-armv7 several] channels, is hosted at https://app.cachix.org/cache/thefloweringash-armv7 .
-'''@tomberek's'''
-https://arm.cachix.org/ - <code>arm.cachix.org-1:5BZ2kjoL1q6nWhlnrbAl+G7ThY7+HaBRD9PZzqZkbnM=</code>
-Last updated:
-*2020-08-10: /nix/store/lxi1hq2p0fx5wr123q9yyfmh3ngqnrq5-nix-2.4pre20200721_ff314f1 @ nixpkgs: 96069f7d890b90cbf4e8b4b53e15b036210ac146)
-*2020-08-11: NixOS image build uploaded
 == Build your own image ==