Cross Compiling: Difference between revisions
m add section for basics. |
Improve examples |
||
(4 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
{{Expansion}} | {{Expansion}} | ||
[[Nixpkgs]] provides excellent support in configuring it for cross-platform compiling tasks since 18.09<sup>[citation needed]</sup>. | |||
In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the <code>buildPlatform</code>, while the latter is <code>hostPlatform</code>.<blockquote>If you were compiling a program from your system for a Raspberry PI, you would be the <code>buildPlatform</code> whereas the Raspberry PI would be the <code>hostPlatform</code>.</blockquote>Furthermore, in order to provide more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. A full reference to these can be found in the [https://nixos.org/manual/nixpkgs/unstable/#ssec-stdenv-dependencies-propagated Nixpkgs manual]. | |||
== Getting Started == | == Getting Started == | ||
Nixpkgs exposes two configuration attributes that map internally to the expected behaviors of the build/host platforms as described above. These attributes can be set when importing Nixpkgs as a Nix expression:<syntaxhighlight lang="nix"> | |||
let | let | ||
pkgs = import <nixpkgs> {}; | pkgs = import <nixpkgs> { | ||
in pkgs | localSystem = "x86_64-linux"; # buildPlatform | ||
</syntaxhighlight>The above will provide a derivation result for the hello derivation that can run on an <code>aarch64</code> system. | crossSystem = "aarch64-linux"; # hostPlatform | ||
}; | |||
in pkgs.hello | |||
</syntaxhighlight>The above will provide a derivation result for the hello derivation that can run on an <code>aarch64-linux</code> system. This can sometimes be tedious especially for common <code>hostPlatform</code> targets. Fortunately, Nixpkgs exposes a <code>pkgsCross</code> attribute that provides pre-configured cross compiling targets. The snippet above converted to using <code>pkgsCross</code> can be shorted to:<syntaxhighlight lang="nix"> | |||
let | let | ||
pkgs = import <nixpkgs> { | pkgs = import <nixpkgs> { | ||
localSystem = "x86_64-linux"; | |||
localSystem = "x86_64-linux"; | |||
}; | }; | ||
in pkgs. | in pkgs.pkgsCross.aarch64-multiplatform.hello | ||
</syntaxhighlight>You can perform the same operations using the CLI, | </syntaxhighlight>You can perform the same operations using the CLI, and Nix will correctly evaluate the <code>localSystem</code> based on your current system:<syntaxhighlight lang="bash"> | ||
nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy | nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy | ||
nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3 | nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3 | ||
Line 23: | Line 24: | ||
$ nix-instantiate --eval --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix-shell -p jq --command 'jq' # nix-legacy | $ nix-instantiate --eval --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix-shell -p jq --command 'jq' # nix-legacy | ||
$ nix eval --impure --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix run nixpkgs#jq # nix3 | $ nix eval --impure --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix run nixpkgs#jq # nix3 | ||
</syntaxhighlight>If you instead prefer to write your systems directly, through <code>localSystem</code> and <code>crossSystem</code>, you can refer to [https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix nixpkgs/lib/systems/examples.nix] for | </syntaxhighlight>If you instead prefer to write your systems directly, through <code>localSystem</code> and <code>crossSystem</code>, you can refer to [https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix nixpkgs/lib/systems/examples.nix] for examples of platforms exposed as attributes. These can be directly used in-place for the aforementioned arguments:<syntaxhighlight lang="nix"> | ||
let | let | ||
lib = import <nixpkgs/lib>; | lib = import <nixpkgs/lib>; | ||
pkgs = import <nixpkgs> { | pkgs = import <nixpkgs> { | ||
#localSystem = ...; | |||
crossSystem = lib.systems.examples.aarch64-multiplatform; | crossSystem = lib.systems.examples.aarch64-multiplatform; | ||
}; | }; | ||
Line 40: | Line 42: | ||
{ | { | ||
pkgs ? import <nixpkgs> { | pkgs ? import <nixpkgs> { | ||
localSystem = "x86_64-linux"; | |||
crossSystem = "aarch64-linux"; | crossSystem = "aarch64-linux"; | ||
}, | }, | ||
}: | }: | ||
pkgs.mkShell { | pkgs.callPackage ( | ||
{ | |||
} | mkShell, | ||
}: | |||
mkShell { | |||
# By default this provides gcc, ar, ld, and some other bare minimum tools | |||
} | |||
) { } | |||
</syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash"> | </syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash"> | ||
$ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ... | $ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ... | ||
Line 59: | Line 67: | ||
If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix"> | If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix"> | ||
{ | { | ||
pkgs ? import <nixpkgs> { | pkgs ? import <nixpkgs> { | ||
localSystem = "x86_64-linux"; | |||
crossSystem = "aarch64-linux"; | crossSystem = "aarch64-linux"; | ||
}, | }, | ||
Line 68: | Line 76: | ||
{ | { | ||
mkShell, | mkShell, | ||
hello, | |||
pkg-config, | pkg-config, | ||
libGL, | libGL, | ||
}: | }: | ||
mkShell { | mkShell { | ||
# | strictDeps = true; | ||
# host/target agnostic programs | |||
depsBuildBuild = [ | |||
hello | |||
]; | |||
# compilers & linkers & dependecy finding programs | |||
nativeBuildInputs = [ | nativeBuildInputs = [ | ||
pkg-config | pkg-config | ||
]; | ]; | ||
# libraries | |||
# | |||
buildInputs = [ | buildInputs = [ | ||
libGL | libGL | ||
]; | ]; | ||
} | } | ||
) {} | ) { } | ||
</syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>. | </syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>. | ||
For more information regarding the above, namely the usage of <code>nativeBuildInputs</code> and <code>buildInputs</code>, see [https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies stdenv dependencies] for a in-depth explanation. Alternatively, a simplified explanation can be found in a comment on the [https://github.com/NixOS/nixpkgs/pull/50881 | For more information regarding the above, namely the usage of <code>nativeBuildInputs</code> and <code>buildInputs</code>, see [https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies stdenv dependencies] for a in-depth explanation. Alternatively, a simplified explanation can be found in a comment on the [https://github.com/NixOS/nixpkgs/pull/50881 Nixpkgs repo]. | ||
== Tips and tricks == | == Tips and tricks == | ||
Line 93: | Line 105: | ||
By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary. | By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary. | ||
If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly.<syntaxhighlight lang="bash"> | If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly through the [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html binfmt_misc kernel feature].<syntaxhighlight lang="bash"> | ||
$ ./result | $ ./result | ||
Hello World! | Hello World! | ||
Line 106: | Line 118: | ||
=== Leveraging the binary cache === | === Leveraging the binary cache === | ||
You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. | You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. Fortunately, there are a small set of systems that are actively built and cached officially. At the time of writing, this only includes <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. If your platform targets include these, you may be able to leverage a slight hack to avoid large-scale builds.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time, and the storage requirements will be higher due to duplicate(but different system) packages.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix"> | ||
let | let | ||
# this will use aarch64 binaries from binary cache, so no need to build those | # this will use aarch64 binaries from binary cache, so no need to build those | ||
pkgsArm = import <nixpkgs> { | pkgsArm = import <nixpkgs> { | ||
localSystem = "aarch64-linux"; | |||
}; | }; | ||
Line 121: | Line 131: | ||
# we want to hack on SDL, don't want to hack on those. Some even don't cross-compile | # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile | ||
inherit (pkgsArm) | inherit (pkgsArm) | ||
xorg libpulseaudio libGL guile systemd libxkbcommon | xorg | ||
libpulseaudio | |||
libGL | |||
guile | |||
systemd | |||
libxkbcommon | |||
; | ; | ||
}) | }) | ||
]; | ]; | ||
localSystem = "x86_64-linux"; | |||
crossSystem = "aarch64-linux"; | crossSystem = "aarch64-linux"; | ||
}; | }; | ||
in pkgsCross. | in | ||
pkgsCross.callPackage ( | |||
{ | |||
SDL2, | |||
wayland, | |||
wayland-protocols, | |||
wayland-scanner, | |||
wayland wayland-protocols | }: | ||
SDL2.override { | |||
} | inherit | ||
wayland | |||
wayland-protocols | |||
wayland-scanner | |||
; | |||
} | |||
) { } | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 150: | Line 172: | ||
* [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org]) | * [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org]) | ||
[[Category:nix]] | [[Category:nix]] | ||
[[Category:Development]] | [[Category:Development]] |
Latest revision as of 21:14, 29 October 2024
Nixpkgs provides excellent support in configuring it for cross-platform compiling tasks since 18.09[citation needed].
In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the buildPlatform
, while the latter is hostPlatform
.
If you were compiling a program from your system for a Raspberry PI, you would be the
buildPlatform
whereas the Raspberry PI would be thehostPlatform
.
Furthermore, in order to provide more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. A full reference to these can be found in the Nixpkgs manual.
Getting Started
Nixpkgs exposes two configuration attributes that map internally to the expected behaviors of the build/host platforms as described above. These attributes can be set when importing Nixpkgs as a Nix expression:
let
pkgs = import <nixpkgs> {
localSystem = "x86_64-linux"; # buildPlatform
crossSystem = "aarch64-linux"; # hostPlatform
};
in pkgs.hello
The above will provide a derivation result for the hello derivation that can run on an aarch64-linux
system. This can sometimes be tedious especially for common hostPlatform
targets. Fortunately, Nixpkgs exposes a pkgsCross
attribute that provides pre-configured cross compiling targets. The snippet above converted to using pkgsCross
can be shorted to:
let
pkgs = import <nixpkgs> {
localSystem = "x86_64-linux";
};
in pkgs.pkgsCross.aarch64-multiplatform.hello
You can perform the same operations using the CLI, and Nix will correctly evaluate the localSystem
based on your current system:
nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy
nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3
All of the above snippets will resolve to the exact same derivation result, which will provide a binary for GNU Hello that can execute only on an aarch64
system. There are many other systems pkgsCross
has defined, you can see an exhaustive list of all of them on your system:
$ nix-instantiate --eval --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix-shell -p jq --command 'jq' # nix-legacy
$ nix eval --impure --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix run nixpkgs#jq # nix3
If you instead prefer to write your systems directly, through localSystem
and crossSystem
, you can refer to nixpkgs/lib/systems/examples.nix for examples of platforms exposed as attributes. These can be directly used in-place for the aforementioned arguments:
let
lib = import <nixpkgs/lib>;
pkgs = import <nixpkgs> {
#localSystem = ...;
crossSystem = lib.systems.examples.aarch64-multiplatform;
};
in pkgs.hello
Development
Basics
Using the same ideas as above, we can create development environments which provide us with a compilation suite that can perform cross-compilation for us. A very simple development shell (colloquially called a "devshell") can be written as:
# shell.nix
{
pkgs ? import <nixpkgs> {
localSystem = "x86_64-linux";
crossSystem = "aarch64-linux";
},
}:
pkgs.callPackage (
{
mkShell,
}:
mkShell {
# By default this provides gcc, ar, ld, and some other bare minimum tools
}
) { }
Entering this development shell via nix-shell shell.nix
will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing gcc
will not work. However, the provided mkShell
will introduce environment variables for your devshell, such as $CC
, $AR
, $LD
, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:
$ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ...
Given these environment variables, you can run compile your software using the exact same commands with fairly minimal changes (changing hardcoded gcc
values into $CC, for example):
$ $CC -o main src/main.c
$ file main
main: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/qa51m8r8rjnigk5hf7sxv0hw7qr7l4bc-glibc-aarch64-unknown-linux-gnu-2.39-52/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped
The above snippet will have minor differences depending on when you run it, but the main thing to notice is ARM aarch64
, which tells us our software was able to successfully cross compile.
Declaring dependencies
If you try to declare build-time dependencies within the devshell (such as pkgs.cmake
), you will quickly realize that these derivations are actually being built for the crossSystem
, making them unusable on your system architecture (see #49526). There are ways around this, but in general once you've gotten to this point you should prefer writing a derivation, which will make it not only easier to write both derivations, but will allow you to follow the recommended practices for using Nix.
If you would prefer to continue building within the devshell, you can use callPackage, which will magically resolve the dependencies for the correct architecture, provided you place them in the correct attributes:
{
pkgs ? import <nixpkgs> {
localSystem = "x86_64-linux";
crossSystem = "aarch64-linux";
},
}:
pkgs.callPackage (
{
mkShell,
hello,
pkg-config,
libGL,
}:
mkShell {
strictDeps = true;
# host/target agnostic programs
depsBuildBuild = [
hello
];
# compilers & linkers & dependecy finding programs
nativeBuildInputs = [
pkg-config
];
# libraries
buildInputs = [
libGL
];
}
) { }
The above snippet will drop you into a devshell that provides pkg-config
as a native binary (accessible through $PKG_CONFIG
), while also allowing linking to a valid libGL
for the crossSystem
.
For more information regarding the above, namely the usage of nativeBuildInputs
and buildInputs
, see stdenv dependencies for a in-depth explanation. Alternatively, a simplified explanation can be found in a comment on the Nixpkgs repo.
Tips and tricks
Executing cross compiled binaries
By using QEMU, we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary.
If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting boot.binfmt.emulatedSystems in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke qemu
indirectly through the binfmt_misc kernel feature.
$ ./result
Hello World!
$ ./result-aarch64-linux
Hello World!
Otherwise, you can use the pkgs.qemu-user
to download qemu user space programs (or use any installed by your distro) to run your package easily.
$ ./result
Hello World!
$ qemu-aarch64 ./result-aarch64-linux
Hello World!
Leveraging the binary cache
You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official binary cache. Fortunately, there are a small set of systems that are actively built and cached officially. At the time of writing, this only includes aarch64-linux
, aarch64-darwin
, i686-linux
, x86_64-linux
, and x86_64-darwin
. If your platform targets include these, you may be able to leverage a slight hack to avoid large-scale builds.
Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time, and the storage requirements will be higher due to duplicate(but different system) packages.
An example of this using pkgs.SDL2
:
let
# this will use aarch64 binaries from binary cache, so no need to build those
pkgsArm = import <nixpkgs> {
localSystem = "aarch64-linux";
};
# these will be your cross packages
pkgsCross = import <nixpkgs> {
overlays = [
(self: super: {
# we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
inherit (pkgsArm)
xorg
libpulseaudio
libGL
guile
systemd
libxkbcommon
;
})
];
localSystem = "x86_64-linux";
crossSystem = "aarch64-linux";
};
in
pkgsCross.callPackage (
{
SDL2,
wayland,
wayland-protocols,
wayland-scanner,
}:
SDL2.override {
inherit
wayland
wayland-protocols
wayland-scanner
;
}
) { }
See also
- NixOS on ARM
- Packaging/32bit Applications
- Cheatsheet#Cross-compile packages
- Nixpkgs manual on cross compiling