Cross Compiling: Difference between revisions

(4 intermediate revisions by 3 users not shown)

Line 2:

[[Nixpkgs]] provides excellent support in configuring it for cross-platform compiling tasks since 18.09<sup>[citation needed]</sup>.

In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the <code>buildPlatform</code>, while the latter is <code>hostPlatform</code>.<blockquote>If you were compiling a program from your system for a Raspberry PI, you would be the <code>buildPlatform</code> whereas the Raspberry PI would be the <code>hostPlatform</code>.</blockquote>Furthermore, in order to provide a more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. ~~An exhaustive~~ reference to these can be found in the [https://nixos.org/manual/nixpkgs/unstable/#ssec-stdenv-dependencies-propagated Nixpkgs manual].

In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the <code>buildPlatform</code>, while the latter is <code>hostPlatform</code>.<blockquote>If you were compiling a program from your system for a Raspberry PI, you would be the <code>buildPlatform</code> whereas the Raspberry PI would be the <code>hostPlatform</code>.</blockquote>Furthermore, in order to provide more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. A full reference to these can be found in the [https://nixos.org/manual/nixpkgs/unstable/#ssec-stdenv-dependencies-propagated Nixpkgs manual].

== Getting Started ==

Line 20:

</syntaxhighlight>You can perform the same operations using the CLI, and Nix will correctly evaluate the <code>localSystem</code> based on your current system:<syntaxhighlight lang="bash">

nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy

nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs> {}).lib.systems.examples.aarch64-multiplatform' -A hello # alternative way

nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3

</syntaxhighlight>All of the above snippets will resolve to the exact same derivation result, which will provide a binary for GNU Hello that can execute only on an <code>aarch64</code> system. There are many other systems <code>pkgsCross</code> has defined, you can see an exhaustive list of all of them on your system:<syntaxhighlight lang="bash">

Line 28:

Line 29:

lib = import <nixpkgs/lib>;

pkgs = import <nixpkgs> {

#localSystem = ...;

crossSystem = lib.systems.examples.aarch64-multiplatform;

};

Line 43:

Line 45:

localSystem = "x86_64-linux";

crossSystem = "aarch64-linux";

},

}:

pkgs.mkShell {

pkgs.callPackage (

# By default this provides gcc, ar, ld, and some other bare minimum tools

{

}

mkShell,

}:

mkShell {

# By default this provides gcc, ar, ld, and some other bare minimum tools

}

) { }

</syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash">

$ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ...

Line 60:

Line 68:

If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix">

~~# shell.nix~~

{

pkgs ? import <nixpkgs> {

localSystem = "x86_64-linux";

crossSystem = "aarch64-linux";

},

Line 69:

Line 77:

{

mkShell,

hello,

pkg-config,

libGL,

}:

mkShell {

# ~~Derivations that must run on the localSystem, referred to as the buildPlatform.~~

strictDeps = true;

# host/target agnostic programs

depsBuildBuild = [

hello

];

# compilers & linkers & dependecy finding programs

nativeBuildInputs = [

pkg-config

];

# libraries

# ~~Derivations that must link with the crossSystem, referred to as the targetPlatform.~~

buildInputs = [

libGL

];

}

) {}

) { }

</syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>.

Line 94:

Line 106:

By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary.

If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly.<syntaxhighlight lang="bash">

If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly through the [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html binfmt_misc kernel feature].<syntaxhighlight lang="bash">

$ ./result

Hello World!

Line 107:

Line 119:

=== Leveraging the binary cache ===

You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. ~~However~~, ~~for some~~ systems~~, natively compiled binaries~~ are cached. At the time of writing, this only includes <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. ~~As a result~~, ~~this section is only applicable~~ to a ~~very small number of cross~~-~~compilation situations.~~

You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. Fortunately, there are a small set of systems that are actively built and cached officially. At the time of writing, this only includes <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. If your platform targets include these, you may be able to leverage a slight hack to avoid large-scale builds.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time, and the storage requirements will be higher due to duplicate(but different system) packages.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">

~~You can leverage the binary cache to correctly substitute some applicable derivations without causing a local build~~.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">

let

# this will use aarch64 binaries from binary cache, so no need to build those

pkgsArm = import <nixpkgs> {

~~system~~ = "aarch64-linux";

localSystem = "aarch64-linux";

};

Line 122:

Line 132:

# we want to hack on SDL, don't want to hack on those. Some even don't cross-compile

inherit (pkgsArm)

xorg libpulseaudio libGL guile systemd libxkbcommon

xorg

libpulseaudio

libGL

guile

systemd

libxkbcommon

;

})

];

localSystem = "x86_64-linux";

crossSystem = "aarch64-linux";

};

in pkgsCross.SDL2.override {

in

~~# These should be neither pkgsCross~~, ~~nor pkgsArm~~

pkgsCross.callPackage (

~~# because those trigger~~

{

# > ~~cannot execute binary file~~: ~~Exec format error~~

SDL2,

# ~~In this case it was enough to just use buildPackages variants,~~

wayland,

# ~~but~~ in ~~general, there may be problems~~

wayland-protocols,

~~inherit (pkgsCross~~.~~buildPackages)~~

wayland-scanner,

~~wayland wayland~~-~~protocols~~

}:

;

SDL2.override {

inherit

wayland

wayland-protocols

wayland-scanner

;

}

) { }

</syntaxhighlight>

=== Using cross-compiled packages in the host NixOS config ===

First, configure your builder as a trusted [[Binary Cache|binary cache]].

Then copy the cross-compiled package to the host using <code>nix copy</code>.

Finally modify your <code>configuration.nix</code> as follows:

let pkgsCross = import <nixpkgs> {

localSystem = "x86_64-linux"; # <-- put your builder's platform here and below

hostSystem = "x86_64-linux";

crossSystem = "aarch64-linux"; # <-- put your host's platform here

};

in

{

environment.systemPackages = [

pkgsCross.hello # <-- put your cross-compiled package here

];

}

</syntaxhighlight>

Line 151:

Line 194:

* [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org])

* [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html Kernel documentation on binfmt_misc]

[[Category:nix]]

[[Category:Development]]

@@ Line 2: / Line 2: @@
 [[Nixpkgs]] provides excellent support in configuring it for cross-platform compiling tasks since 18.09<sup>[citation needed]</sup>.
-In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the <code>buildPlatform</code>, while the latter is <code>hostPlatform</code>.<blockquote>If you were compiling a program from your system for a Raspberry PI, you would be the <code>buildPlatform</code> whereas the Raspberry PI would be the <code>hostPlatform</code>.</blockquote>Furthermore, in order to provide a more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. An exhaustive reference to these can be found in the [https://nixos.org/manual/nixpkgs/unstable/#ssec-stdenv-dependencies-propagated Nixpkgs manual].
+In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the <code>buildPlatform</code>, while the latter is <code>hostPlatform</code>.<blockquote>If you were compiling a program from your system for a Raspberry PI, you would be the <code>buildPlatform</code> whereas the Raspberry PI would be the <code>hostPlatform</code>.</blockquote>Furthermore, in order to provide more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. A full reference to these can be found in the [https://nixos.org/manual/nixpkgs/unstable/#ssec-stdenv-dependencies-propagated Nixpkgs manual].
 == Getting Started ==
@@ Line 20: / Line 20: @@
 </syntaxhighlight>You can perform the same operations using the CLI, and Nix will correctly evaluate the <code>localSystem</code> based on your current system:<syntaxhighlight lang="bash">
 nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy
+nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs> {}).lib.systems.examples.aarch64-multiplatform' -A hello # alternative way
 nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3
 </syntaxhighlight>All of the above snippets will resolve to the exact same derivation result, which will provide a binary for GNU Hello that can execute only on an <code>aarch64</code> system. There are many other systems <code>pkgsCross</code> has defined, you can see an exhaustive list of all of them on your system:<syntaxhighlight lang="bash">
@@ Line 28: / Line 29: @@
    lib = import <nixpkgs/lib>;
    pkgs = import <nixpkgs> {
+    #localSystem = ...;
      crossSystem = lib.systems.examples.aarch64-multiplatform;
    };
@@ Line 43: / Line 45: @@
      localSystem = "x86_64-linux";
      crossSystem = "aarch64-linux";
    },
 }:
-pkgs.mkShell {
+pkgs.callPackage (
-  # By default this provides gcc, ar, ld, and some other bare minimum tools
+  {
-}
+    mkShell,
+  }:
+  mkShell {
+    # By default this provides gcc, ar, ld, and some other bare minimum tools
+  }
+) { }
 </syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash">
 $ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ...
@@ Line 60: / Line 68: @@
 If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix">
-# shell.nix
 {
    pkgs ? import <nixpkgs> {
+    localSystem = "x86_64-linux";
      crossSystem = "aarch64-linux";
    },
@@ Line 69: / Line 77: @@
    {
      mkShell,
+    hello,
      pkg-config,
      libGL,
    }:
    mkShell {
-     # Derivations that must run on the localSystem, referred to as the buildPlatform.
+    strictDeps = true;
+     # host/target agnostic programs
+    depsBuildBuild = [
+      hello
+    ];
+    # compilers & linkers & dependecy finding programs
      nativeBuildInputs = [
        pkg-config
      ];
+     # libraries
-     # Derivations that must link with the crossSystem, referred to as the targetPlatform.
      buildInputs = [
        libGL
      ];
    }
-) {}
+) { }
 </syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>.
@@ Line 94: / Line 106: @@
 By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary.
-If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly.<syntaxhighlight lang="bash">
+If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly through the [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html binfmt_misc kernel feature].<syntaxhighlight lang="bash">
 $ ./result
 Hello World!
@@ Line 107: / Line 119: @@
 === Leveraging the binary cache ===
-You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. However, for some systems, natively compiled binaries are cached. At the time of writing, this only includes <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. As a result, this section is only applicable to a very small number of cross-compilation situations.
+You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. Fortunately, there are a small set of systems that are actively built and cached officially. At the time of writing, this only includes <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. If your platform targets include these, you may be able to leverage a slight hack to avoid large-scale builds.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time, and the storage requirements will be higher due to duplicate(but different system) packages.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">
-You can leverage the binary cache to correctly substitute some applicable derivations without causing a local build.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">
 let
    # this will use aarch64 binaries from binary cache, so no need to build those
    pkgsArm = import <nixpkgs> {
-     system = "aarch64-linux";
+     localSystem = "aarch64-linux";
    };
@@ Line 122: / Line 132: @@
          # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
          inherit (pkgsArm)
-           xorg libpulseaudio libGL guile systemd libxkbcommon
+           xorg
+          libpulseaudio
+          libGL
+          guile
+          systemd
+          libxkbcommon
            ;
        })
      ];
+    localSystem = "x86_64-linux";
      crossSystem = "aarch64-linux";
    };
-in pkgsCross.SDL2.override {
+in
-   # These should be neither pkgsCross, nor pkgsArm
+pkgsCross.callPackage (
-  # because those trigger
+  {
-  # > cannot execute binary file: Exec format error
+    SDL2,
-   # In this case it was enough to just use buildPackages variants,
+    wayland,
-   # but in general, there may be problems
+    wayland-protocols,
-   inherit (pkgsCross.buildPackages)
+    wayland-scanner,
-     wayland wayland-protocols
+  }:
-    ;
+  SDL2.override {
+    inherit
+      wayland
+      wayland-protocols
+      wayland-scanner
+      ;
+   }
+) { }
+</syntaxhighlight>
+=== Using cross-compiled packages in the host NixOS config ===
+First, configure your builder as a trusted [[Binary Cache|binary cache]].
+Then copy the cross-compiled package to the host using <code>nix copy</code>.
+Finally modify your <code>configuration.nix</code> as follows:
+<syntaxhighlight lang="nix">
+let pkgsCross = import <nixpkgs> {
+   localSystem = "x86_64-linux"; # <-- put your builder's platform here and below
+  hostSystem = "x86_64-linux";
+   crossSystem = "aarch64-linux"; # <-- put your host's platform here
+};
+in
+{
+   environment.systemPackages = [
+     pkgsCross.hello # <-- put your cross-compiled package here
+  ];
 }
 </syntaxhighlight>
@@ Line 151: / Line 194: @@
 * [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org])
-* [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html Kernel documentation on binfmt_misc]
 [[Category:nix]]
 [[Category:Development]]