NixOS Wiki

Cross Compiling: Difference between revisions

← Older editNewer edit →
VisualWikitext
Klinger (talk | contribs)
trusted
903 edits
Frontear (talk | contribs)
27 edits
majorly overhaul the cross-compilation page with more details, better examples, and modernized nix code.
Line 1: Line 1:
{{Expansion}}
{{Expansion}}
For building arm software check out the Article [[NixOS on ARM]]<br/>
Cross compiling is well supported in [[Nixpkgs]] since 18.09<sup>[citation needed]</sup>.
If you are looking for building 32bit software, check out [[Packaging/32bit Applications]]<br/>
Quick example to cross compile a package: [[Cheatsheet#Cross-compile_packages]].


== Cross-Compiling a package in nixpkgs ==
== Getting Started ==
Cross-compilation is well supported in nixpkgs since 18.09.
There are two main entry points to configure a Nixpkgs instance for cross compilation. The simplest one is to leverage <code>pkgs.pkgsCross</code>:<syntaxhighlight lang="nix">
The basic idea is to use <code>pkgsCross.platform</code> instead of <code>pkgs</code>:
let
<syntaxHighlight lang=bash>
  pkgs = import <nixpkgs> {};
nix-build '<nixpkgs>' -A pkgsCross.raspberryPi.openssl
in pkgs.pkgsCross.aarch64-multiplatform.hello
</syntaxHighlight>
</syntaxhighlight>The above will provide a derivation result for the hello derivation that can run on an <code>aarch64</code> system. A slightly less simple, but "purer" (i.e. reproducible) snippet involves configuring Nixpkgs as you import it:<syntaxhighlight lang="nix">
let
  pkgs = import <nixpkgs> {
    # localSystem will always default to your current system, meaning it is not essential to define here.
    # It is however, encouraged if you are trying to follow nix's purity model.


== How to obtain a shell with a cross compiler ==
    localSystem = "x86_64-linux"; # This should match the system you are compiling from (your running system)
Create a file <code>crossShell.nix</code> as follows:
    crossSystem = "aarch64-linux"; # This should match the system you are compiling for (your target system)
<syntaxhighlight lang="nix">
  };
with import <nixpkgs> {
in pkgs.hello # The hello derivation will be built for aarch64-linux
   crossSystem = {
</syntaxhighlight>You can perform the same operations using the CLI, though you will probably prefer to use <code>pkgsCross</code> in that case, as its less verbose:<syntaxhighlight lang="bash">
     config = "aarch64-unknown-linux-gnu";
nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy
nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3
</syntaxhighlight>All of the above snippets will resolve to the exact same derivation result, which will provide a binary for GNU Hello that can execute only on an <code>aarch64</code> system. There are many other systems <code>pkgsCross</code> has defined, you can see an exhaustive list of all of them on your system:<syntaxhighlight lang="bash">
$ nix-instantiate --eval --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix-shell -p jq --command 'jq' # nix-legacy
$ nix eval --impure --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix run nixpkgs#jq # nix3
</syntaxhighlight>If you instead prefer to write your systems directly, through <code>localSystem</code> and <code>crossSystem</code>, you can refer to [https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix nixpkgs/lib/systems/examples.nix] for a list of platforms exposed as attributes (though it will be easier to use <code>pkgsCross</code> in this case). These can be directly used in-place for the aforementioned arguments:<syntaxhighlight lang="nix">
let
  lib = import <nixpkgs/lib>;
   pkgs = import <nixpkgs> {
     crossSystem = lib.systems.examples.aarch64-multiplatform;
   };
   };
};
in pkgs.hello
</syntaxhighlight>


mkShell {
== Development ==
   packages = [ zlib ]; # your dependencies here
Using the same ideas as above, we can create development environments which provide us with a compilation suite that can perform cross-compilation for us. A very simple [[Development environment with nix-shell|development shell]] (colloquially called a "devshell") can be written as:<syntaxhighlight lang="nix">
# shell.nix
{
   pkgs ? import <nixpkgs> {
    # localSystem = "x86_64-linux"; # not necessary but encouraged
    crossSystem = "aarch64-linux";
  },
}:
pkgs.mkShell {
  # By default this provides gcc, ar, ld, and some other bare minimum tools
}
}
</syntaxhighlight>
</syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash">
and then use it to obtain a shell:
$ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ...
{{commands|nix-shell crossShell.nix}}
</syntaxhighlight>Given these environment variables, you can run compile your software using the exact same commands with fairly minimal changes (changing hardcoded <code>gcc</code> values into $CC, for example):<syntaxhighlight lang="bash">
The resulting shell contains a cross toolchain and zlib in this example. Note that contrary to native shells, the compiler and some other tools are prefixed: there is no <code>gcc</code> but a <code>aarch64-unknown-linux-gnu-gcc</code>. Some convenience environment variables expand to the prefixed version of tools: <code>$CC</code>, <code>$LD</code>...
$ $CC -o main src/main.c
$ file main
main: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/qa51m8r8rjnigk5hf7sxv0hw7qr7l4bc-glibc-aarch64-unknown-linux-gnu-2.39-52/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped
</syntaxhighlight>The above snippet will have minor differences depending on when you run it, but the main thing to notice is <code>ARM aarch64</code>, which tells us our software was able to successfully cross compile.


Examples of how to specify your target system can be found in [https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix lib/systems/examples.nix]. If the exact system you are targeting is available in this file then you can use the existing definition as in the following example:
=== Declaring dependencies ===
<syntaxhighlight lang="nix">
If you try to declare build-time dependencies within the devshell (such as <code>pkgs.cmake</code>), you will quickly realize that these derivations are actually being built for the <code>crossSystem</code>, making them unusable on your system architecture (see [https://github.com/NixOS/nixpkgs/issues/49526 #49526]). There are ways around this, but in general once you've gotten to this point you should prefer [https://nixos.org/manual/nixpkgs/unstable/#chap-stdenv writing a derivation], which will make it not only easier to write both derivations, but will allow you to follow the recommended practices for using Nix.
let pkgs = import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.armv7l-hf-multiplatform;
};
in
mkShell {}
</syntaxhighlight>


Even shorter:
If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
# shell.nix
let pkgs = import <nixpkgs> {}; in
{
pkgs.pkgsCross.armv7l-hf-multiplatform.mkShell {}
  pkgs ? import <nixpkgs> {
</syntaxhighlight>
    crossSystem = "aarch64-linux";
  },
}:
pkgs.callPackage (
  {
    mkShell,


The examples above do not work as is with build dependencies (<code>nativeBuildInputs</code>). A solution is to use <code>callPackage</code> to enable splicing:
    pkg-config,
<syntaxhighlight lang="nix">
     libGL,
let pkgs = import <nixpkgs> {
   }:
  crossSystem = {
   mkShell {
     config = "aarch64-unknown-linux-gnu";
     # Derivations that must run on the localSystem, referred to as the buildPlatform.
   };
     nativeBuildInputs = [
};
      pkg-config
in
    ];
   pkgs.callPackage (
     {mkShell, pkg-config, zlib}:
     mkShell {
      nativeBuildInputs = [ pkg-config ]; # you build dependencies here
      buildInputs = [ zlib ]; # your dependencies here
    }
  ) {}
</syntaxhighlight>
See also {{issue|49526}}.


== Lazy cross-compiling ==
    # Derivations that must link with the crossSystem, referred to as the targetPlatform.
    buildInputs = [
      libGL
    ];
  }
) {}
</syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>.


If you target "aarch64-unknown-linux-gnu", there is a nice way to reduce amount of cross-compiling and side-step journey to fix cross errors. The idea is to fetch non-essential dependencies from binary cache of regular aarch64 binaries.
For more information regarding the above, namely the usage of <code>nativeBuildInputs</code> and <code>buildInputs</code>, see [https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies stdenv dependencies] for a in-depth explanation. Alternatively, a simplified explanation can be found in a comment on the [https://github.com/NixOS/nixpkgs/pull/50881 nixpkgs repo].


Say we are building SDL2.
== Tips and tricks ==


<syntaxhighlight lang="nix">
=== Executing cross compiled binaries ===
let
By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary.
    # this will use aarch64 binaries from binary cache, so no need to build those
    pkgsArm = import <nixpkgs> {
        config = {};
        overlays = [];
        system = "aarch64-linux";
    };


    # these will be your cross packages
If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly.<syntaxhighlight lang="bash">
    pkgsCross = import <nixpkgs> {
$ ./result
Hello World!
$ ./result-aarch64-linux
Hello World!
</syntaxhighlight>Otherwise, you can use the <code>pkgs.qemu-user</code> to download qemu user space programs (or use any installed by your distro) to run your package easily.<syntaxhighlight lang="bash">
$ ./result
Hello World!
$ qemu-aarch64 ./result-aarch64-linux
Hello World!
</syntaxhighlight>


      overlays = [(self: super: {
=== Leveraging the binary cache ===
You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. However, for some systems, natively compiled binaries are provided. At the time of writing, it only supports <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. As a result, this section is only applicable to a very small number of cross-compilation situations.


        # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
You can leverage the binary cache to correctly substitute some applicable derivations without causing a local build.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">
        inherit (pkgsArm)
let
          xorg libpulseaudio libGL guile systemd libxkbcommon
  # this will use aarch64 binaries from binary cache, so no need to build those
        ;
  pkgsArm = import <nixpkgs> {
    system = "aarch64-linux";
      })];
  };
      crossSystem = {
        config = "aarch64-unknown-linux-gnu";
      };
    };


  # these will be your cross packages
  pkgsCross = import <nixpkgs> {
    overlays = [
      (self: super: {
        # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
        inherit (pkgsArm)
          xorg libpulseaudio libGL guile systemd libxkbcommon
          ;
      })
    ];
    crossSystem = "aarch64-linux";
  };
in pkgsCross.SDL2.override {  
in pkgsCross.SDL2.override {  
      # those shouldn't be neither pkgsCross, nor pkgsArm
  # These should be neither pkgsCross, nor pkgsArm
      # because those trigger
  # because those trigger
      #     cannot execute binary file: Exec format error
  # > cannot execute binary file: Exec format error
      # in this case it was enough to just use buildPackages variants
  # In this case it was enough to just use buildPackages variants,
      # but in general there may be problems
  # but in general, there may be problems
      inherit (pkgsCross.buildPackages)  
  inherit (pkgsCross.buildPackages)  
        wayland wayland-protocols
    wayland wayland-protocols
      ;
    ;
  }
}
</syntaxhighlight>
</syntaxhighlight>


== See also ==


== How to specify dependencies ==
* [[NixOS on ARM]]
 
* [[Packaging/32bit Applications]]
Depending in which if packages are required at build time or at runtime they need to go to different inputs the derivation.
* [[Cheatsheet#Cross-compile packages]]
 
* [https://nixos.org/nixpkgs/manual/#chap-cross Nixpkgs manual on cross compiling]
* If it is used at build-time it's <code>depsBuildXXX</code>
** compiler producing native binaries go to <code>depsBuildBuild</code>;
** compiler producing cross binaries, all setup hooks and programs executed by the builder go to <code>depsBuildHost</code>:
*** common examples: <code>pkg-config, autoreconfHook, makeWrapper, intltool, bison, flex</code>.
* If it is used at run-time it's <code>depsHostXXX</code>. [Static linking doesn't effect this, even if it allows us to forget where things came from.]
** if it’s an interpreter that will be needed by an installed script, it should go in <code>depsHostTarget</code>.
** otherwise it is probably only needed at build time and can go in <code>depsBuildHost</code>
* If it is a tool and "acts" (e.g. helps build) on build-time stuff, then it's <code>depsXXXBuild</code>.
* If it is a tool and "acts" on run-time stuff, then it's <code>depsXXXHost</code>.
* If it is not a tool, it's <code>depsXXX(XXX+1)</code>(build + 1 == host, host +1 == target). For backwards compatibility use <code>nativeBuildInputs</code> instead of <code>depsBuildHost</code> and <code>buildInputs</code> instead of <code>depsHostTarget</code>.
 
Source: https://github.com/NixOS/nixpkgs/pull/50881#issuecomment-440772499
 
 
== References ==
 
* Nixpkgs manual on [https://nixos.org/nixpkgs/manual/#chap-cross cross compiling]
* Nixpkgs manual on [https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies Specifying dependencies]


* [https://matthewbauer.us/blog/beginners-guide-to-cross.html Introduction to Cross Compilation with nix by Matthew Bauer]
* [https://matthewbauer.us/blog/beginners-guide-to-cross.html Introduction to Cross Compilation with nix by Matthew Bauer]
* [https://docs.google.com/presentation/d/11B3Igxj0KmsxgT_UQvKufd5El_oH-T__Sl3JNCZeOnY/edit?usp=sharing Slides from the cross-compilation workshop on 35c3]


== Additional resources ==
* [https://docs.google.com/presentation/d/11B3Igxj0KmsxgT_UQvKufd5El_oH-T__Sl3JNCZeOnY/edit?usp=sharing Slides from the cross-compilation workshop on 35c3]
* [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org])
* [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org])
* [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html Kernel documentation on binfmt_misc]


[[Category:nix]]
[[Category:nix]]
[[Category:Development]]
[[Category:Development]]
Retrieved from "https://wiki.nixos.org/wiki/Cross_Compiling"