NixOS Wiki

Cross Compiling: Difference between revisions

← Older edit
Visual
Frontear (talk | contribs)
27 edits
majorly overhaul the cross-compilation page with more details, better examples, and modernized nix code.
m add another way via nix-build cli to cross compile
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Expansion}}
{{Expansion}}
Cross compiling is well supported in [[Nixpkgs]] since 18.09<sup>[citation needed]</sup>.
[[Nixpkgs]] provides excellent support in configuring it for cross-platform compiling tasks since 18.09<sup>[citation needed]</sup>.
 
In order to prepare Nixpkgs for a cross-compiling environment, it needs to be aware of both the platform that performs the build-step, and the platform that will execute the resulting binaries. The former is referred to as the <code>buildPlatform</code>, while the latter is <code>hostPlatform</code>.<blockquote>If you were compiling a program from your system for a Raspberry PI, you would be the <code>buildPlatform</code> whereas the Raspberry PI would be the <code>hostPlatform</code>.</blockquote>Furthermore, in order to provide more granular control to declaring dependencies in these environments, Nixpkgs derivations expose an exhaustive set of attributes that can explicitly define when are where dependencies are required. A full reference to these can be found in the [https://nixos.org/manual/nixpkgs/unstable/#ssec-stdenv-dependencies-propagated Nixpkgs manual].


== Getting Started ==
== Getting Started ==
There are two main entry points to configure a Nixpkgs instance for cross compilation. The simplest one is to leverage <code>pkgs.pkgsCross</code>:<syntaxhighlight lang="nix">
Nixpkgs exposes two configuration attributes that map internally to the expected behaviors of the build/host platforms as described above. These attributes can be set when importing Nixpkgs as a Nix expression:<syntaxhighlight lang="nix">
let
let
   pkgs = import <nixpkgs> {};
   pkgs = import <nixpkgs> {
in pkgs.pkgsCross.aarch64-multiplatform.hello
    localSystem = "x86_64-linux"; # buildPlatform
</syntaxhighlight>The above will provide a derivation result for the hello derivation that can run on an <code>aarch64</code> system. A slightly less simple, but "purer" (i.e. reproducible) snippet involves configuring Nixpkgs as you import it:<syntaxhighlight lang="nix">
    crossSystem = "aarch64-linux"; # hostPlatform
  };
in pkgs.hello
</syntaxhighlight>The above will provide a derivation result for the hello derivation that can run on an <code>aarch64-linux</code> system. This can sometimes be tedious especially for common <code>hostPlatform</code> targets. Fortunately, Nixpkgs exposes a <code>pkgsCross</code> attribute that provides pre-configured cross compiling targets. The snippet above converted to using <code>pkgsCross</code> can be shorted to:<syntaxhighlight lang="nix">
let
let
   pkgs = import <nixpkgs> {
   pkgs = import <nixpkgs> {
    # localSystem will always default to your current system, meaning it is not essential to define here.
     localSystem = "x86_64-linux";
    # It is however, encouraged if you are trying to follow nix's purity model.
 
     localSystem = "x86_64-linux"; # This should match the system you are compiling from (your running system)
    crossSystem = "aarch64-linux"; # This should match the system you are compiling for (your target system)
   };
   };
in pkgs.hello # The hello derivation will be built for aarch64-linux
in pkgs.pkgsCross.aarch64-multiplatform.hello
</syntaxhighlight>You can perform the same operations using the CLI, though you will probably prefer to use <code>pkgsCross</code> in that case, as its less verbose:<syntaxhighlight lang="bash">
</syntaxhighlight>You can perform the same operations using the CLI, and Nix will correctly evaluate the <code>localSystem</code> based on your current system:<syntaxhighlight lang="bash">
nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy
nix-build '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello # nix-legacy
nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs> {}).lib.systems.examples.aarch64-multiplatform' -A hello # alternative way
nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3
nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello # nix3
</syntaxhighlight>All of the above snippets will resolve to the exact same derivation result, which will provide a binary for GNU Hello that can execute only on an <code>aarch64</code> system. There are many other systems <code>pkgsCross</code> has defined, you can see an exhaustive list of all of them on your system:<syntaxhighlight lang="bash">
</syntaxhighlight>All of the above snippets will resolve to the exact same derivation result, which will provide a binary for GNU Hello that can execute only on an <code>aarch64</code> system. There are many other systems <code>pkgsCross</code> has defined, you can see an exhaustive list of all of them on your system:<syntaxhighlight lang="bash">
$ nix-instantiate --eval --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix-shell -p jq --command 'jq' # nix-legacy
$ nix-instantiate --eval --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix-shell -p jq --command 'jq' # nix-legacy
$ nix eval --impure --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix run nixpkgs#jq # nix3
$ nix eval --impure --expr 'builtins.attrNames (import <nixpkgs> {}).pkgsCross' --json | nix run nixpkgs#jq # nix3
</syntaxhighlight>If you instead prefer to write your systems directly, through <code>localSystem</code> and <code>crossSystem</code>, you can refer to [https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix nixpkgs/lib/systems/examples.nix] for a list of platforms exposed as attributes (though it will be easier to use <code>pkgsCross</code> in this case). These can be directly used in-place for the aforementioned arguments:<syntaxhighlight lang="nix">
</syntaxhighlight>If you instead prefer to write your systems directly, through <code>localSystem</code> and <code>crossSystem</code>, you can refer to [https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix nixpkgs/lib/systems/examples.nix] for examples of platforms exposed as attributes. These can be directly used in-place for the aforementioned arguments:<syntaxhighlight lang="nix">
let
let
   lib = import <nixpkgs/lib>;
   lib = import <nixpkgs/lib>;
   pkgs = import <nixpkgs> {
   pkgs = import <nixpkgs> {
    #localSystem = ...;
     crossSystem = lib.systems.examples.aarch64-multiplatform;
     crossSystem = lib.systems.examples.aarch64-multiplatform;
   };
   };
Line 33: Line 36:


== Development ==
== Development ==
=== Basics ===
Using the same ideas as above, we can create development environments which provide us with a compilation suite that can perform cross-compilation for us. A very simple [[Development environment with nix-shell|development shell]] (colloquially called a "devshell") can be written as:<syntaxhighlight lang="nix">
Using the same ideas as above, we can create development environments which provide us with a compilation suite that can perform cross-compilation for us. A very simple [[Development environment with nix-shell|development shell]] (colloquially called a "devshell") can be written as:<syntaxhighlight lang="nix">
# shell.nix
# shell.nix
{
{
   pkgs ? import <nixpkgs> {
   pkgs ? import <nixpkgs> {
     # localSystem = "x86_64-linux"; # not necessary but encouraged
     localSystem = "x86_64-linux";
     crossSystem = "aarch64-linux";
     crossSystem = "aarch64-linux";
   },
   },
}:
}:
pkgs.mkShell {
pkgs.callPackage (
  # By default this provides gcc, ar, ld, and some other bare minimum tools
  {
}
    mkShell,
  }:
  mkShell {
    # By default this provides gcc, ar, ld, and some other bare minimum tools
  }
) { }
</syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash">
</syntaxhighlight>Entering this development shell via <code>nix-shell shell.nix</code> will add the relevant compiler tools to your PATH temporarily. Similar to other Linux systems, all cross-compiling tools are prefixed with relevant platform prefixes, which means simply typing <code>gcc</code> will not work. However, the provided <code>mkShell</code> will introduce environment variables for your devshell, such as <code>$CC</code>, <code>$AR</code>, <code>$LD</code>, and more. At the time of writing, official documentation on an exhaustive list of these variables does not exist, but you can view them for your devshell through the command-line:<syntaxhighlight lang="bash">
$ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ...
$ $EDITOR $(nix-build ./shell.nix) # opens your EDITOR with a massive bash script full of declare -x ...
Line 56: Line 68:


If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix">
If you would prefer to continue building within the devshell, you can use [https://nixos.org/guides/nix-pills/13-callpackage-design-pattern callPackage], which will ''magically'' resolve the dependencies for the correct architecture, provided you place them in the correct attributes:<syntaxhighlight lang="nix">
# shell.nix
{
{
   pkgs ? import <nixpkgs> {
   pkgs ? import <nixpkgs> {
    localSystem = "x86_64-linux";
     crossSystem = "aarch64-linux";
     crossSystem = "aarch64-linux";
   },
   },
Line 65: Line 77:
   {
   {
     mkShell,
     mkShell,
 
    hello,
     pkg-config,
     pkg-config,
     libGL,
     libGL,
   }:
   }:
   mkShell {
   mkShell {
     # Derivations that must run on the localSystem, referred to as the buildPlatform.
    strictDeps = true;
     # host/target agnostic programs
    depsBuildBuild = [
      hello
    ];
    # compilers & linkers & dependecy finding programs
     nativeBuildInputs = [
     nativeBuildInputs = [
       pkg-config
       pkg-config
     ];
     ];
 
     # libraries
     # Derivations that must link with the crossSystem, referred to as the targetPlatform.
     buildInputs = [
     buildInputs = [
       libGL
       libGL
     ];
     ];
   }
   }
) {}
) { }
</syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>.
</syntaxhighlight>The above snippet will drop you into a devshell that provides <code>pkg-config</code> as a native binary (accessible through <code>$PKG_CONFIG</code>), while also allowing linking to a valid <code>libGL</code> for the <code>crossSystem</code>.


For more information regarding the above, namely the usage of <code>nativeBuildInputs</code> and <code>buildInputs</code>, see [https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies stdenv dependencies] for a in-depth explanation. Alternatively, a simplified explanation can be found in a comment on the [https://github.com/NixOS/nixpkgs/pull/50881 nixpkgs repo].
For more information regarding the above, namely the usage of <code>nativeBuildInputs</code> and <code>buildInputs</code>, see [https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies stdenv dependencies] for a in-depth explanation. Alternatively, a simplified explanation can be found in a comment on the [https://github.com/NixOS/nixpkgs/pull/50881 Nixpkgs repo].


== Tips and tricks ==
== Tips and tricks ==
Line 90: Line 106:
By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary.
By using [[QEMU]], we can natively execute a cross-compiled binary through an emulation layer. This will result in degraded performance but is very suitable for testing the functionality of a binary.


If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly.<syntaxhighlight lang="bash">
If you are on NixOS, this functionality can be provided automatically on any cross-compiled binary by setting [https://nixos.org/manual/nixos/unstable/options#opt-boot.binfmt.emulatedSystems boot.binfmt.emulatedSystems] in your configuration. After rebuilding, attempting to run a cross-compiled binary will automatically invoke <code>qemu</code> indirectly through the [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html binfmt_misc kernel feature].<syntaxhighlight lang="bash">
$ ./result
$ ./result
Hello World!
Hello World!
Line 103: Line 119:


=== Leveraging the binary cache ===
=== Leveraging the binary cache ===
You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. However, for some systems, natively compiled binaries are provided. At the time of writing, it only supports <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. As a result, this section is only applicable to a very small number of cross-compilation situations.
You will likely have noticed that resolving derivations through either pkgsCross or a configured Nixpkgs instance results in your system needing to build the binary. This is because cross-compiled binaries are not cached on the official [[Binary Cache|binary cache]]. Fortunately, there are a small set of systems that are actively built and cached officially. At the time of writing, this only includes <code>aarch64-linux</code>, <code>aarch64-darwin</code>, <code>i686-linux</code>, <code>x86_64-linux</code>, and <code>x86_64-darwin</code>. If your platform targets include these, you may be able to leverage a slight hack to avoid large-scale builds.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time, and the storage requirements will be higher due to duplicate(but different system) packages.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">
 
You can leverage the binary cache to correctly substitute some applicable derivations without causing a local build.<blockquote>Please note that this is not recommended, as it hacks around some internal details of Nixpkgs which are subject to change at any time.</blockquote>An example of this using <code>pkgs.SDL2</code>:<syntaxhighlight lang="nix">
let
let
   # this will use aarch64 binaries from binary cache, so no need to build those
   # this will use aarch64 binaries from binary cache, so no need to build those
   pkgsArm = import <nixpkgs> {
   pkgsArm = import <nixpkgs> {
     system = "aarch64-linux";
     localSystem = "aarch64-linux";
   };
   };


Line 118: Line 132:
         # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
         # we want to hack on SDL, don't want to hack on those. Some even don't cross-compile
         inherit (pkgsArm)
         inherit (pkgsArm)
           xorg libpulseaudio libGL guile systemd libxkbcommon
           xorg
          libpulseaudio
          libGL
          guile
          systemd
          libxkbcommon
           ;
           ;
       })
       })
     ];
     ];
    localSystem = "x86_64-linux";
     crossSystem = "aarch64-linux";
     crossSystem = "aarch64-linux";
   };
   };
in pkgsCross.SDL2.override {  
in
   # These should be neither pkgsCross, nor pkgsArm
pkgsCross.callPackage (
  # because those trigger
  {
  # > cannot execute binary file: Exec format error
    SDL2,
   # In this case it was enough to just use buildPackages variants,
    wayland,
   # but in general, there may be problems
    wayland-protocols,
   inherit (pkgsCross.buildPackages)
    wayland-scanner,
     wayland wayland-protocols
  }:
    ;
  SDL2.override {
    inherit
      wayland
      wayland-protocols
      wayland-scanner
      ;
   }
) { }
</syntaxhighlight>
 
=== Using cross-compiled packages in the host NixOS config ===
 
First, configure your builder as a trusted [[Binary Cache|binary cache]].
Then copy the cross-compiled package to the host using <code>nix copy</code>.
 
Finally modify your <code>configuration.nix</code> as follows:
 
<syntaxhighlight lang="nix">
let pkgsCross = import <nixpkgs> {
   localSystem = "x86_64-linux"; # <-- put your builder's platform here and below
  hostSystem = "x86_64-linux";
   crossSystem = "aarch64-linux"; # <-- put your host's platform here
};
in
{
   environment.systemPackages = [
     pkgsCross.hello # <-- put your cross-compiled package here
  ];
}
}
</syntaxhighlight>
</syntaxhighlight>
Line 147: Line 194:


* [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org])
* [https://logs.nix.samueldr.com/nixos/2018-08-03#1533327247-1533327971; 2018-08-03 discussion on #nixos] ([https://matrix.to/#/!AinLFXQRxTuqNpXyXk:matrix.org/$15333274371713496LOAor:matrix.org Mirror of chat on Matrix.org])
* [https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html Kernel documentation on binfmt_misc]


[[Category:nix]]
[[Category:nix]]
[[Category:Development]]
[[Category:Development]]
Retrieved from "https://wiki.nixos.org/wiki/Cross_Compiling"