Derivations: Difference between revisions
m typo |
|||
| (15 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
{{dock|{{external|The Nix Reference Manual|§ 5.4.1. {{manual|nix|language/derivations|Derivations}}}}{{external|The Nixpkgs Reference Manual|{{manual|nixpkgs|sec-functions-library-derivations|lib.derivations}}}}{{external|The Nixpkgs Reference Manual|{{manual|nixpkgs|sec-using-stdenv|Using stdenv}}}}}} | |||
'''Derivations''' are the [[Nix ecosystem]] way of describing any reproducible build process. While [[NixOS]] comes with a plethora of packages, applications and options, there will inevitably come a time when you need to build an application, a library, a package, etc. that is not available ''off the shelf'' already — those are all derivations ''under the hood''. This makes the build process ''reproducible'' and ''predictable''; without changing the derivation's input configuration, the output will remain the same. In essence, a derivation is a ''pure function'' of an executable, and a set of input configuration, that produces exactly the same output for every invocation, in unique locations on the filesystem. | '''Derivations''' are the [[Nix ecosystem]] way of describing any reproducible build process. While [[NixOS]] comes with a plethora of packages, applications and options, there will inevitably come a time when you need to build an application, a library, a package, etc. that is not available ''off the shelf'' already — those are all derivations ''under the hood''. This makes the build process ''reproducible'' and ''predictable''; without changing the derivation's input configuration, the output will remain the same. In essence, a derivation is a ''pure function'' of an executable, and a set of input configuration, that produces exactly the same output for every invocation, in unique locations on the filesystem. | ||
== Motivation == | == Motivation == | ||
While the need to build software, package libraries and execute build processes is clear to anyone using any operating system, the natural question that may arise is ''Why go out of our way dealing with this complicated process when I can just run a few terminal commands?'' For most distributions, the answer is ''they don't do things this way''. Most Linux distributions, and most operating systems for that matter, are designed to change over time; the same build process will yield different results each time it's invoked. For example, remember trying to build a package twice; the first time you build it the installation will be successful, but the second time it's built you might get an error about the paths it's trying to write to already | While the need to build software, package libraries and execute build processes is clear to anyone using any operating system, the natural question that may arise is ''Why go out of our way dealing with this complicated process when I can just run a few terminal commands?'' For most distributions, the answer is ''they don't do things this way''. Most Linux distributions, and most operating systems for that matter, are designed to change over time; the same build process will yield different results each time it's invoked. For example, remember trying to build a package twice; the first time you build it the installation will be successful, but the second time it's built you might get an error about the paths it's trying to write to already existing. Build processes in most Linux distributions are ''stateful'', the context in which they're run might change as you're using that system. | ||
However, the Nix ecosystem is ''fundamentally'' different in this regard; when you build a derivation, a unique path in the [[Nix store]] is assigned, and all possible outputs (including filesystem operations) produced by it will be persisted under that path. No other derivation can modify those files; the result of the derivation is '''uniquely determined''' by its input configuration, and subsequent reruns will produce ''exactly'' the same outputs, | However, the Nix ecosystem is ''fundamentally'' different in this regard; when you build a derivation, a unique path in the [[Nix store]] is assigned, and all possible outputs (including filesystem operations) produced by it will be persisted under that path. No other derivation can modify those files; the result of the derivation is '''uniquely determined''' by its input configuration, and subsequent reruns will produce ''exactly'' the same outputs {{foot|In actuality, if the derivation build was successful, Nix will recognise that the inputs haven't changed, and dynamically link to the already build Nix store path.}}. Any potential issues regarding being able to reproduce a build process are addressed ''by design'', if a derivation was successful once, it will always build successfully as long as its inputs don't change. | ||
Derivations are a powerful fundamental part of [[Nix]] and provide the core platform for managing packages in NixOS. Every package and library you include in your NixOS configuration is a derivation. | Derivations are a powerful fundamental part of [[Nix]] and provide the core platform for managing packages in NixOS. Every package and library you include in your NixOS configuration is a derivation. | ||
== Definition == | == Definition == | ||
A '''derivation''' is defined as ''a specification for running an executable on precisely defined input files to repeatably produce output files at uniquely determined file system paths'' | A '''derivation''' is defined as ''a specification for running an executable on precisely defined input files to repeatably produce output files at uniquely determined file system paths''{{cite manual|nix|language/derivations|5.4.1|Derivations}}. Simply put, it describes a set of steps to take some input and produce some output in a deterministic manner. | ||
Derivations can be written manually using the <code>derivation</code> function; this is the most fundamental way in which they can be defined. However, since this low-level function is quite simple, building derivations this way can easily become unwieldy and repetitive. To aid in the process of creating derivations, [[Nixpkgs]] contains [https://nixos.org/manual/nixpkgs/unstable/#chap-stdenv the standard environment] (also known as the <code>stdenv</code>), which provides a set of utility functions for the most common derivation types (e.g. a [[Python libraries|Python package]], a shell script, a [[Docker]] container, etc.) | Derivations can be written manually using the <code>derivation</code> function; this is the most fundamental way in which they can be defined. However, since this low-level function is quite simple, building derivations this way can easily become unwieldy and repetitive. To aid in the process of creating derivations, [[Nixpkgs]] contains [https://nixos.org/manual/nixpkgs/unstable/#chap-stdenv the standard environment] (also known as the <code>stdenv</code>), which provides a set of utility functions for the most common derivation types (e.g. a [[Python libraries|Python package]], a shell script, a [[Docker]] container, etc.) | ||
== | == Writing a derivation == | ||
The most fundamental way to create a derivation is using the built-in <code>derivation</code> function. While you'll rarely write derivations this way in practice, understanding the low-level mechanics helps clarify what higher-level tools are doing for you. An example is provided in the first subsection. While it wouldn't be feasible to write every single derivation in this way, working through an example is an important step in one's Nix ecosystem journey. The latter subsections include more ''production-ready'' examples using the higher level utilities available in the ecosystem. | |||
Regardless of how they're built, derivations take an ''input'' and produce an ''output'' through a series of ''steps''. For most packages, the inputs refer to the source files, the steps refer to the compilation process and are called [[Derivations#Phases|phases]], and the outputs refer to finalized executable binaries written to some file/directory. This sequence of events can be well described within a standard environment, which the latter sections address. | |||
=== Low-level derivations === | |||
{{Main|Low-level derivations}} | |||
As mentioned above, writing derivations in this manner can quickly become unwieldy and unfeasible. However, in order to understand why, do check out the main article above to follow the process of writing a derivation while hitting every hurdle that you might hit when building one yourself. Doing so is the best way to understand conceptually how derivations operate. You can see what a well defined low-level derivation might look like, in this case simply creating a script that displays the message ''"Hello, world!"'': | |||
{{File|example.nix|nix|<nowiki> | |||
let | |||
pkgs = import </nowiki><<nowiki>nixpkgs</nowiki>><nowiki> { }; | |||
in | |||
derivation { | |||
name = "hello-world"; | |||
system = builtins.currentSystem; | |||
builder = "${pkgs.bash}/bin/bash"; | |||
args = [ | |||
"-c" | |||
'' | |||
export PATH="$PATH:${pkgs.coreutils}/bin" | |||
echo '#!${pkgs.bash}/bin/bash' </nowiki>><nowiki> $out | |||
echo 'echo "Hello, World!"' </nowiki>>><nowiki> $out | |||
chmod +x $out | |||
'' | |||
]; | |||
}</nowiki>}} | |||
=== The result of a derivation === | |||
When building a derivation, its result will be added to the Nix store, and a symlink will be provided in the current directory, called <code>result</code>. | |||
The <code>$out</code> variable above has a special meaning in the Nix context: it points to the file that will become the result of the derivation. Thus executing <code>chmod +x $out</code> makes the derivation executable. Because directories on POSIX systems are files themselves, the derivation's result can be an entire directory of files. | |||
The actual directory of <code>$out</code> is an implementation detail abstracted away by Nix and the stdenv builder. Anything placed within <code>$out</code> will then be part of the final derivation. Most derivation aim to follow a FHS-like structure, with the following common subdirectories: | |||
* <code>$out/bin</code> contains binaries; | |||
* <code>$out/lib</code> contains shared objects | |||
* <code>$out/include</code> contains headers | |||
In the context of systems like NixOS or [[Home Manager]], these paths will usually be symlinked into the top-level derivation's directories (the resulting system build). | |||
=== Standard environment derivations === | |||
The standard environment provides us with a useful utility function for creating derivations called <code>stdenv.mkDerivation</code>. Among other things, it ensures we have all the common binaries and libraries we might expect from a Linux build environment already available to us. For example, the derivation above can be rewritten, removing the need to point to the bash and <code>coreutil</code> paths ourselves, as follows:{{File|example.nix|nix|<nowiki> | |||
{ pkgs ? import </nowiki><<nowiki>nixpkgs</nowiki>><nowiki> {} }: | |||
pkgs.stdenv.mkDerivation { | |||
name = "hello-world"; | |||
buildCommand = '' | |||
echo '#!/bin/bash' </nowiki>><nowiki> $out | |||
echo 'echo "Hello, World!"' </nowiki>>><nowiki> $out | |||
chmod +x $out | |||
''; | |||
} | |||
</nowiki>|highlight=}} | |||
==== Including source code ==== | |||
{{Main|Fetchers}} | |||
However, when building applications, packages, or derivations in general, we are almost never manually writing the source code ourselves within the derivation's build steps. Usually, we need to point to the derivation's source code. The <code>stdenv.mkDerivation</code> function takes in an attribute called <code>src</code>, which points to a location that should be included as one of the derivation's inputs. | |||
When building the derivation, the source code will be copied in the Nix store before proceeding with the build. The reason for this is due to the nature of source code (and external files in general themselves): they ''change over time''. Source code gets edited, improved, reverted, etc. By copying the source code ''at the time of the derivation'' to the Nix store, it becomes ''immutable''; any subsequent changes would result in it getting copied to a ''different path'' in the store. Thus, Nix ensures the guarantees we expect from a derivation: it still is a deterministic output, a function of its inputs. | |||
Grabbing the source code can be done in a number of different ways: it can be a store path, a location on the current system, or it can be downloaded via a [https://nixos.org/manual/nixpkgs/unstable/#chap-pkgs-fetchers fetcher], which is a special-purpose utility function for exactly this task:<syntaxhighlight lang="nix"> | |||
stdenv.mkDerivation { | stdenv.mkDerivation { | ||
src = ./relative-path/to/src; | |||
# or | |||
src = fetchFromGitHub { | src = fetchFromGitHub { | ||
owner = "torvalds"; | owner = "torvalds"; | ||
| Line 34: | Line 92: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== | === Nixpkgs packages === | ||
The Nixpkgs repository is relevant in two main ways: it itself contains many higher-level utilities that abstract over the <code>stdenv.mkDerivation</code> requirements for the most common use cases, and every [[Nixpkgs]] package is a derivation in of itself. | |||
==== Utility functions ==== | |||
For example, the derivation at the beginning of this section can be rewritten in just four lines using the <code>writeShellScript</code> utility package. It handles most of the little setup required by the standard environment for us. The majority of the derivations that need to be written almost definitely can make use of one of these utility packages. For example:{{File|example.nix|nix|{ pkgs ? import <nixpkgs> {} }: | |||
pkgs.writeShellScript "hello-world" '' | |||
echo "Hello, World!" | |||
''}} | |||
==== Package metadata ==== | |||
Derivations meant to be included as part of Nixpkgs usually include some metadata under the meta attribute. Common fields include: | |||
* <code>meta.homepage</code> and <code>meta.description</code> are used to describe and link to relevant information about the upstream source this derivation builds; | |||
* <code>meta.platforms</code> is useful for Nixpkgs to determine whether a package can be built on a different system (and whether to allow so in evaluation); | |||
* and <code>meta.licenses</code> is useful to check if a package has a suitable license that allows for re-distribution (caching in the official [[Binary Cache|binary cache]] or in one of the community caches); | |||
</ | * <code>meta.mainProgram</code> describes the binary that can be considered the "main" program. For example, the cmake derivation would have this attribute set to <code>cmake</code>, which would be resolved as <code>$out/bin/cmake</code> when needed. | ||
A fully exhaustive documentation on all meta-attributes can be found in the [https://nixos.org/manual/nixpkgs/unstable/#chap-meta Nixpkgs manual]. | |||
== Phases == | |||
{{Wip|date=29 June 2025|scope=section}} | |||
The vast majority of derivations follow a fairly established set of stages to their building—for example, configuration, building, installation, etc. Thus, the vast majority of derivations that use the standard environment's <code>stdenv.mkDerivation</code>, and any language-specific builders that make use of it, follow a convention of grouping the derivation steps in '''phases''', which follow a common established pattern. However, this setup can be overridden for specific packaging systems (e.g. docker, etc.); to reduce complexity, this section will start with the generic phase setup first, expanding on how these can be overridden or modified in later sections. | |||
By default, derivations follow a standard pre-established set of phases, that are meant to mirror how the vast majority of packages are built. Each phase is meant to control a distinct part of the build process, an approach largely inspired by GNU's Autoconf convention of <code>./configure</code>, <code>make</code>, and <code>make install</code>. | |||
Taking the basic definition of a derivation to heart, that of being ''a specification for supplying a set of steps to a builder'', each phase is defined to be a sequence of steps written in shell syntax. The phase scripts can use any program or package provided by the standard environment, such as <code>core-utils</code>, <code>gcc</code>, <code>gnumake</code>, <code>bash</code>, <code>gnuinstall</code>, etc. | |||
By common convention, standard environment derivations contain the following phases, executed in this sequence: | |||
# Unpack: handles the preparation of the build environment (e.g. extracting archives, touching files, etc.); | |||
# Patch: handles changes to the underlying source code (e.g. patching bugs, adapting the source code to work in a Nix environment, etc.); | |||
# Configure: handles the configuration of the build environment, for example detecting system capabilities and setting build parameters; | |||
# Build: compiles the source code into binaries, bytecode, or otherwise a distributable form of the source code; | |||
# Check: performs any tests on the compiled package, for example the package's test suite; | |||
# Install: copies the build artefacts to the output directory, handling any needed changes (e.g. directory structure reorganisation); | |||
# Fixup: process the output artefacts to work in a Nix environment (e.g.: strip binaries, override ELF paths, handle dynamic library linking, etc.); | |||
# Install Check: performs any tests on the final output, essentially acting as a integration test into the Nix environment; | |||
# Dist: creates distribution archives (rarely used). | |||
{{references|group=footnotes|heading=Notes}} | |||
{{references}} | |||
[[Category:Nix]] | [[Category:Nix]] | ||