Low-level derivations

Writing low-level derivations without the convenience of the standard environment requires understanding the core fundamentals of how Nix works. While writing derivations in this manner is not common, they can be useful when no utility present in the standard environment matches the required use case, as well as being an essential learning tool for understanding why stdenv.mkDerivation is needed in most cases.

In the section below, we'll start by writing a simple derivation that creates a script which displays the message "Hello, world" when run. While developing the Nix file, we'll be encountering the same challenges that led the Nix community to develop more ergonomic tooling. Each obstacle we hit - from missing system dependencies to path management - illustrates a core principle of Nix's purely functional build model. By the end, you'll understand both what makes Nix powerful and why most derivations use abstraction layers.

A Hello, world example

Our goal is simple: creating a script that displays "Hello, world!". If we were to simply create a script ourselves, it might look like this:

≡︎ example.sh

#!/bin/bash
echo "Hello, world!"

When running this script (e.g. $ ./example.sh) we'll get exactly the output we want. However, we're trying to build the script using Nix. Our first approach might be to write a simple derivation similar to:

❄︎ example.nix

derivation {
  name = "hello-world";
  system = builtins.currentSystem;
  builder = "/bin/bash";
  args = [
    "-c"
    ''
      echo '#!/bin/bash' > $out
      echo 'echo "Hello, World!"' >> $out
      chmod +x $out
    ''
  ];
}

There are a number of elements in this Nix file, but remembering the conceptual model of a framework makes quick work of figuring them out: a derivation is a set of inputs along with an executable that produces a deterministic output, following a list of steps. Here our inputs are quite literally the key-value pairs in the derivation attrset: the name of the derivation and the system it's being built for. The executable (called the builder) is the program found at /bin/bash. And the steps are:

1. Write the string #!/bin/bash to the output file ($out);
2. Write the string echo "Hello, World!" to the output file;
3. Make the output file executable.

However, if we try to build this example:

$ nix-build example.nix
this derivation will be built:
  /nix/store/l6s955asmyc22nx7y5pg3ngnzg30r4vb-hello-world.drv
building '/nix/store/l6s955asmyc22nx7y5pg3ngnzg30r4vb-hello-world.drv'...
error: executing '/bin/bash': No such file or directory
error: builder for '/nix/store/l6s955asmyc22nx7y5pg3ngnzg30r4vb-hello-world.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/bin/bash': No such file or directory
       For full logs, run:
         nix log /nix/store/l6s955asmyc22nx7y5pg3ngnzg30r4vb-hello-world.drv

This failure could potentially be surprising. We don't have any syntactic errors, and the logic of the script sure looks good. Let's dive in into what's happening. First, we can see that the derivation is being built under the /nix/store/l6s955asmyc22nx7y5pg3ngnzg30r4vb-hello-world.drv path in the Nix store. The hash for the derivation is how Nix is able to guarantee isolation between different invocations of a derivation with different inputs. The hash is essentially generated as a function of the input attrset.

However, the builder failed, because it couldn't find /bin/bash. This happens because the Nix ecosystem isolates our build environment from the system itself, we cannot use a binary that isn't defined in our derivation, in this case bash itself! In order to use it, we need to add it as an input to our derivation, by referencing the package directly:

❄︎ example.nix

let
  pkgs = import <nixpkgs> { };
in
derivation {
  name = "hello-world";
  system = builtins.currentSystem;
  builder = "${pkgs.bash}/bin/bash";
  args = [
    "-c"
    ''
      echo '#!${pkgs.bash}/bin/bash' > $out
      echo 'echo "Hello, World!"' >> $out
      chmod +x $out
    ''
  ];
}

Building the package again, we'll come across another surprising error:

$ nix-build example.nix
this derivation will be built:
  /nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv
building '/nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv'...
bash: line 3: chmod: command not found
error: builder for '/nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv' failed with exit code 127;
       last 1 log lines:
       > bash: line 3: chmod: command not found
       For full logs, run:
         nix log /nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv

Turns out chmod is not part of our build environment either; that's how minimal the low-level derivation environment is! However, before we fix the script in a similar manner, let's observe another important detail: the store path. Compare the build location of this example, and the previous one; the SHAs are different! That is because our derivation has different inputs to the one before, therefore it's built under a different location in the Nix store. And to highlight this, without fixing the script, let's build it again:

$ nix-build example.nix
this derivation will be built:
  /nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv
building '/nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv'...
bash: line 3: chmod: command not found
error: builder for '/nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv' failed with exit code 127;
       last 1 log lines:
       > bash: line 3: chmod: command not found
       For full logs, run:
         nix log /nix/store/kbxagjwnr916bg4ajn5fz13bmc3lmr8v-hello-world.drv

The SHA is the same, because our inputs haven't changed. This is the essence to why derivations are pure functions of their inputs.

In rewriting our derivation, we need to import chmod, which is part of the coreutils package. We could point to the binary similarly to how we did it before, but for variation, let's modify our PATH variable instead:

❄︎ example.nix

let
  pkgs = import <nixpkgs> { };
in
derivation {
  name = "hello-world";
  system = builtins.currentSystem;
  builder = "${pkgs.bash}/bin/bash";
  args = [
    "-c"
    ''
      export PATH="$PATH:${pkgs.coreutils}/bin"
      echo '#!${pkgs.bash}/bin/bash' > $out
      echo 'echo "Hello, World!"' >> $out
      chmod +x $out
    ''
  ];
}

Let's rebuild it again, noticing how the SHA will change again, as we changed our derivation's inputs:

$ nix-build example.nix
this derivation will be built:
  /nix/store/ipjigfvbj371a5g966xbnj790sjvq5hp-hello-world.drv
building '/nix/store/ipjigfvbj371a5g966xbnj790sjvq5hp-hello-world.drv'...
/nix/store/lsm49a32nmc3ihi3f119p17lvwadr5ms-hello-world

This time it built successfully. Finally, let's run our script:

$ ./result
Hello, world!

As mentioned above, building derivations this way can be unwieldy. But working through this example should shed some light onto why the community has built the standard environment, providing utility functions to build derivations for the most common use cases.

Standard environment

For practicality purposes, let's compare this to the standard environment way of doing this. Using stdenv.mkDerivation gives us access to the common Linux utilities pre-included for us, so we don't have to do the package importing ourselves:

❄︎ example.nix

{ pkgs ? import <nixpkgs> {} }:

pkgs.stdenv.mkDerivation {
  name = "hello-world";
  
  buildCommand = ''
    echo '#!/bin/bash' > $out
    echo 'echo "Hello, World!"' >> $out
    chmod +x $out
  '';
}

Nixpkgs utilities

Even further up the chain of abstractions, Nixpkgs contains many pre-built utilities for us that handle some of the configuration involved in the standard environment as well. In our case, we can use the writeShellScript:

❄︎ example.nix

{ pkgs ? import <nixpkgs> {} }:

pkgs.writeShellScript "hello-world" ''
  echo "Hello, World!"
''