Overlays

From NixOS Wiki
Revision as of 14:07, 18 May 2021 by imported>Gytis-ivaskevicius

Overlays provide a method to extend and change nixpkgs. They replace constructs like packageOverride and overridePackages.

Consider a simple example of setting the default proxy in Google Chrome:

let overlay1 = self: super:
{
   google-chrome = super.google-chrome.override {
     commandLineArgs =
       "--proxy-server='https=127.0.0.1:3128;http=127.0.0.1:3128'";
   };
};

Data flow of overlays

The data flow around overlays, especially regarding super and self arguments can be a bit confusing if you are not familiar with how overlays work. This graph shows the data flow:

 

Here the main package set is extended with two overlays, ext-1 and ext-2. x // y is represented by a // box with x coming in from the left and y from above.

As you can see, self is the same for every stage, but super comes from only the stage before. So when you define an attribute foo in the set to override it, within that overlay self.foo will be it's version, and super.foo will be the non-overriden version. This is why you see patterns like foo = super.foo.override { ... }.

The names self and super might remind you of inheritance in object-oriented languages. In fact, overlays are exactly the same thing as subclasses, with regards to overriding and calling methods. This data flow is also how objects know which method to call. This is probably why the two arguments got their names, too.

Data flow of overlays (Better explanation???)

Source: https://discourse.nixos.org/t/how-to-exclude-packages/13039/4

I recommend final: prev. That's also easier to explain. The first argument is nixpkgs with your overlay applied, and the second argument is nixpkgs without your overlay. So the “final” nixpkgs and the “previous” nixpkgs. This allows you to access things you defined in your overlay along with things from nixpkgs itself.

final: prev: { f = final.firefox; }

would work, but

final: prev: { f = prev.firefox; }

would make more sense.

This could be useful:

final: prev: {
  firefox = prev.firefox.override { ... };
  myBrowser = final.firefox;
}


And

final: prev: firefox = final.firefox.override { ... };

would cause infinite recursion.

Using overlays

Applying overlays manually

In standalone nix code

When writing standalone nix code, for example a shell.nix for a project, one usually starts by importing nixpkgs: let pkgs = import <nixpkgs> {}. To use an overlay in this context, replace that by:

import <nixpkgs> { overlays = [ overlay1 overlay2 ]; }

In NixOS

In /etc/nixos/configuration.nix, use the nixpkgs.overlays option:

{ config, pkgs, lib, ... }:
{
   # [...]
   nixpkgs.overlays = [ (self: super: /* overlay goes here */) ];
}

Note that this does not impact the usage of nix on the command line, only your NixOS configuration.

In Home Manager

In ~/.config/nixpkgs/home.conf, use the nixpkgs.overlays option:

{ config, pkgs, lib, ... }:
{
   # [...]
   nixpkgs.overlays = [ (self: super: /* overlay goes here */) ];
}


Note that this does not impact the usage of nix on the command line or in your your NixOS configuration, only your home-manager configuration.

Applying overlays automatically

On the user level

A list of overlays placed into ~/.config/nixpkgs/overlays.nix will be automatically loaded by all nix tools run as your user (hence not nixos-rebuild).

Alternatively, you can put each overlay in its own .nix file under your ~/.config/nixpkgs/overlays directory.

On the system level

If you want your overlays to be accessible by nix tools and also in the system-wide configuration, add nixpkgs-overlays to your NIX_PATH:

NIX_PATH="$NIX_PATH:nixpkgs-overlays=/etc/nixos/overlays"

Currently nixos-rebuild only works with a <nixpkgs-overlays> path that is a directory.

There is a configuration option nixpkgs.overlays. Overlays set here will not be automatically applied by nix tools.

Using nixpkgs.overlays from configuration.nix as <nixpkgs-overlays> in your NIX_PATH

Configuration below will allow all of the Nix tools to see the exact same overlay as is defined in your configuration.nix in the nixpkgs.overlays option.

The core of the idea here is to point the nixpkgs-overlays element of NIX_PATH to a "compatibility" overlay, which will load all of the overlays defined in your NixOS system configuration and apply them to its own input. Thus, when various Nix tools attempt to load the overlays from the nixpkgs-overlays element of NIX_PATH, they will get contents of overlays defined in your NixOS system config.

First, in the configuration.nix file, depending on whether your configuration.nix already defines nix.nixPath, add one of these definitions:

{ config, pkgs, options, ... }: {
  # With an existing `nix.nixPath` entry:
  nix.nixPath = [
    # Add the following to existing entries.
    "nixpkgs-overlays=/etc/nixos/overlays-compat/"
  ];

  # Without any `nix.nixPath` entry:
  nix.nixPath =
    # Prepend default nixPath values.
    options.nix.nixPath.default ++ 
    # Append our nixpkgs-overlays.
    [ "nixpkgs-overlays=/etc/nixos/overlays-compat/" ]
  ;
}

Then, add the following contents to /etc/nixos/overlays-compat/overlays.nix[1]:

self: super:
with super.lib;
let
  # Load the system config and get the `nixpkgs.overlays` option
  overlays = (import <nixpkgs/nixos> { }).config.nixpkgs.overlays;
in
  # Apply all overlays to the input of the current "main" overlay
  foldl' (flip extends) (_: super) overlays self

The /etc/nixos/overlays-compat directory should contain a single overlays.nix file to be understood by the Nix tooling, but the location of this directory can be arbitrary, as long as it is set correctly in the nix.nixPath option.

External Documentation

Examples of overlays

Here are a few example usages of overlays.

Overriding a version

Assume you want the original version of sl, not the fork that nixpkgs ships. First, you have to choose the exact revision you want nix to build. Here we will build revision 923e7d7ebc5c1f009755bdeb789ac25658ccce03. The core of the method is to override the attribute src of the derivation with an updated value. Here we use fetchFromGitHub because sl is hosted on github, but other locations need other functions. To see the original derivation, run nix edit -f "<nixpkgs>" sl. This method will fail if the build system changed or new dependencies are required.

self: super:
{
  sl = super.sl.overrideAttrs (old: {
    src = super.fetchFromGitHub {
      owner = "mtoyoda";
      repo = "sl";
      rev = "923e7d7ebc5c1f009755bdeb789ac25658ccce03";
      # If you don't know the hash, the first time, set:
      # sha256 = "0000000000000000000000000000000000000000000000000000";
      # then nix will fail the build with such an error message:
      # hash mismatch in fixed-output derivation '/nix/store/m1ga09c0z1a6n7rj8ky3s31dpgalsn0n-source':
      # wanted: sha256:0000000000000000000000000000000000000000000000000000
      # got:    sha256:173gxk0ymiw94glyjzjizp8bv8g72gwkjhacigd1an09jshdrjb4
      sha256 = "173gxk0ymiw94glyjzjizp8bv8g72gwkjhacigd1an09jshdrjb4";
    };
  });
}

Adding patches

It is easy to add patches to a nix package:

self: super:
{
  sl = super.sl.overrideAttrs (old: {
    patches = (old.patches or []) ++ [
      (super.fetchpatch {
        url = "https://github.com/charlieLehman/sl/commit/e20abbd7e1ee26af53f34451a8f7ad79b27a4c0a.patch";
        sha256 = "07sx98d422589gxr8wflfpkdd0k44kbagxl3b51i56ky2wfix7rc";
      })
      # alternatively if you have a local patch, 
      /path/to/file.patch
      # or a relative path (relative to the current nix file)
      ./relative.patch
    ];
  });
}

Compilation options

Some packages provide compilation options. Those are not easily disoverable; to find them you need to have a look at the source. For example, with nix edit -f "<nixpkgs>" pass one can see that pass can be compiled with or without dependencies on X11 with the x11Support argument. Here is how you can remove X11 dependencies:

self: super:
{
  pass = super.pass.override { x11Support = false; }; 
}

Overriding a package inside a scope

Some packages are not in the top level of nixpkgs but inside a scope. For example all GNOME packages are in the gnome3 attribute set and Xfce packages inside xfce. These attributes are often scopes and must be overriden specially. Here is an example of patching xfce.xfce4-terminal.

# elements of nixpkgs must be taken from self and super
self: super:
{
  xfce = super.xfce.overrideScope' (
    # elements of pkgs.xfce must be taken from selfx and superx
    selfx: superx: {
      xfce4-terminal = superx.xfce4-terminal.overrideAttrs (
        old: {
          patches = (old.patches or []) ++ [
            (
              # https://bugzilla.xfce.org/show_bug.cgi?id=16682
              super.fetchpatch {
                url = "https://bugzilla.xfce.org/attachment.cgi?id=9709";
                sha256 = "1shxzzvwsybri772lbm17rpd1l0g9y2vc7f7xgqlgkgz72wzr5zp";
              }
            )
          ];
        }
      );
    }
  );
}


Python Packages Overlay

Here is an example of Python packages overlay. The trick is to also override python itself with `packageOverrides`.

Github issue with the snippet below: [[1]]

self: super:
# Within the overlay we use a recursive set, though I think we can use `self` as well.
rec {
  # nix-shell -p python.pkgs.my_stuff
  python = super.python.override {
    # Careful, we're using a different self and super here!
    packageOverrides = self: super: {
      my_stuff = super.buildPythonPackage rec {
        pname = "pyaes";
        version = "1.6.0";
        src = super.fetchPypi {
          inherit pname version;
          sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
        };
      };
    };
  };
  # nix-shell -p pythonPackages.my_stuff
  pythonPackages = python.pkgs;

  # nix-shell -p my_stuff
  my_stuff = pythonPackages.buildPythonPackage rec {
    pname = "pyaes";
    version = "1.6.0";
    src = pythonPackages.fetchPypi {
      inherit pname version;
      sha256 = "0bp9bjqy1n6ij1zb86wz9lqa1dhla8qr1d7w2kxyn7jbj56sbmcw";
    };
  };
}

References

  1. Based on @samueldr's configuration: overlays-compat