Binary Cache

A binary cache builds Nix packages and caches the result for other machines. Any machine with Nix installed can be a binary cache for another one, no matter the operating system.

Setting up a binary cache

This tutorial explains how to setup a machine as a binary cache for other machines, serving the nix store on TCP port 80 with signing turned on. It assumes that an nginx service is already running, that port 80 is open, and that the hostname binarycache.example.com resolves to the server.

1. Generating a private/public keypair

A keypair is necessary to sign Nix packages. Replace binarycache.example.com with your domain.

cd /var
nix-store --generate-binary-cache-key binarycache.example.com cache-priv-key.pem cache-pub-key.pem
chown nix-serve cache-priv-key.pem
chmod 600 cache-priv-key.pem
cat cache-pub-key.pem

The packages can be signed before adding them to the binary cache, or on the fly as they are served. In this tutorial we'll set up nix-serve to sign packages on the fly when it serves them. In this case it is important that only nix-serve can access the private key. The location /var/cache-priv-key.pem is just an example.

2. Activating nix-serve

nix-serve is the service that speaks the binary cache protocol via HTTP.

To start it on NixOS:

services.nix-serve = {
  enable = true;
  secretKeyFile = "/var/cache-priv-key.pem";
};

To start it on a non-NixOS machine at boot, add to /etc/crontab:

NIX_SECRET_KEY_FILE=/var/cache-priv-key.pem
@reboot /home/USER/.nix-profile/bin/nix-serve --listen :5000 --error-log /var/log/nix-serve.log --pid /var/run/nix-serve.pid --user USER --daemonize

nix-serve will by default serve on port 5000. We are not going to open a firewall port for it, because we will let nginx redirect to it.

3. Creating a virtual hostname in nginx

We redirect the HTTP(s) traffic from port 80 to nix-serve. As nix-serve is capable of serving only on IPv4, redirecting is also useful to make the binary cache available on IPv6.

services.nginx = {
  enable = true;
  virtualHosts = {
    # ... existing hosts config etc. ...
    "binarycache.example.com" = {
      serverAliases = [ "binarycache" ];
      locations."/".extraConfig = ''
        proxy_pass http://localhost:${toString config.services.nix-serve.port};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      '';
    };
  };
};

Add HTTPS settings to this config if possible. This tutorial will simply continue with insecure HTTP.

To set up Nginx on a non-NixOS machine, create for example /etc/nginx/sites-enabled/nix-serve.conf:

server {
    listen      80 default_server;
    listen      [::]:80 default_server;
			
    location / {
        proxy_pass  http://127.0.0.1:5000;
    }
}

4. Testing

To apply the previous settings to your NixOS machine, run:

# nixos-rebuild switch

Check the general availability:

$ curl http://binarycache.example.com/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30

On the binary cache server, build some package:

$ nix-build '<nixpkgs>' -A pkgs.hello
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10

To verify the signing on the fly, make sure the following request contains a Sig: line:

$ curl http://binarycache.example.com/gdh8165b7rg4y53v64chjys7mbbw89f9.narinfo
StorePath: /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
URL: nar/gdh8165b7rg4y53v64chjys7mbbw89f9.nar
Compression: none
NarHash: sha256:0mkfk4iad66xkld3b7x34n9kxri9lrpkgk8m17p97alacx54h5c7
NarSize: 205920
References: 6yaj6n8l925xxfbcd65gzqx3dz7idrnn-glibc-2.27 gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
Deriver: r6h5b3wy0kwx38rn6s6qmmfq0svcnf86-hello-2.10.drv
Sig: binarycache.example.com:EmAANryZ1FFHGmz5P+HXLSDbc0KckkBEAkHsht7gEIOUXZk9yhhZSBV+eSX9Kj+db/b36qmYmffgiOZbAe21Ag==

Next, with the public key that was generated to cache-pub-key.pem, setup a client machine to use the binary cache, and see if Nix successfully fetches the cached package.

Using a binary cache

To configure Nix to use a certain binary cache, refer to the Nix manual. Add the binary cache as substituter (see the options substituters and extra-substituters) and the public key to the trusted keys (see trusted-public-keys).

Permanent use of binary cache:

# /etc/nixos/configuration.nix

  nix = {
    binaryCaches = [
      "http://binarycache.example.com"
      "https://nix-community.cachix.org"
      "https://cache.nixos.org/"
    ];
    binaryCachePublicKeys = [
      "binarycache.example.com-1:dsafdafDFW123fdasfa123124FADSAD"
      "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
    ];
  };

Temporary use of binary cache:

$ nix-store -r /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10 --option substituters http://binarycache.example.com --option trusted-public-keys binarycache.example.com:dsafdafDFW123fdasfa123124FADSAD
these paths will be fetched (0.00 MiB download, 24.04 MiB unpacked):
  /nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27
  /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
copying path '/nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27' from 'http://binarycache.example.com'...
copying path '/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10' from 'http://binarycache.example.com'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10

Populating a binary cache

As the cache is served from the nix store of the machine serving the binary cache, one option is to build the packages directly on that machine.

Another option is to build the packages on a separate machine and push them only when all the checks pass using nix copy:

$ nix copy --to ssh://binarycache.example.com PACKAGE

For details see the Sharing Packages Between Machines in the Nix manual.

Hosted binary cache

https://cachix.org provides hosted binary caches starting with a free plan for public caches.

How to check if content is on a binary cache

We can use curl to check if a binary cache contains a given derivation. curl https://cache/store_hash.narinfo

$ curl https://fzakaria.cachix.org/949dxjmz632id67hjic04x6f3ljldzxh.narinfo

StorePath: /nix/store/949dxjmz632id67hjic04x6f3ljldzxh-mvn2nix-0.1
URL: nar/4026897ef85219b5b697c1c4ef30d50275423857cb7a81e138c4b1025f550935.nar.xz
Compression: xz
FileHash: sha256:4026897ef85219b5b697c1c4ef30d50275423857cb7a81e138c4b1025f550935
FileSize: 24392
NarHash: sha256:0kk3d8rk82ynqwg8isk83hvq8vszh4354fqg4hhaz40kd49rmm9n
NarSize: 29208
References: 2hhmmj0vbb5d181nfx2mx3p7k54q44ij-apache-maven-3.6.3 6737cq9nvp4k5r70qcgf61004r0l2g3v-bash-4.4-p23 949dxjmz632id67hjic04x6f3ljldzxh-mvn2nix-0.1 hrlxlk768vy5rgl6hc4xiba6gxg6s0yz-mvn2nix-0.1-dependencies qybd7j6v7kb7yhizc7gklgg3lyrxf38y-openjdk-headless-11.0.8+10
Deriver: 585w6p8rclbvz97fwgixvfgnh5493ia2-mvn2nix-0.1.drv
Sig: fzakaria.cachix.org-1:MkOrZCa9qdxHFdE2mtFRsbEzmLUgWGgSrqD3advKfdHLW+SKxj/V2n6+4a/qy6dhCoR+gWQfGzda/jNkER10CQ==

Or use nix path-info:

$ nix path-info -r /nix/store/sb7nbfcc1ca6j0d0v18f7qzwlsyvi8fz-ocaml-4.10.0 --store https://cache.nixos.org/
[0.0 MiB DL] querying libunistring-0.9.10 on https://cache.nixos.org/nix/store/0gc9dr71ldp79cla2qbl3kwdd4ig46pi-linux-headers-5.5
/nix/store/2jysm3dfsgby5sw5jgj43qjrb5v79ms9-bash-4.4-p23
/nix/store/4wy9j24psf9ny4di3anjs7yk2fvfb0gq-glibc-2.31-dev
/nix/store/4z79ipgxqn80ns7mpax25zmb77i3ndfw-gawk-5.1.0
/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31
/nix/store/czc3c1apx55s37qx4vadqhn3fhikchxi-libunistring-0.9.10
/nix/store/fgn3sih5vi7543jcw389a7qqax8nwkhz-glibc-2.31-bin
/nix/store/sb7nbfcc1ca6j0d0v18f7qzwlsyvi8fz-ocaml-4.10.0
/nix/store/xim9l8hym4iga6d4azam4m0k0p1nw2rm-libidn2-2.3.0

See also

• How to use binary cache in NixOS