Binary Cache

From NixOS Wiki
Revision as of 10:45, 17 October 2019 by imported>Aleb

A binary cache builds Nix packages and caches the result for other machines. Any machine with Nix installed can be a binary cache for another one, no matter the operating system.

Setting up a binary cache

This tutorial explains how to setup a server running NixOS as a binary cache for other machines, serving the store on TCP port 80 with signing turned on. It assumes that an nginx service is already running, that port 80 is open,[cf. 1] and that the hostname binarycache.example.com resolves to the server.[cf. 2]

1. Generating a private/public keypair

A keypair is necessary to sign Nix packages.

$ nix-store --generate-binary-cache-key binarycache.example.com cache-priv-key.pem cache-pub-key.pem
# mv cache-priv-key.pem /var/cache-priv-key.pem
# chown nix-store /var/cache-priv-key.pem
# chmod 600 /var/cache-priv-key.pem

The packages can be signed before adding them to the binary cache, or on the fly as they are served. In this tutorial we'll set up nix-serve to sign packages on the fly when it serves them. In this case it is important that only nix-serve can access the private key. The location /var/cache-priv-key.pem is just an example.

2. Activating nix-serve

nix-serve is the service that speaks the binary cache protocol via HTTP.

To start it on NixOS:

services.nix-serve = {
  enable = true;
  secretKeyFile = "/var/cache-priv-key.pem";
};

To start it on a different machine at boot, the simplest is to add to /etc/crontab:

NIX_SECRET_KEY_FILE=/var/cache-priv-key.pem
@reboot /home/USER/.nix-profile/bin/nix-serve --listen :5000 --error-log /var/log/nix-serve.log --pid /var/run/nix-serve.pid --user USER --daemonize

nix-serve will by default serve on port 5000. We are not going to open a firewall port for it, because we will let nginx redirect to it.

3. Creating a virtual hostname in nginx

We redirect the HTTP(s) traffic from port 80 to nix-serve. As nix-serve is capable of serving only on IPv4, redirecting is also useful to make the binary cache available on IPv6.

services.nginx = {
  enable = true;
  virtualHosts = {
    # ... existing hosts config etc. ...
    "binarycache.example.com" = {
      serverAliases = [ "binarycache" ];
      locations."/".extraConfig = ''
        proxy_pass http://localhost:${toString config.services.nix-serve.port};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      '';
    };
  };
};

Add HTTPS settings to this config if possible.[cf. 3] This tutorial will simply continue with insecure HTTP.

To set up Nginx on a non-NixOS machine, create for example /etc/nginx/sites-enabled/nix-serve.conf:

server {
    listen      80 default_server;
    listen      [::]:80 default_server;
			
    location / {
        proxy_pass  http://127.0.0.1:5000;
    }
}

4. Rebuilding and testing

# nixos-rebuild switch

Check the general availability:

$ curl http://binarycache.example.com/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30

On the binary cache server, build some package:

$ nix-build '<nixpkgs>' -A pkgs.hello
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10

Next, with the public key that was generated to cache-pub-key.pem, setup another machine to use the binary cache, and see if Nix successfully fetches the cached package.

Using a binary cache

To configure Nix to use a certain binary cache, refer to the Nix manual.[cf. 4] Add the binary cache as substituter (see the options substituters and extra-substituters) and the public key to the trusted keys (see trusted-public-keys).

$ nix-store -r /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10 --option substituters http://binarycache.example.com --option trusted-public-keys binarycache.example.com:dsafdafDFW123fdasfa123124FADSAD
these paths will be fetched (0.00 MiB download, 24.04 MiB unpacked):
  /nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27
  /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
copying path '/nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27' from 'http://binarycache.example.com'...
copying path '/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10' from 'http://binarycache.example.com'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10

See also