Binary Cache

Every machine with nix installed can be a configured to be a binary cache for another nix machine - no matter if it is a NixOS, other Linux distro, or MacOS machine.

Setting up a NixOS Machine as Binary Cache for Others

This tutorial assumes:

• NixOS on the serving machine
• You want to serve the nix store via port 80 and already have an nginx service running (See here how to configure nginx [1])
• The store is served with signing activated
• Firewall rules (port 80) are already set up (See firewall settings in NixOS manual[2])
• The hostname "binarycache.mydomain.com" points to the server (see networking.hostName[3])

This tutorial explains the following steps one by one in the following:

1. Generate a private/public keypair for signing packages
   • This step is not mandatory, but we are going to serve signed nix packages
2. Activate nix-serve
   • This is the service that speaks the binary cache protocol via HTTP.
   • It will use the private key to sign the packages
3. Create a virtual hostname in nginx
   • It will redirect the HTTP(s) traffic from port 80 to nix-serve
4. Rebuild the NixOS config and test

1. Generate a private/public keypair

$ nix-store --generate-binary-cache-key binarycache.mydomain.com1 cache-priv-key.pem cache-pub-key.pem
$ sudo mv cache-priv-key.pem /var/cache-priv-key.pem
$ sudo chown nix-store /var/cache-priv-key.pem
$ sudo chmod 600 /var/cache-priv-key.pem

It is important that only nix-serve can access the private key. The location /var/cache-priv-key.pem is just an example.

2. Activate nix-serve

services.nix-serve = {
  enable = true;
  secretKeyFile = "/var/cache-priv-key.pem";
};

nix-serve will by default serve on port 5000. We are not going to open a firewall port for it, because we will let nginx redirect to it.

3. Create a virtual hostname in nginx

services.nginx = {
  enable = true;
  virtualHosts = {
    # ... existing hosts config etc. ...
    "binarycache.mydomain.com" = {
      serverAliases = [ "binarycache" ];
      locations."/".extraConfig = ''
        proxy_pass http://localhost:${toString config.services.nix-serve.port};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      '';
    };
  };
};

Add HTTPS settings to this config if possible. The tutorial will simply continue with unsecure HTTP. (See NixOS manual on how to set up HTTPS with letsencrypt[4])

4. Rebuild the NixOS config and test

sudo nixos-rebuild switch

Check the general availability:

$ curl http://binarycache.mydomain.com/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30

On the binary cache server, build some package:

$ nix-build '<nixpkgs>' -A pkgs.hello
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10

On some other machine, test if nix can realize the same package via the binary cache (the content of the trusted-public-keys is the content of the file cache-pub-key.pem which you should share with users of the binary cache!):

$ nix-store -r /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10 --option substituters http://binarycache.mydomain.com --option trusted-public-keys binarycache.mydomain.com1:dsafdafDFW123fdasfa123124FADSAD
these paths will be fetched (0.00 MiB download, 24.04 MiB unpacked):
  /nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27
  /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
copying path '/nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27' from 'http://binarycache.mydomain.com'...
copying path '/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10' from 'http://binarycache.mydomain.com'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10

It works, great! Now you can configure your nix machines to look up packages always/also on this server. For that, have a look into the Nix manual [5] and the following settings:

• substituters
• extra-substituters
• trusted-public-keys