Official NixOS Wiki

Binary Cache: Difference between revisions

← Older edit
Axka (talk | contribs)
92 edits
VisualWikitext
imported>Mth
No edit summary
Axka (talk | contribs)
92 edits
 
(38 intermediate revisions by 24 users not shown)
Line 1: Line 1:
{{Expansion|What is the format of a binary cache? How does it differ from a local/remote Nix store? See [https://github.com/NixOS/nix/pull/6870 `NixOS/nix` PR #6870]. (Maybe even splitting it into a guide and a reference?)}}
A binary cache builds Nix packages and caches the result for other machines. Any machine with Nix installed can be a binary cache for another one, no matter the operating system.
A binary cache builds Nix packages and caches the result for other machines. Any machine with Nix installed can be a binary cache for another one, no matter the operating system.


== Setting up a binary cache ==
== Setting up a binary cache with attic and caddy ==
Here's a snippet enabling [https://github.com/zhaofengli/attic Attic] and [https://caddyserver.com/ Caddy].
 
Please refer to the [https://docs.attic.rs/ Attic documentation] to set it up correctly. The goal here is to show how those two services can be used together to provide a solid solution.<syntaxhighlight lang="nix" line="1">
{
  networking.firewall = {
    allowedTCPPorts = [ 8080 ];
  };
 
  services = {
    atticd = {
      enable = true;
      settings = {
        listen = "127.0.0.1:8081";
      };
      # Path to an EnvironmentFile containing required environment variables:
      # ATTIC_SERVER_TOKEN_RS256_SECRET_BASE64: The base64-encoded RSA PEM PKCS1 of the RS256 JWT secret. Generate it with openssl genrsa -traditional 4096 | base64 -w0.
      environmentFile = "/root/.attic-env-file";
    };
 
    # Inspired from:
    # 1. https://github.com/phanirithvij/system/blob/main/nixos/applications/nix/selfhosted/proxy-cache.nix
    # 2. https://github.com/rnl-dei/nixrnl/blob/master/profiles/proxy-cache.nix
    caddy = {
      enable = true;
 
      package = pkgs.caddy.withPlugins {
        plugins = [ "github.com/caddyserver/cache-handler@v0.16.0" ];
        hash = "sha256-CecAx6KelOHEDiOKDTKLlDcnWtRNnDzBw1AzgN5JaFw=";
      };
 
      globalConfig = ''
        order cache before rewrite
        cache {
          # Global default cache duration (if not overridden below)
          ttl 1h
          log_level debug
        }
      '';
 
      virtualHosts.":8080" = {
        extraConfig = ''
          log {
            format console
          }
 
          # Nix cache info endpoint
          @nix_cache_info path /nix-cache-info
          handle @nix_cache_info {
            header Cache-Control "public, max-age=300"
 
            # 2. Tell Caddy's internal cache to hold this for 5 minutes
            cache {
              ttl 300s
            }
 
            reverse_proxy https://cache.nixos.org {
              header_up Host cache.nixos.org
            }
          }
 
          # NAR files (the actual packages)
          @nar path /nar/*
          handle @nar {
            header Cache-Control "public, max-age=31536000, immutable"
 
            # Cache the actual nar packages for a year
            cache {
              ttl 8760h
            }
 
            reverse_proxy https://cache.nixos.org {
              header_up Host cache.nixos.org
            }
          }
 
          # Narinfo files (metadata about packages)
          @narinfo path_regexp ^/[^/]+\.narinfo$
          handle @narinfo {
            header Cache-Control "public, max-age=86400"
 
            # Narinfo can change, so cache them locally for 24 hours
            cache {
              ttl 24h
            }
 
            reverse_proxy https://cache.nixos.org {
              header_up Host cache.nixos.org
            }
          }
 
          # Fallback for other requests
          handle {
            # We omit the `cache` directive here so Caddy doesn't interfere
            # with Attic's API operations or package pushing (PUT/POST requests).
            reverse_proxy 127.0.0.1:8081
          }
        '';
      };
    };
  };
};


This tutorial explains how to setup a server running NixOS as a binary cache for other machines, serving the store on TCP port 80 with signing turned on. It assumes that an {{ic|[[nginx]]}} service is already running, that port 80 is open,<ref group="cf."> {{manual:nixos|sec=#sec-firewall|chapter=11.5. Firewall}}</ref> and that the hostname {{ic|binarycache.example.com}} resolves to the server.<ref group="cf.">{{nixos:option|networking.hostName}}</ref>
</syntaxhighlight>


=== 1. Generating a private/public keypair ===
== Setting up a binary cache with nix-serve and nginx ==


A keypair is necessary to sign Nix packages.  
This tutorial explains how to setup a machine as a binary cache for other machines, serving the nix store on TCP port 80 with signing turned on. It assumes that an {{ic|[[nginx]]}} service is already running, that port 80 is open,<ref group="cf."> {{manual:nixos|sec=#sec-firewall|chapter=11.5. Firewall}}</ref> and that the hostname {{ic|binarycache.example.com}} resolves to the server.<ref group="cf.">{{nixos:option|networking.hostName}}</ref>


{{bc|
=== 1. Generating a signing key ===
$ nix-store --generate-binary-cache-key binarycache.example.com cache-priv-key.pem cache-pub-key.pem
# mv cache-priv-key.pem /var/cache-priv-key.pem
# chown nix-store /var/cache-priv-key.pem
# chmod 600 /var/cache-priv-key.pem
}}


It is important that only {{ic|nix-serve}} can access the private key.
Follow the generation process at [[Signing store paths#Signing Key]], replacing the example hostname with your own. In this tutorial, {{ic|nix-serve}} signs packages on the fly when it serves them so any [[Nix (package manager)|Nix]] configuration isn't required.
The location {{ic|/var/cache-priv-key.pem}} is just an example.


=== 2. Activating {{ic|nix-serve}} ===
=== 2. Activating {{ic|nix-serve}} ===


{{ic|nix-serve}} is the service that speaks the binary cache protocol via HTTP.
{{ic|nix-serve}} is the service that serves the binary cache protocol via HTTP.


<syntaxhighlight lang="nix">
To start it on NixOS:
services.nix-serve = {
<syntaxhighlight lang="nix">services.nix-serve = {
   enable = true;
   enable = true;
   secretKeyFile = "/var/cache-priv-key.pem";
 
};
  # Note: You don't need to give nix-serve ownership of the file because systemd reads it.
   secretKeyFile = "/var/secrets/nix-cache-priv-key";
};</syntaxhighlight>
 
To start it on a non-NixOS machine at boot, add to {{ic|/etc/crontab}}:
<syntaxhighlight>
NIX_SECRET_KEY_FILE=/var/secrets/nix-cache-priv-key
@reboot /home/USER/.nix-profile/bin/nix-serve --listen :5000 --error-log /var/log/nix-serve.log --pid /var/run/nix-serve.pid --user USER --daemonize
</syntaxhighlight>
</syntaxhighlight>


Line 34: Line 138:
=== 3. Creating a virtual hostname in {{ic|nginx}} ===
=== 3. Creating a virtual hostname in {{ic|nginx}} ===


We redirect the HTTP(s) traffic from port 80 to {{ic|nix-serve}}.
We redirect the HTTP(s) traffic from port 80 to {{ic|nix-serve}}. As {{ic|nix-serve}} is capable of serving only on IPv4, redirecting is also useful to make the binary cache available on IPv6.


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
services.nginx = {
services.nginx = {
   enable = true;
   enable = true;
  recommendedProxySettings = true;
   virtualHosts = {
   virtualHosts = {
     # ... existing hosts config etc. ...
     # ... existing hosts config etc. ...
     "binarycache.example.com" = {
     "binarycache.example.com" = {
      serverAliases = [ "binarycache" ];
       locations."/".proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}";
       locations."/".extraConfig = ''
        proxy_pass http://localhost:${toString config.services.nix-serve.port};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      '';
     };
     };
   };
   };
Line 56: Line 155:
Add HTTPS settings to this config if possible.<ref group="cf.">{{manual:nixos|sec=#module-security-acme-nginx|chapter=26.3. Using ACME certificates in Nginx}}</ref> This tutorial will simply continue with insecure HTTP.
Add HTTPS settings to this config if possible.<ref group="cf.">{{manual:nixos|sec=#module-security-acme-nginx|chapter=26.3. Using ACME certificates in Nginx}}</ref> This tutorial will simply continue with insecure HTTP.


=== 4. Rebuilding and testing ===
To set up Nginx on a non-NixOS machine, create for example {{ic|/etc/nginx/sites-enabled/nix-serve.conf}}:
 
<syntaxhighlight lang="nginx">
server {
    listen      80 default_server;
    listen      [::]:80 default_server;
    location / {
        proxy_pass  http://127.0.0.1:5000;
    }
}
</syntaxhighlight>


{{bc|# nixos-rebuild switch}}
=== 4. Testing ===


Check the general availability:
To apply the previous settings to your NixOS machine, run:<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="bash">
# nixos-rebuild switch
</syntaxhighlight>Check the general availability:
<syntaxhighlight lang="shell-session">
$ curl http://binarycache.example.com/nix-cache-info
$ curl http://binarycache.example.com/nix-cache-info
StoreDir: /nix/store
StoreDir: /nix/store
Line 69: Line 181:


On the binary cache server, build some package:
On the binary cache server, build some package:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="shell-session">
$ nix-build '<nixpkgs>' -A pkgs.hello
$ nix-build '<nixpkgs>' -A pkgs.hello
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
</syntaxhighlight>
</syntaxhighlight>


Next, with the public key that was generated to {{ic|cache-pub-key.pem}}, setup another machine to use the binary cache, and see if Nix successfully fetches the cached package.
To verify the signing on the fly, make sure the following request contains a {{ic|Sig:}} line:
<syntaxhighlight lang="shell-session">
$ curl http://binarycache.example.com/gdh8165b7rg4y53v64chjys7mbbw89f9.narinfo
StorePath: /nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
URL: nar/gdh8165b7rg4y53v64chjys7mbbw89f9.nar
Compression: none
NarHash: sha256:0mkfk4iad66xkld3b7x34n9kxri9lrpkgk8m17p97alacx54h5c7
NarSize: 205920
References: 6yaj6n8l925xxfbcd65gzqx3dz7idrnn-glibc-2.27 gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
Deriver: r6h5b3wy0kwx38rn6s6qmmfq0svcnf86-hello-2.10.drv
Sig: binarycache.example.com:EmAANryZ1FFHGmz5P+HXLSDbc0KckkBEAkHsht7gEIOUXZk9yhhZSBV+eSX9Kj+db/b36qmYmffgiOZbAe21Ag==
</syntaxhighlight>
 
Next, with the public key that was generated to {{ic|cache-pub-key.pem}}, setup a client machine to use the binary cache, and see if Nix successfully fetches the cached package.


== Using a binary cache ==
== Using a binary cache ==


To configure Nix to use a certain binary cache, refer to the Nix manual.<ref group="cf.">[https://nixos.org/nix/manual/#ch-files Nix Manual, 21. Files]</ref> Add the binary cache as substituter (see the options {{ic|substituters}} and {{ic|extra-substituters}}) and the public key to the trusted keys (see {{ic|trusted-public-keys}}).
To configure Nix to use a certain binary cache, refer to the Nix manual.<ref group="cf.">[https://nixos.org/nix/manual/#ch-files Nix Manual, 21. Files]</ref> Add the binary cache as substituter (see the option {{ic|substituters}}) and the public key to the trusted keys (see {{ic|trusted-public-keys}}).
{{Warning|When adding a third-party binary cache you now trust all packages being served from that cache. Make sure this is a conscious decision. Trusting arbitrary caches can open you up to suppply chain attacks.
For more context: https://discourse.nixos.org/t/garnix-blog-stop-trusting-nix-caches/70177 (if source unavailable, https://web.archive.org/web/20251001172145/https://garnix.io/blog/stop-trusting-nix-caches)}}{{tip|If you are facing problems with derivations not being in a cache, try switching to a release version. Most caches will have many derivations for a specific release.}}
 
Permanent use of binary cache:
 
<syntaxhighlight lang="nix">
# /etc/nixos/configuration.nix
 
  nix = {
    settings = {
      substituters = [
        "http://binarycache.example.com"
        "https://nix-community.cachix.org"
        "https://cache.nixos.org/"
      ];
      trusted-public-keys = [
        "binarycache.example.com-1:dsafdafDFW123fdasfa123124FADSAD"
        "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
      ];
    };
  };
</syntaxhighlight>As described on [https://search.nixos.org/options?show=nix.settings.substituters&type=packages&query=substituters search.nixos.org] by default <nowiki>https://cache.nixos.org/</nowiki> is added to the substituters. You may need to use lib.mkForce to override this and ensure your substituter is the primary choice.<syntaxhighlight lang="nix">
# /etc/nixos/configuration.nix
 
 
{ config, lib, pkgs, ... }:
 
{
  nix = {
    settings = {
      substituters = lib.mkForce [
        "http://binarycache.example.com"
      ];
      trusted-public-keys = [
        "binarycache.example.com-1:dsafdafDFW123fdasfa123124FADSAD"
      ];
    };
  };
}
</syntaxhighlight>{{Warning|Keys that are entered incorrectly or are otherwise invalid, aside from preventing you from benefiting from the cached derivations, may also prevent you from rebuilding your system. This is most likely to occur after garbage collection (e.g., via <code>nix-collect-garbage -d</code>). Consult [https://github.com/NixOS/nix/issues/8271 NixOS/nix#8271] for additional details and a workaround.}}
 
Temporary use of binary cache:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 90: Line 257:
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
/nix/store/gdh8165b7rg4y53v64chjys7mbbw89f9-hello-2.10
</syntaxhighlight>
</syntaxhighlight>
=== Using a binary cache on non-NixOS installations ===
To use a binary cache with a Nix that has been installed on an operating system other than NixOS (e.g. Ubuntu or macOS)  {{ic|/etc/nix/nix.conf}} will need to be edited manually. This can be done by adding something similar to the following lines to {{ic|/etc/nix/nix.conf}}:
<pre>
trusted-public-keys = nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters = https://nix-community.cachix.org https://cache.nixos.org
trusted-users = root @wheel
</pre>
Note that not all of that information is needed, see the manual for the respective options ([https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-trusted-public-keys trusted-public-keys], [https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-trusted-substituters trusted-substituters], [https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-trusted-users trusted-users]).
With the {{ic|trusted-*}} options set correctly, a user can benefit permanently from a substituter by add the following to their {{ic|~/.config/nix/nix.conf}}
<pre>
substituters = https://nix-community.cachix.org https://cache.nixos.org
</pre>
or temporarily as explained above.
=== Binary cache hint in Flakes ===
You can place a hint to your binary cache in your Flake so when someone builds an output of your Flake, the nix command will ask interactively to trust the specified binary cache.
<syntaxHighlight lang="nix">
{
  nixConfig = {
    extra-substituters = [
      "https://colmena.cachix.org"
    ];
    extra-trusted-public-keys = [
      "colmena.cachix.org-1:7BzpDnjjH8ki2CT3f6GdOk7QAzPOl+1t3LvTLXqYcSg="
    ];
  };
  outputs = { ... }: {
    ...
  };
}
</syntaxHighlight>
=== Binary cache priority ===
Each binary cache has a priority field. A lower number indicates a higher priority.<syntaxhighlight lang="shell-session">
$curl https://cache.nixos.org/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
</syntaxhighlight>You may want to override this value by appending <code>?priority=n</code> at the end of the cache url.<syntaxhighlight lang="nix">
substituters = https://nix-community.cachix.org?priority=1 https://cache.nixos.org?priority=2
</syntaxhighlight>
== Populating a binary cache ==
As the cache is served from the nix store of the machine serving the binary cache, one option is to build the packages directly on that machine.
Another option is to build the packages on a separate machine and push them only when all the checks pass using {{ic|nix copy}}:
<syntaxhighlight lang="bash">
$ nix copy --to ssh://binarycache.example.com PACKAGE
</syntaxhighlight>
For details see the [https://nixos.org/manual/nix/stable/package-management/sharing-packages.html Sharing Packages Between Machines] in the Nix manual.
== Signing Existing Packages ==
It is also possible to sign all the packages that already exist in the nix store of the machine serving the binary cache to make them immediately available.
<code>$ nix store sign --extra-experimental-features nix-command --all --key-file /var/cache-priv-key.pem</code>
Note : As of NixOS 24.11 {{ic|--extra-experimental-features nix-command}} is required for {{ic|store sign}} if this is not in your configuration.nix.
== Hosted binary cache ==
https://cachix.org provides hosted binary caches starting with a free plan for public caches.
== How to check if content is on a binary cache ==
We can use '''curl''' to check if a binary cache contains a given derivation.
''curl https://cache/store_hash.narinfo''
<syntaxhighlight lang="bash">
$ curl https://fzakaria.cachix.org/949dxjmz632id67hjic04x6f3ljldzxh.narinfo
StorePath: /nix/store/949dxjmz632id67hjic04x6f3ljldzxh-mvn2nix-0.1
URL: nar/4026897ef85219b5b697c1c4ef30d50275423857cb7a81e138c4b1025f550935.nar.xz
Compression: xz
FileHash: sha256:4026897ef85219b5b697c1c4ef30d50275423857cb7a81e138c4b1025f550935
FileSize: 24392
NarHash: sha256:0kk3d8rk82ynqwg8isk83hvq8vszh4354fqg4hhaz40kd49rmm9n
NarSize: 29208
References: 2hhmmj0vbb5d181nfx2mx3p7k54q44ij-apache-maven-3.6.3 6737cq9nvp4k5r70qcgf61004r0l2g3v-bash-4.4-p23 949dxjmz632id67hjic04x6f3ljldzxh-mvn2nix-0.1 hrlxlk768vy5rgl6hc4xiba6gxg6s0yz-mvn2nix-0.1-dependencies qybd7j6v7kb7yhizc7gklgg3lyrxf38y-openjdk-headless-11.0.8+10
Deriver: 585w6p8rclbvz97fwgixvfgnh5493ia2-mvn2nix-0.1.drv
Sig: fzakaria.cachix.org-1:MkOrZCa9qdxHFdE2mtFRsbEzmLUgWGgSrqD3advKfdHLW+SKxj/V2n6+4a/qy6dhCoR+gWQfGzda/jNkER10CQ==
</syntaxhighlight>
Or use ''nix path-info'':
<syntaxhighlight lang="bash">
$ nix path-info -r /nix/store/sb7nbfcc1ca6j0d0v18f7qzwlsyvi8fz-ocaml-4.10.0 --store https://cache.nixos.org/
[0.0 MiB DL] querying libunistring-0.9.10 on https://cache.nixos.org/nix/store/0gc9dr71ldp79cla2qbl3kwdd4ig46pi-linux-headers-5.5
/nix/store/2jysm3dfsgby5sw5jgj43qjrb5v79ms9-bash-4.4-p23
/nix/store/4wy9j24psf9ny4di3anjs7yk2fvfb0gq-glibc-2.31-dev
/nix/store/4z79ipgxqn80ns7mpax25zmb77i3ndfw-gawk-5.1.0
/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31
/nix/store/czc3c1apx55s37qx4vadqhn3fhikchxi-libunistring-0.9.10
/nix/store/fgn3sih5vi7543jcw389a7qqax8nwkhz-glibc-2.31-bin
/nix/store/sb7nbfcc1ca6j0d0v18f7qzwlsyvi8fz-ocaml-4.10.0
/nix/store/xim9l8hym4iga6d4azam4m0k0p1nw2rm-libidn2-2.3.0
</syntaxhighlight>
Example: Fetch metadata of <code>bash</code>
<pre>
curl https://cache.nixos.org/$(readlink -f $(which bash) | cut -c12-43).narinfo
</pre>
== Command Line Options ==
It is also possible to pass {{ic|substituters}} and {{ic|trusted-public-keys}} on the command line if they are not in {{ic|configuration.nix}} or you want to use a particular binary cache server.
$ nix-build --option substituters "<nowiki>http://binarycache.example.com</nowiki>" --option trusted-public-keys "binarycache.example.com-1:dsafdafDFW123fdasfa123124FADSAD" '<nixpkgs>' -A pkgs.PACKAGE
$ nixos-rebuild --option substituters "<nowiki>http://binarycache.example.com</nowiki>" --option trusted-public-keys "binarycache.example.com-1:dsafdafDFW123fdasfa123124FADSAD" switch
To do an offline install (providing your binary cache contains all the packages required);
$ nixos-install --option substituters "<nowiki>http://binarycache.example.com</nowiki>" --option trusted-public-keys "binarycache.example.com-1:dsafdafDFW123fdasfa123124FADSAD"


== See also ==
== See also ==
<references group="cf."/>


<references group="cf."/>
* [https://discourse.nixos.org/t/how-to-use-binary-cache-in-nixos/5202/4 How to use binary cache in NixOS]
 
* [https://github.com/zhaofengli/attic attic: Multi-tenant Nix Binary Cache]
 
* [https://discourse.nixos.org/t/nix-binary-cache-for-macos-nix-darwin-with-attic/40118 Nix Binary Cache for MacOS/Nix-Darwin with Attic]
Retrieved from "https://wiki.nixos.org/wiki/Binary_Cache"