CCache: Difference between revisions

VisualWikitext
imported>Milahu
init page Ccache
 
Xz (talk | contribs)
2 edits
Not related to NixOS, read https://ccache.dev/manual/4.8.2.html#_cache_statistics and run ccache -vs
 
(9 intermediate revisions by 6 users not shown)
Line 1: Line 1:
ccache is useful for [[Packaging]] large packages with [[Incremental builds]]
CCache is useful for [[Packaging]] large packages with [[Incremental builds]].


with ccache, recompile time can be reduced from many hours to a few minutes
With CCache, recompile time can be reduced from many hours to a few minutes.


== derivation ccache ==
== NixOS ==
On NixOS, the <code>programs.ccache</code> module can be used to partially enable CCache.


to enable ccache only for one derivation
<syntaxhighlight lang="nix">
programs.ccache.enable = true;
</syntaxhighlight>


add to <code>/etc/nixos/configuration.nix</code>
However, without specifying <code>programs.ccache.packageNames</code> the CCache wrapper is not configured. The wrapper configuration can be added to your Nix overlays.
 
<syntaxhighlight lang="nix">
nixpkgs.overlays = [
  (self: super: {
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="${config.programs.ccache.cacheDir}"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
  })
];
</syntaxhighlight>
 
The CCache directory also needs to be added to the builder sandboxes.
 
<syntaxhighlight lang="nix">
nix.settings.extra-sandbox-paths = [ config.programs.ccache.cacheDir ];
</syntaxhighlight>
 
Run <code>sudo nixos-rebuild switch</code> to enable these options before attempting to use CCache for a derivation.
 
=== Derivation CCache ===
Packages can built with CCache by overriding <code>stdenv</code> in the derivation.
 
<syntaxhighlight lang="nix">
nixpkgs.overlays = [
  (self: super: {
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
  })
];
</syntaxhighlight>
 
Some packages do not use <code>stdenv</code> directly. You may need to plumb it through other dependencies first.
 
Note, that if the package is a top-level package, you may instead add it to the <code>programs.ccache.packageNames</code> list.
 
<syntaxhighlight lang="nix">
programs.ccache.packageNames = [ "ffmpeg" ];
</syntaxhighlight>
 
=== System CCache ===
 
todo
 
=== Monitor CCache ===
 
The NixOS module creates a script that can be used to monitor the CCache directory without sudo.


<pre>
<pre>
          nix.extraOptions = ''
nix-ccache --show-stats
            sandbox = false
          '';
</pre>
</pre>


run <code>sudo nixos-rebuild switch</code>
== Non-NixOS ==


now <code>/etc/nix/nix.conf</code> should have <code>sandbox = false</code>
Create the cache folder:


patch the <code>default.nix</code> file
<pre>
sudo mkdir -m0770 -p /nix/var/cache/ccache


<pre>
# Linux
{ stdenv
sudo chown --reference=/nix/store /nix/var/cache/ccache
, ccacheStdenv
# ...
}:


#stdenv.mkDerivation {
# macOS workaround for chown --reference
ccacheStdenv.mkDerivation {
nix-shell -p coreutils --run 'sudo chown --reference=/nix/store /nix/var/cache/ccache'
</pre>


  preConfigure = ''
Add the path to the derivation sandbox by adding <code>extra-sandbox-paths</code> to <code>nix.conf</code>
    export CCACHE_DIR=/var/cache/ccache
    export CCACHE_UMASK=007
  ''


# ...
<pre>
extra-sandbox-paths = /nix/var/cache/ccache
</pre>
</pre>


add <code>ccacheStdenv</code> to dependencies
Then configure the CCache wrapper script.


run <code>nix-build</code>
<syntaxhighlight lang="nix">
nixpkgs.overlays = [
  (self: super: {
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="/nix/var/cache/ccache"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
  })
];
</syntaxhighlight>


== system ccache ==
=== Derivation CCache ===
Packages can built with CCache by overriding <code>stdenv</code> in the derivation.


todo
<syntaxhighlight lang="nix">
nixpkgs.overlays = [
  (self: super: {
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
  })
];
</syntaxhighlight>
 
Some packages do not use <code>stdenv</code> directly. You may need to plumb it through other dependencies first.


== monitor ccache status ==
=== Monitor CCache status ===


<pre>
<pre>
# watch ccache size
# watch ccache size
sudo watch du -sh /var/cache/ccache
sudo watch du -sh /nix/var/cache/ccache


# watch ccache stats
# watch ccache stats
sudo watch ccache --dir /var/cache/ccache --show-stats --verbose
sudo watch ccache --dir /nix/var/cache/ccache --show-stats
</pre>
 
== Sloppiness ==
 
By default, <code>stdenv</code> inserts <code>-frandom-seed</code> C compiler flag with a value that changes whenever the derivation hash has changed.
Consequently, this behavior completely defeats any usage of <code>ccacheWrapper</code>
To counterpart this behavior, add the following line to the ccache config (typically <code>/var/cache/ccache/ccache.conf</code>):
<pre>
sloppiness = random_seed
</pre>
</pre>
Be warned that this configuration option might affect reproducibility of builds, and could lead to cache poisoning.
See [https://github.com/NixOS/nixpkgs/issues/109033 issue 109033] for more details.
== See also ==
* https://leanprover.github.io/lean4/doc/make/nix.html
* [https://nixos.org/manual/nix/stable/command-ref/conf-file.html sandbox-paths in nix.conf]
[[Category: Development]]
Retrieved from "https://wiki.nixos.org/wiki/CCache"