CCache: Difference between revisions

From NixOS Wiki
imported>Atry
mNo edit summary
imported>Terinjokes
rewrote to use NixOS module and to cleanup non-nixos instructions
Line 3: Line 3:
With CCache, recompile time can be reduced from many hours to a few minutes.
With CCache, recompile time can be reduced from many hours to a few minutes.


== Add CCache to sandbox ==
== NixOS ==
On NixOS, the <code>programs.ccache</code> module can be used to partially enable CCache.


Add to <code>/etc/nixos/configuration.nix</code>
<syntaxhighlight lang="nix">
programs.ccache.enable = true;
</syntaxhighlight>
 
However, without specifying <code>programs.ccache.packageNames</code> the CCache wrapper is not configured. The wrapper configuration can be added to your Nix overlays.


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
nix.extraOptions = ''
nixpkgs.overlays = [
  extra-sandbox-paths = /nix/var/cache/ccache
  (self: super: {
'';
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="${config.programs.ccache.cacheDir}"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
  })
];
</syntaxhighlight>
</syntaxhighlight>


TODO: use <code>nix.sandboxPaths</code>? [https://github.com/NixOS/nixpkgs/issues/153343]
The CCache directory also needs to be added to the builder sandboxes.


Run <code>sudo nixos-rebuild switch</code>
<syntaxhighlight lang="nix">
nix.settings.extra-sandbox-paths = [ config.programs.ccache.cacheDir ];
</syntaxhighlight>


Now <code>/etc/nix/nix.conf</code> should have <code>sandbox-paths = /nix/var/cache/ccache</code>
Run <code>sudo nixos-rebuild switch</code> to enable these options before attempting to use CCache for a derivation.


Ceate the cache folder
=== Derivation CCache ===
Packages can built with CCache by overriding <code>stdenv</code> in the derivation.
 
<syntaxhighlight lang="nix">
nixpkgs.overlays = [
  (self: super: {
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
  })
];
</syntaxhighlight>
 
Note, that if the package is a top-level package, you can also add it to the <code>programs.ccache.packageNames</code> list.
 
<syntaxhighlight lang="nix">
programs.ccache.packageNames = [ "ffmpeg" ];
</syntaxhighlight>
 
Some packages do not use <code>stdenv</code> directly. You may need to plumb it through other dependencies first.
 
=== System CCache ===
 
todo
 
=== Monitor CCache ===
 
The NixOS module creates a script that can be used to monitor the CCache directory without sudo.
 
<pre>
nix-ccache --show-stats
</pre>
 
== Non-NixOS ==
 
Ceate the cache folder:


<pre>
<pre>
Line 31: Line 95:
</pre>
</pre>


== Derivation CCache ==
Add the path to the derivation sandbox by adding <code>extra-sandbox-paths</code> to <code>nix.conf</code>
 
<pre>
extra-sandbox-paths = /nix/var/cache/ccache
</pre>


To enable CCache only for one derivation, patch the <code>default.nix</code> file
Then configure the CCache wrapper script.


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ stdenv
nixpkgs.overlays = [
, ccacheStdenv
  (self: super: {
# ...
    ccacheWrapper = super.ccacheWrapper.override {
}:
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="/nix/var/cache/ccache"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
  })
];
</syntaxhighlight>


#stdenv.mkDerivation {
=== Derivation CCache ===
ccacheStdenv.mkDerivation {
Packages can built with CCache by overriding <code>stdenv</code> in the derivation.


  preConfigure = ''
<syntaxhighlight lang="nix">
    export CCACHE_DIR=/nix/var/cache/ccache
nixpkgs.overlays = [
    export CCACHE_UMASK=007
   (self: super: {
   '';
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
 
  })
# ...
];
</syntaxhighlight>
</syntaxhighlight>


Add <code>ccacheStdenv</code> to dependencies.
Some packages do not use <code>stdenv</code> directly. You may need to plumb it through other dependencies first.
 
Run <code>nix-build</code>.
 
== System CCache ==
 
todo


== Monitor CCache status ==
=== Monitor CCache status ===


<pre>
<pre>
Line 67: Line 153:


# watch ccache stats
# watch ccache stats
sudo watch ccache --dir /nix/var/cache/ccache --show-stats --verbose
sudo watch ccache --dir /nix/var/cache/ccache --show-stats
</pre>
</pre>



Revision as of 14:26, 18 January 2023

CCache is useful for Packaging large packages with Incremental builds.

With CCache, recompile time can be reduced from many hours to a few minutes.

NixOS

On NixOS, the programs.ccache module can be used to partially enable CCache.

programs.ccache.enable = true;

However, without specifying programs.ccache.packageNames the CCache wrapper is not configured. The wrapper configuration can be added to your Nix overlays.

nixpkgs.overlays = [
  (self: super: {
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="${config.programs.ccache.cacheDir}"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
  })
];

The CCache directory also needs to be added to the builder sandboxes.

nix.settings.extra-sandbox-paths = [ config.programs.ccache.cacheDir ];

Run sudo nixos-rebuild switch to enable these options before attempting to use CCache for a derivation.

Derivation CCache

Packages can built with CCache by overriding stdenv in the derivation.

nixpkgs.overlays = [
  (self: super: {
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
  })
];

Note, that if the package is a top-level package, you can also add it to the programs.ccache.packageNames list.

programs.ccache.packageNames = [ "ffmpeg" ];

Some packages do not use stdenv directly. You may need to plumb it through other dependencies first.

System CCache

todo

Monitor CCache

The NixOS module creates a script that can be used to monitor the CCache directory without sudo.

nix-ccache --show-stats

Non-NixOS

Ceate the cache folder:

sudo mkdir -m0770 -p /nix/var/cache/ccache

# Linux
sudo chown --reference=/nix/store /nix/var/cache/ccache

# macOS workaround for chown --reference
nix-shell -p coreutils --run 'sudo chown --reference=/nix/store /nix/var/cache/ccache'

Add the path to the derivation sandbox by adding extra-sandbox-paths to nix.conf

extra-sandbox-paths = /nix/var/cache/ccache

Then configure the CCache wrapper script.

nixpkgs.overlays = [
  (self: super: {
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="/nix/var/cache/ccache"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
  })
];

Derivation CCache

Packages can built with CCache by overriding stdenv in the derivation.

nixpkgs.overlays = [
  (self: super: {
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
  })
];

Some packages do not use stdenv directly. You may need to plumb it through other dependencies first.

Monitor CCache status

# watch ccache size
sudo watch du -sh /nix/var/cache/ccache

# watch ccache stats
sudo watch ccache --dir /nix/var/cache/ccache --show-stats

Uncacheable

TODO: why are some compilations "uncacheable" according to ccache --show-stats

See also