View source for Storage optimization

A recurring problem with NixOS is lack of space on <code>/</code>. Even if you are using Nix only occasionally, it is easy for <code>/nix/store</code> to go beyond 50GiB. Here are generic notes on how to not run out of space too often.

== Garbage collection ==

The Nix store sometimes contains entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage -d}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>

Note that if a result file still exists in the file system, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:

<syntaxhighlight lang="console">$ nix-store --gc --print-roots
/home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4
/home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv
/home/danbst/dev/test-shell/.shell.drv-2-doc -> /nix/store/14gnv1q1w0n9qwa3q23idsqvn51354y8-bash-4.3-p42-doc
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/danbst/testing/.nix-gc-roots/shell.drv -> /nix/store/v3vqf48awjjzjivrx15kfqdh1d7cg4mq-sshpass-1.05
...
/home/danbst/testing/.nix-gc-roots/shell.drv-12 -> /nix/store/a2li4sl9pxh9aflqia2gp7w88ayvjwci-bash-4.3-p42
/home/danbst/testing/.nix-gc-roots/shell.drv-12-doc -> /nix/store/kcswyb1d8zimkym0pjfi2fj1dly1w34w-bash-4.3-p42-doc
/home/danbst/testing/.nix-gc-roots/shell.drv-12-info -> /nix/store/njb817fwiafswzwvj9skw7w7k6b3fnbi-bash-4.3-p42-info
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/analyt/system-3-link -> /nix/store/snrj72189wh9va23fawl3v80v92xnxlm-nixos-system-iron-16.03.1291.efe2d64
/nix/var/nix/profiles/per-container/d-live/system-6-link -> /nix/store/cp2c58hnczsjk5h69ksajq5xfhsyhl6v-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/d-test/system-4-link -> /nix/store/n1w7ywjg65x8iimchznxcyygbgmyfh55-nixos-system-iron-16.03.1287.6ac7ffd
/nix/var/nix/profiles/per-container/dashboard/system-41-link -> /nix/store/7qk19pkwgq0h3a1q9dcql3nks40rr75s-nixos-system-iron-16.03.1340.5a090dd
...
/nix/var/nix/profiles/per-container/ttt/system-1-link -> /nix/store/1kj9qs5gl3421jlkl3jfc2kqdsl8akwr-nixos-system-ttt-16.03.977.1da05df
/nix/var/nix/profiles/per-user/danbst/channels-1-link -> /nix/store/s0qay9qyqrn92zayldbvvj3zrfcl7a72-user-environment
/nix/var/nix/profiles/per-user/danbst/profile-28-link -> /nix/store/69ds606146dqml04sm0fbpqwnv2w8i3q-user-environment
/nix/var/nix/profiles/per-user/ec2-user/profile-7-link -> /nix/store/y2hc7zsnkzys9ba6xaijvjhff03rcgpy-user-environment
/nix/var/nix/profiles/per-user/root/channels-4-link -> /nix/store/254b6pkhhnjywvj5c0lp2vdai8nz4p0g-user-environment
/nix/var/nix/profiles/system-398-link -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom</syntaxhighlight>

=== Run as root ===
<code>nix-collect-garbage -d</code> operates only for the current user. To clear system profiles, run it with root privileges.

=== Look for <code>result</code> symlinks ===

If you use <code>nix-build</code>, but not <code>--no-build-output</code>, your FS will be filled with <code>result</code> symlinks to various derivations. In the example above, note the following symlinks:
<syntaxhighlight lang="bash">/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git</syntaxhighlight>
How much space do these (apparently) abandoned derivations use?

<syntaxhighlight lang="console">$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)
...
3.4G    total</syntaxhighlight>
Not all of the derivations are garbage in this case, but quite a few are:

<syntaxhighlight lang="console">
# rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result
# nix-collect-garbage -d
...
690 store paths deleted, 1817.99 MiB freed
</syntaxhighlight>
Look for system derivations in particular. Those are created on many occasions, for example when running <code>nixos-rebuild  build-vm</code>

=== Reboot ===

As you see, the reference in <code>/run/booted-system</code> is a GC root, so it won't be cleared until reboot. If you don't want to reboot, just <code>rm /run/booted-system</code> that link and rerun <code>sudo nix-collect-garbage</code>.

=== Pinning ===

When you invoke <code>nix-shell</code> with

<syntaxhighlight lang="console">
$ nix-instantiate shell.nix --indirect --add-root $DIR/.nix-gc-roots/shell.drv ...
</syntaxhighlight>
then you'll have a persistent environment which won't be garbage collected. It is useful when you don't want to spend time waiting for redownloads every time you enter the shell.

A little problem exists though. GC roots are numbered sequentially, so if you change <code>shell.nix</code> to contain '''less''' derivations, and name of last GC root will start with <code>shell.drv-7</code>, then <code>shell.drv-{8,9,10,11,12}*</code> will be dangling and unused. To overcome this problem you should remove GC roots dir periodically (or just before <code>nix-shell</code>)

Obviously, you should remove the GC roots directory for projects you don't plan to work on.

=== Automation ===

It is possible to enable periodic automatic GC,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example like this:

<syntaxhighlight lang="nix">
nix.gc = {
  automatic = true;
  dates = "weekly";
  options = "--delete-older-than 30d";
};
</syntaxhighlight>

This can result in redownloads (tarballs fetched with <code>import (builtins.fetchFromTarball ...)</code> for example are not referenced anywhere and removed on GC), but it frees you from runnning GC manually.

It is also possible to automatically run garbage collection whenever there is not enough space left.<ref group="cf.">{{nix:option|min-free}} and {{nix:option|max-free}}</ref> For example, to free up to 1GiB whenever there is less than 100MiB left:
<syntaxhighlight lang="nix">
nix.extraOptions = ''
  min-free = ${toString (100 * 1024 * 1024)}
  max-free = ${toString (1024 * 1024 * 1024)}
'';
</syntaxhighlight>
This is particularly useful when the store is on its own partition, see [[#Moving_the_store|below]].

== Moving the store ==

{{ic|/nix}} can reside on another device. This is useful if your root device is very small, and that you have another, larger drive at hand. It is easiest to set up when installing NixOS. To move <code>/nix</code> to another device on an existing NixOS installation:

# Create a new ext4 partition and mount it over <code>/mnt</code>
# <code>rsync -a</code> everything from <code>/nix</code> to <code>/mnt</code>
# bind <code>/mnt</code> to <code>/nix</code> (e.g. using <code>mount</code>) and rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };</syntaxhighlight>
# reboot to be sure <code>/nix/store</code> is properly mounted
# bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>

Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot!

== Optimizing the store ==

Run <code>nix-store --optimise</code> in case you really need some a bit of space. It is a lengthy operation, and after multiple-outputs PR landing, it became less useful.

<syntaxhighlight lang="console">
# nix-store --optimise
985.41 MiB freed by hard-linking 251942 files
</syntaxhighlight>

== See also ==

<references group="cf."/>