Storage optimization: Difference between revisions

Revision as of 15:49, 15 April 2019

A recurring problem with NixOS is lack of space on /. Even if you are using Nix only occasionally, it is easy for /nix/store to go beyond 50G. Here are generic notes on how to not run out of space too often.

Garbage collection

The Nix store sometimes contains entries which are no longer useful.^{[cf. 1]} They can be deleted with nix-collect-garbage -d ^{[cf. 2]} or nix-store --gc.^{[cf. 3]}

Note that if a result file still exists in the file system, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:

$ nix-store --gc --print-roots
/home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4
/home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv
/home/danbst/dev/test-shell/.shell.drv-2-doc -> /nix/store/14gnv1q1w0n9qwa3q23idsqvn51354y8-bash-4.3-p42-doc
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/danbst/testing/.nix-gc-roots/shell.drv -> /nix/store/v3vqf48awjjzjivrx15kfqdh1d7cg4mq-sshpass-1.05
...
/home/danbst/testing/.nix-gc-roots/shell.drv-12 -> /nix/store/a2li4sl9pxh9aflqia2gp7w88ayvjwci-bash-4.3-p42
/home/danbst/testing/.nix-gc-roots/shell.drv-12-doc -> /nix/store/kcswyb1d8zimkym0pjfi2fj1dly1w34w-bash-4.3-p42-doc
/home/danbst/testing/.nix-gc-roots/shell.drv-12-info -> /nix/store/njb817fwiafswzwvj9skw7w7k6b3fnbi-bash-4.3-p42-info
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/analyt/system-3-link -> /nix/store/snrj72189wh9va23fawl3v80v92xnxlm-nixos-system-iron-16.03.1291.efe2d64
/nix/var/nix/profiles/per-container/d-live/system-6-link -> /nix/store/cp2c58hnczsjk5h69ksajq5xfhsyhl6v-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/d-test/system-4-link -> /nix/store/n1w7ywjg65x8iimchznxcyygbgmyfh55-nixos-system-iron-16.03.1287.6ac7ffd
/nix/var/nix/profiles/per-container/dashboard/system-41-link -> /nix/store/7qk19pkwgq0h3a1q9dcql3nks40rr75s-nixos-system-iron-16.03.1340.5a090dd
...
/nix/var/nix/profiles/per-container/ttt/system-1-link -> /nix/store/1kj9qs5gl3421jlkl3jfc2kqdsl8akwr-nixos-system-ttt-16.03.977.1da05df
/nix/var/nix/profiles/per-user/danbst/channels-1-link -> /nix/store/s0qay9qyqrn92zayldbvvj3zrfcl7a72-user-environment
/nix/var/nix/profiles/per-user/danbst/profile-28-link -> /nix/store/69ds606146dqml04sm0fbpqwnv2w8i3q-user-environment
/nix/var/nix/profiles/per-user/ec2-user/profile-7-link -> /nix/store/y2hc7zsnkzys9ba6xaijvjhff03rcgpy-user-environment
/nix/var/nix/profiles/per-user/root/channels-4-link -> /nix/store/254b6pkhhnjywvj5c0lp2vdai8nz4p0g-user-environment
/nix/var/nix/profiles/system-398-link -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom

Don't forget to run `nix-collect-garbage -d` as root

Run nix-collect-garbage -d with sudo or su, or logged in as root, otherwise you are clearing only your old user profiles, not system-wide profiles.

Look for `result` symlinks

If you use nix-build, but don't use --no-build-output, your FS will be filled with result symlinks to various derivations. In the example above, note the following symlinks:

/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git

How much space do these (apparently) abandoned derivations use?

$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)
...
3.4G    total

Not all of the derivations are garbage in this case, but quite a few are:

# rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result
# nix-collect-garbage -d
...
690 store paths deleted, 1817.99 MiB freed

Especially look for system derivations. Those are created in many cases, for example, when running nixos-rebuild build-vm

Don't forget to reboot

As you see, the reference in /run/booted-system is a GC root, so it won't be cleared until reboot. If you don't want to reboot, just rm /run/booted-system that link and rerun sudo nix-collect-garbage.

Proper `nix-shell` GC pinning

When you invoke nix-shell with

nix-instantiate shell.nix --indirect --add-root $DIR/.nix-gc-roots/shell.drv ...

then you'll have a persistent environment which won't be garbage collected. It is useful when you don't want to spend time waiting for redownloads every time you enter the shell.

A little problem exists though. GC roots are numbered sequentially, so if you change shell.nix to contain less derivations, and name of last GC root will start with shell.drv-7, then shell.drv-{8,9,10,11,12}* will be dangling and unused. To overcome this problem you should remove GC roots dir periodically (or just before nix-shell)

Obviously, you should remove the GC roots directory for projects you don't plan to work on.

Automatic garbage collection

Next you may enable periodic auto GC, for example like this:

nix.gc.automatic = true;
nix.gc.dates = "weekly";
nix.gc.options = "--delete-older-than 30d";

This can result into some redownloads (if you ever use import (builtins.fetchFromTarball ...) all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.

Moving the store

/nix can reside on another device. This is useful if your root device is very small, and that you have another, larger drive at hand. It is easiest to set up when installing NixOS. To move /nix to another device on an existing NixOS installation:

Create a new ext4 partition and mount it over /mnt
rsync -a everything from /nix to /mnt

bind /mnt to /nix (e.g. using mount) and rerun nixos-rebuild switch with something like

fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };

reboot to be sure /nix/store is properly mounted
bind mount / to /old_root, and remove /old_root/nix

Keep in mind that all commands like mount and bash point to some executable in /nix/store, so never mount an empty disk over /nix or /nix/store, otherwise you will be locked out until reboot!

Optimizing the store

Run nix-store --optimise in case you really need some a bit of space. It is a lengthy operation, and after multiple-outputs PR landing, it became less useful.

# nix-store --optimise
985.41 MiB freed by hard-linking 251942 files

@@ Line 1: / Line 1: @@
-A recurring problem with NixOS is lack of space on <code>/</code>. Even if you are using Nix only occasionally, it is easy for <code>/nix/store</code> to go beyond 50G.
+A recurring problem with NixOS is lack of space on <code>/</code>. Even if you are using Nix only occasionally, it is easy for <code>/nix/store</code> to go beyond 50G. Here are generic notes on how to not run out of space too often.
-Here are generic notes on how to not run out of space too often. I assume you already know about <code>nix-collect-garbage</code> and do care about used disk space.
+== Garbage collection ==
-== Move the store ==
+The Nix store sometimes contains entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage -d}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>
-{{ic|/nix}} can reside on another device. This is useful if your root device is very small, and that you have another, larger drive at hand. It is easiest to set up when installing NixOS. To move <code>/nix</code> to another device on an existing NixOS installation:
+Note that if a result file still exists in the file system, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:
-# Create a new ext4 partition and mount it over <code>/mnt</code>
-# <code>rsync -a</code> everything from <code>/nix</code> to <code>/mnt</code>
-# bind <code>/mnt</code> to <code>/nix</code> (e.g. using <code>mount</code>) and rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };</syntaxhighlight>
-# reboot to be sure <code>/nix/store</code> is properly mounted
-# bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>
-Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot!
-== Enable periodic Garbage Collection ==
-Next you may enable periodic auto GC, for example like this:
-<syntaxhighlight lang="bash">
-nix.gc.automatic = true;
-nix.gc.dates = "weekly";
-nix.gc.options = "--delete-older-than 30d";</syntaxhighlight>
-This can result into some redownloads (if you ever use <code>import (builtins.fetchFromTarball ...)</code> all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.
-== You will never be free of manual GC! ==
-One of least mentioned problems with <code>nix-collect-garbage -d</code> is it doesn't collect all the garbage! The key to understand how's that, is to run this command:
 <syntaxhighlight lang="bash">$ nix-store --gc --print-roots
@@ Line 92: / Line 70: @@
 Obviously, you should remove the GC roots directory for projects you don't plan to work on.
-== Last resort - <code>--optimise</code> ==
+=== Automatic garbage collection ===
+Next you may enable periodic auto GC, for example like this:
+<syntaxhighlight lang="bash">
+nix.gc.automatic = true;
+nix.gc.dates = "weekly";
+nix.gc.options = "--delete-older-than 30d";</syntaxhighlight>
+This can result into some redownloads (if you ever use <code>import (builtins.fetchFromTarball ...)</code> all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.
+== Moving the store ==
+{{ic|/nix}} can reside on another device. This is useful if your root device is very small, and that you have another, larger drive at hand. It is easiest to set up when installing NixOS. To move <code>/nix</code> to another device on an existing NixOS installation:
+# Create a new ext4 partition and mount it over <code>/mnt</code>
+# <code>rsync -a</code> everything from <code>/nix</code> to <code>/mnt</code>
+# bind <code>/mnt</code> to <code>/nix</code> (e.g. using <code>mount</code>) and rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };</syntaxhighlight>
+# reboot to be sure <code>/nix/store</code> is properly mounted
+# bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>
+Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot!
+== Optimizing the store ==
 Run <code>nix-store --optimise</code> in case you really need some a bit of space. It is a lengthy operation, and after multiple-outputs PR landing, it became less useful.
-<syntaxhighlight lang="bash"># nix-store --optimise
+<pre>
-.41 MiB freed by hard-linking 251942 files</syntaxhighlight>
+# nix-store --optimise
+.41 MiB freed by hard-linking 251942 files
+</pre>
+== See also ==
+<references group="cf."/>

Revision as of 15:49, 15 April 2019

Garbage collection

Don't forget to run nix-collect-garbage -d as root

Look for result symlinks