NixOS Wiki

Storage optimization: Difference between revisions

← Older edit
VisualWikitext
imported>Mth
No edit summary
DoggoBit (talk | contribs)
trusted
1,085 edits
m propose merge with Nix store
 
(37 intermediate revisions by 21 users not shown)
Line 1: Line 1:
A recurring problem with NixOS is lack of space on <code>/</code>. Even if you are using Nix only occasionally, it is easy for <code>/nix/store</code> to go beyond 50G. Here are generic notes on how to not run out of space too often.
{{Merge|Nix store}}
 
A recurring problem with NixOS is lack of space on <code>/</code>. Even if you only occasionally use Nix, it is easy for <code>/nix/store</code> to grow beyond reasonable sizes. What follows are generic notes on how to reduce the growth of the [[Nix store]].
 
== Optimising the store ==
Here, we demonstrate how to configure <code>nix</code> to save space via hardlinking store files.
 
=== Automatic ===
To turn on periodic optimisation of the nix store, set the following option in <code>/etc/nixos/configuration.nix</code>:
{{file|configuration.nix|nix|<nowiki>
nix.optimise.automatic = true;
nix.optimise.dates = [ "03:45" ]; # Optional; allows customizing optimisation schedule
</nowiki>}}
 
Alternatively, the store can be optimised during ''every'' build. This may slow down builds, as discussed [https://github.com/NixOS/nix/issues/6033 here]. To enable this behavior, set the following option:
{{file|configuration.nix|nix|<nowiki>
nix.settings.auto-optimise-store = true;
</nowiki>}}
 
{{Tip|This option only applies to new files, so we recommend manually optimising your nix store when first setting this option.}}
 
=== Manual ===
Run
{{ic|# nix-store --optimise}}.
This is a potentially long operation.


== Garbage collection ==
== Garbage collection ==


The Nix store sometimes contains entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage -d}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>
By default, the Nix store will not remove any entries that are no longer used, thus it can accumulate derivations you might no longer need.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>


Note that if a result file still exists in the file system, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:
=== Removing old generations ===
NixOS keeps old configurations of your system around so that you can always rollback to a previous configuration if something goes wrong. You can also select which generation to boot into via GRUB. However these previous generations are [[Storage optimization#Garbage collection roots|GC roots]] (see below) that can keep around old, unnecessary software in your nix store. You can check what system generations you have by:<syntaxhighlight lang="console">$ sudo nix-env -p /nix/var/nix/profiles/system --list-generations
...
  58  2021-09-04 02:56:54
  59  2021-09-05 07:12:43
  60  2021-09-05 22:12:13  (current)</syntaxhighlight>


<syntaxhighlight lang="bash">$ nix-store --gc --print-roots
You can remove all other NixOS generations besides the current one:
/home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4
<syntaxhighlight lang="console">
/home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv
$ sudo nix-collect-garbage -d
/home/danbst/dev/test-shell/.shell.drv-2-doc -> /nix/store/14gnv1q1w0n9qwa3q23idsqvn51354y8-bash-4.3-p42-doc
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/danbst/testing/.nix-gc-roots/shell.drv -> /nix/store/v3vqf48awjjzjivrx15kfqdh1d7cg4mq-sshpass-1.05
...
...
/home/danbst/testing/.nix-gc-roots/shell.drv-12 -> /nix/store/a2li4sl9pxh9aflqia2gp7w88ayvjwci-bash-4.3-p42
4394 store paths deleted, 3467.28 MiB freed
/home/danbst/testing/.nix-gc-roots/shell.drv-12-doc -> /nix/store/kcswyb1d8zimkym0pjfi2fj1dly1w34w-bash-4.3-p42-doc
</syntaxhighlight>
/home/danbst/testing/.nix-gc-roots/shell.drv-12-info -> /nix/store/njb817fwiafswzwvj9skw7w7k6b3fnbi-bash-4.3-p42-info
 
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
There are also user-specific generations for different things (eg. home-manager). These can be removed with:
/nix/var/nix/profiles/per-container/analyt/system-3-link -> /nix/store/snrj72189wh9va23fawl3v80v92xnxlm-nixos-system-iron-16.03.1291.efe2d64
<syntaxhighlight lang="console">
/nix/var/nix/profiles/per-container/d-live/system-6-link -> /nix/store/cp2c58hnczsjk5h69ksajq5xfhsyhl6v-nixos-system-iron-16.03.1299.a8e0739
$ nix-collect-garbage -d
/nix/var/nix/profiles/per-container/d-test/system-4-link -> /nix/store/n1w7ywjg65x8iimchznxcyygbgmyfh55-nixos-system-iron-16.03.1287.6ac7ffd
</syntaxhighlight>
/nix/var/nix/profiles/per-container/dashboard/system-41-link -> /nix/store/7qk19pkwgq0h3a1q9dcql3nks40rr75s-nixos-system-iron-16.03.1340.5a090dd
 
=== Garbage collection roots ===
 
Note that if a result file still exists in the file system, and your Nix configuration has both <code>keep-outputs = true</code> and <code>keep-derivations = true</code>, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:
 
<syntaxhighlight lang="console">
$ nix-store --gc --print-roots
...
...
/nix/var/nix/profiles/per-container/ttt/system-1-link -> /nix/store/1kj9qs5gl3421jlkl3jfc2kqdsl8akwr-nixos-system-ttt-16.03.977.1da05df
/nix/var/nix/profiles/per-user/danbst/channels-1-link -> /nix/store/s0qay9qyqrn92zayldbvvj3zrfcl7a72-user-environment
/nix/var/nix/profiles/per-user/danbst/profile-28-link -> /nix/store/69ds606146dqml04sm0fbpqwnv2w8i3q-user-environment
/nix/var/nix/profiles/per-user/ec2-user/profile-7-link -> /nix/store/y2hc7zsnkzys9ba6xaijvjhff03rcgpy-user-environment
/nix/var/nix/profiles/per-user/root/channels-4-link -> /nix/store/254b6pkhhnjywvj5c0lp2vdai8nz4p0g-user-environment
/nix/var/nix/profiles/per-user/root/channels-4-link -> /nix/store/254b6pkhhnjywvj5c0lp2vdai8nz4p0g-user-environment
/nix/var/nix/profiles/system-398-link -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
/nix/var/nix/profiles/system-398-link -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom</syntaxhighlight>
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
=== Don't forget to run <code>nix-collect-garbage -d</code> as root ===
</syntaxhighlight>
 
GC roots can be found in <code>/nix/var/nix/gcroots</code>. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks:
 
<syntaxhighlight lang="console">
$ find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo'
</syntaxhighlight>
 
This acts a simpler (but faster) version of <code>--print-roots</code> and could be implemented as a bash alias for convenience.


Run <code>nix-collect-garbage -d</code> with <code>sudo</code> or <code>su</code>, or logged in as root, otherwise you are clearing only your old user profiles, not system-wide profiles.
{{Tip|
<code>nix-collect-garbage -d</code> operates only for the current user. To clear system profiles, run it with root privileges.
}}


=== Look for <code>result</code> symlinks ===
=== Look for <code>result</code> symlinks ===


If you use <code>nix-build</code>, but don't use <code>--no-build-output</code>, your FS will be filled with <code>result</code> symlinks to various derivations. In the example above, note the following symlinks:
If you use <code>nix-build</code>, but not <code>--no-build-output</code>, your file system will be filled with <code>result</code> symlinks to various derivations. In the example above, note the following symlinks:
<syntaxhighlight lang="bash">/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
 
<syntaxhighlight lang="bash">
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git</syntaxhighlight>
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
</syntaxhighlight>
 
How much space do these (apparently) abandoned derivations use?
How much space do these (apparently) abandoned derivations use?


<syntaxhighlight lang="bash">$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)
<syntaxhighlight lang="console">
$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)
...
...
3.4G    total</syntaxhighlight>
3.4G    total
</syntaxhighlight>
 
Not all of the derivations are garbage in this case, but quite a few are:
Not all of the derivations are garbage in this case, but quite a few are:


<syntaxhighlight lang="bash"># rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result
<syntaxhighlight lang="console">
# rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result
# nix-collect-garbage -d
# nix-collect-garbage -d
...
...
690 store paths deleted, 1817.99 MiB freed</syntaxhighlight>
690 store paths deleted, 1817.99 MiB freed
Especially look for system derivations. Those are created in many cases, for example, when running <code>nixos-rebuild  build-vm</code>
</syntaxhighlight>


=== Don't forget to reboot ===
Look for system derivations in particular. Those are created on many occasions, for example when running <code>nixos-rebuild  build-vm</code>


=== Reboot ===
As you see, the reference in <code>/run/booted-system</code> is a GC root, so it won't be cleared until reboot. If you don't want to reboot, just <code>rm /run/booted-system</code> that link and rerun <code>sudo nix-collect-garbage</code>.
As you see, the reference in <code>/run/booted-system</code> is a GC root, so it won't be cleared until reboot. If you don't want to reboot, just <code>rm /run/booted-system</code> that link and rerun <code>sudo nix-collect-garbage</code>.


=== Proper <code>nix-shell</code> GC pinning ===
=== Pinning ===
 
Running the following command:
 
<syntaxhighlight lang="console">
$ nix-instantiate shell.nix --indirect --add-root ./.nix-gc-roots/shell.drv ...
</syntaxhighlight>
 
Will create a persistent snapshot of your <code>shell.nix</code> dependencies, which then won't be garbage collected, as long as you have configured <code>keep-outputs = true</code> (and haven't changed the default of <code>keep-derivations = true</code>). This is useful if your project has a dependency with no substitutes available, or you don't want to spend time waiting to re-download your dependencies every time you enter the shell.
 
You need to re-run that <code>nix-instantiate</code> command any time your <code>shell.nix</code> changes.
 
And there is a subtle gotcha if your <code>shell.nix</code> happens to evaluate to more than one derivation: <code>nix-instantiate</code> will number each derivation sequentially, so if you change your <code>shell.nix</code> to contain ''fewer'' derivations, such that (for example) the name of the last GC root starts with <code>shell.drv-7</code>, then <code>shell.drv-{8,9,10,11,12...}</code> will be dangling and unused.
 
The easiest way to get around this is to delete the <code>./.nix-gc-roots</code> directory periodically (i.e., any time you re-run the <code>nix-instantiate</code> command).


When you invoke <code>nix-shell</code> with
Don't forget to periodically check your GC roots, and remove any that you no longer need.


<syntaxhighlight lang="bash">nix-instantiate shell.nix --indirect --add-root $DIR/.nix-gc-roots/shell.drv ...</syntaxhighlight>
=== Automation ===
then you'll have a persistent environment which won't be garbage collected. It is useful when you don't want to spend time waiting for redownloads every time you enter the shell.


A little problem exists though. GC roots are numbered sequentially, so if you change <code>shell.nix</code> to contain '''less''' derivations, and name of last GC root will start with <code>shell.drv-7</code>, then <code>shell.drv-{8,9,10,11,12}*</code> will be dangling and unused. To overcome this problem you should remove GC roots dir periodically (or just before <code>nix-shell</code>)
Garbage collection can be automated,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example:


Obviously, you should remove the GC roots directory for projects you don't plan to work on.
{{file|configuration.nix|nix|<nowiki>
nix.gc = {
  automatic = true;
  dates = "weekly";
  options = "--delete-older-than 30d";
};
</nowiki>}}


=== Automatic garbage collection ===
If using nix-darwin, use this to run on 0th day of every week:
{{file|configuration.nix|nix|<nowiki>
nix.gc = {
  automatic = true;
  interval = { Weekday = 0; Hour = 0; Minute = 0; };
  options = "--delete-older-than 30d";
};
</nowiki>}}
This can result in redownloads (tarballs fetched with <code>import (builtins.fetchTarball ...)</code> for example are not referenced anywhere and removed on GC), but it frees you from runnning GC manually.


Next you may enable periodic auto GC, for example like this:
It is also possible to automatically run garbage collection whenever there is not enough space left.<ref group="cf.">{{nix:option|min-free}} and {{nix:option|max-free}}</ref> For example, to free up to 1GiB whenever there is less than 100MiB left:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="nix">
nix.gc.automatic = true;
nix.extraOptions = ''
nix.gc.dates = "weekly";
  min-free = ${toString (100 * 1024 * 1024)}
nix.gc.options = "--delete-older-than 30d";</syntaxhighlight>
  max-free = ${toString (1024 * 1024 * 1024)}
This can result into some redownloads (if you ever use <code>import (builtins.fetchFromTarball ...)</code> all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.
'';
</syntaxhighlight>
This is particularly useful when the store is on its own partition, see [[#Moving_the_store|below]].


== Moving the store ==
== Moving the store ==


{{ic|/nix}} can reside on another device. This is useful if your root device is very small, and that you have another, larger drive at hand. It is easiest to set up when installing NixOS. To move <code>/nix</code> to another device on an existing NixOS installation:
{{ic|/nix}} can reside on another device, which is useful if your root device is very small, and you have another, larger drive available.
 
If the new partition is on the same device, some benefit can be gained by formatting the partition on which {{ic|nix}} resides with a different file system. For example: on a Raspberry Pi, {{ic|f2fs}} could be used for a gain in I/O throughput.
 
Regardless of <code>/nix</code>'s filesystem, it can also be mounted with <code>noatime</code> (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan.


# Create a new ext4 partition and mount it over <code>/mnt</code>
This is easiest to set up while installing NixOS, but <code>/nix</code> can be moved on a live system:
# <code>rsync -a</code> everything from <code>/nix</code> to <code>/mnt</code>
# bind <code>/mnt</code> to <code>/nix</code> (e.g. using <code>mount</code>) and rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };</syntaxhighlight>
# reboot to be sure <code>/nix/store</code> is properly mounted
# bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>


Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot!
''All commands below are executed with root privileges''


== Optimizing the store ==
# Create a new partition
# Mount this new partition over <code>/mnt</code> <syntaxhighlight lang="console">
# mount -o defaults,noatime /dev/disk/by-label/nix /mnt/nix
</syntaxhighlight>
# Copy everything from <code>/nix</code> to <code>/mnt</code> ''Trailing slashes are important'', in that without them, <code>rsync</code> will create an additional directory of the same name at the destination. <syntaxhighlight lang="console">
# rsync --archive --hard-links --acls --one-file-system --verbose /nix/{store,var} /mnt/nix
</syntaxhighlight>
# Mount the new partition as the new <code>/nix</code> <syntaxhighlight lang="console">
# umount /mnt/nix
# mount /dev/disk/by-label/nix /nix
</syntaxhighlight>
# Restart {{ic|nix-daemon}} <syntaxhighlight lang="console">
$ systemctl stop nix-daemon.service
$ systemctl restart nix-daemon.socket
$ systemctl start nix-daemon.service
</syntaxhighlight>
# Add the new <code>/nix</code> partition to your <code>/etc/nixos/configuration.nix</code> <syntaxhighlight lang="nix">
{
  # ...
  fileSystems."/nix" = {
    device = "/dev/disk/by-label/nix";
    fsType = "ext4";
    neededForBoot = true;
    options = [ "noatime" ];
  };
}
</syntaxhighlight>
# Apply your configuration <syntaxhighlight lang="console">
# nixos-rebuild switch
</syntaxhighlight>
# Reboot to be sure <code>/nix/store</code> is properly mounted


Run <code>nix-store --optimise</code> in case you really need some a bit of space. It is a lengthy operation, and after multiple-outputs PR landing, it became less useful.
''Optionally''
# After reboot, check that <code>/nix</code> is mounted over your partition <syntaxhighlight lang="console">
# mount | grep "/nix" && echo "Nix store is on a new partition" || echo "Nix is on the old partition"
</syntaxhighlight>
# Once '''you are sure''' everything works, you can delete the old store <syntaxhighlight lang="console">
# mkdir /tmp/old_root
# mount --bind / /tmp/old_root
# rm --recursive /tmp/old_root/nix
# umount /tmp/old_root
# rmdir /tmp/old_root
</syntaxhighlight>


<pre>
{{Tip|Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>. It is possible to get locked out of a system if one mistakenly mounted an empty drive to {{ic|nix}}. }}
# nix-store --optimise
985.41 MiB freed by hard-linking 251942 files
</pre>


== See also ==
== See also ==


<references group="cf."/>
<references group="cf."/>
[[Category:NixOS]]
[[Category:Nix]]
Retrieved from "https://wiki.nixos.org/wiki/Storage_optimization"