Storage optimization: Difference between revisions
imported>Mth m bash syntax highlighting makes root prompts look like comments |
m Fix trailing typo. Tags: Mobile edit Mobile web edit |
||
(27 intermediate revisions by 17 users not shown) | |||
Line 1: | Line 1: | ||
A recurring problem with NixOS is lack of space on <code>/</code>. Even if you | A recurring problem with NixOS is lack of space on <code>/</code>. Even if you only occasionally use Nix, it is easy for <code>/nix/store</code> to grow beyond reasonable sizes. What follows are generic notes on how to reduce the growth of the [[Nix store]]. | ||
== Optimising the store == | |||
Here, we demonstrate how to configure <code>nix</code> to save space via hardlinking store files. | |||
=== Automatic === | |||
To turn on periodic optimisation of the nix store, set the following option in <code>/etc/nixos/configuration.nix</code>: | |||
{{file|configuration.nix|nix|<nowiki> | |||
nix.optimise.automatic = true; | |||
nix.optimise.dates = [ "03:45" ]; # Optional; allows customizing optimisation schedule | |||
</nowiki>}} | |||
Alternatively, the store can be optimised during ''every'' build. This may slow down builds, as discussed [https://github.com/NixOS/nix/issues/6033 here]. To enable this behavior, set the following option: | |||
{{file|configuration.nix|nix|<nowiki> | |||
nix.settings.auto-optimise-store = true; | |||
</nowiki>}} | |||
{{Tip|This option only applies to new files, so we recommend manually optimising your nix store when first setting this option.}} | |||
=== Manual === | |||
Run | |||
{{ic|# nix-store --optimise}}. | |||
This is a potentially long operation. | |||
== Garbage collection == | == Garbage collection == | ||
The Nix store | The Nix store accumulates entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref> | ||
Note that if a result file still exists in the file system, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run: | Note that if a result file still exists in the file system, and your Nix configuration has both <code>keep-outputs = true</code> and <code>keep-derivations = true</code>, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run: | ||
<syntaxhighlight lang=" | <syntaxhighlight lang="console"> | ||
$ nix-store --gc --print-roots | |||
/home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4 | /home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4 | ||
/home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv | /home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv | ||
Line 31: | Line 54: | ||
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git | /root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git | ||
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom | /run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom | ||
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom</syntaxhighlight> | /run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom | ||
</syntaxhighlight> | |||
= | GC roots can be found in <code>/nix/var/nix/gcroots</code>. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks: | ||
<syntaxhighlight lang="console"> | |||
$ find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo' | |||
</syntaxhighlight> | |||
This acts a simpler (but faster) version of <code>--print-roots</code> and could be implemented as a bash alias for convenience. | |||
{{Tip| | |||
<code>nix-collect-garbage -d</code> operates only for the current user. To clear system profiles, run it with root privileges. | <code>nix-collect-garbage -d</code> operates only for the current user. To clear system profiles, run it with root privileges. | ||
}} | |||
=== Look for <code>result</code> symlinks === | === Look for <code>result</code> symlinks === | ||
If you use <code>nix-build</code>, but | If you use <code>nix-build</code>, but not <code>--no-build-output</code>, your file system will be filled with <code>result</code> symlinks to various derivations. In the example above, note the following symlinks: | ||
<syntaxhighlight lang="bash">/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0 | |||
<syntaxhighlight lang="bash"> | |||
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0 | |||
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739 | /home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739 | ||
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git</syntaxhighlight> | /root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git | ||
</syntaxhighlight> | |||
How much space do these (apparently) abandoned derivations use? | How much space do these (apparently) abandoned derivations use? | ||
<syntaxhighlight lang=" | <syntaxhighlight lang="console"> | ||
$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result) | |||
... | ... | ||
3.4G total</syntaxhighlight> | 3.4G total | ||
</syntaxhighlight> | |||
Not all of the derivations are garbage in this case, but quite a few are: | Not all of the derivations are garbage in this case, but quite a few are: | ||
< | <syntaxhighlight lang="console"> | ||
# rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result | # rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result | ||
# nix-collect-garbage -d | # nix-collect-garbage -d | ||
... | ... | ||
690 store paths deleted, 1817.99 MiB freed | 690 store paths deleted, 1817.99 MiB freed | ||
</ | </syntaxhighlight> | ||
Look for system derivations in particular. Those are created on many occasions, for example when running <code>nixos-rebuild build-vm</code> | |||
=== Removing old generations === | |||
NixOS keeps old configurations of your system around so that you can always rollback to a previous configuration if something goes wrong. You can also select which generation to boot into via GRUB. | |||
However these previous generations are GC roots that can keep around old, unnecessary software in your nix store. You can check what system generations you have with | |||
<syntaxhighlight lang="console"> | |||
$ sudo nix-env -p /nix/var/nix/profiles/system --list-generations | |||
... | |||
58 2021-09-04 02:56:54 | |||
59 2021-09-05 07:12:43 | |||
60 2021-09-05 22:12:13 (current) | |||
</syntaxhighlight> | |||
You can remove all but the current generation with | |||
<syntaxhighlight lang="console"> | |||
$ sudo nix-collect-garbage -d | |||
... | |||
4394 store paths deleted, 3467.28 MiB freed | |||
</syntaxhighlight> | |||
There are also user-specific generations for different things (eg. home-manager). These can be removed with | |||
<syntaxhighlight lang="console"> | |||
$ nix-collect-garbage -d | |||
</syntaxhighlight> | |||
=== Reboot === | === Reboot === | ||
Line 63: | Line 128: | ||
=== Pinning === | === Pinning === | ||
Running the following command: | |||
<syntaxhighlight lang="console"> | |||
$ nix-instantiate shell.nix --indirect --add-root ./.nix-gc-roots/shell.drv ... | |||
</syntaxhighlight> | |||
< | Will create a persistent snapshot of your <code>shell.nix</code> dependencies, which then won't be garbage collected, as long as you have configured <code>keep-outputs = true</code> (and haven't changed the default of <code>keep-derivations = true</code>). This is useful if your project has a dependency with no substitutes available, or you don't want to spend time waiting to re-download your dependencies every time you enter the shell. | ||
then | |||
You need to re-run that <code>nix-instantiate</code> command any time your <code>shell.nix</code> changes. | |||
And there is a subtle gotcha if your <code>shell.nix</code> happens to evaluate to more than one derivation: <code>nix-instantiate</code> will number each derivation sequentially, so if you change your <code>shell.nix</code> to contain ''fewer'' derivations, such that (for example) the name of the last GC root starts with <code>shell.drv-7</code>, then <code>shell.drv-{8,9,10,11,12...}</code> will be dangling and unused. | |||
The easiest way to get around this is to delete the <code>./.nix-gc-roots</code> directory periodically (i.e., any time you re-run the <code>nix-instantiate</code> command). | |||
Don't forget to periodically check your GC roots, and remove any that you no longer need. | |||
=== Automation === | === Automation === | ||
Garbage collection can be automated,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example: | |||
< | {{file|configuration.nix|nix|<nowiki> | ||
nix.gc = { | nix.gc = { | ||
automatic = true; | automatic = true; | ||
Line 82: | Line 154: | ||
options = "--delete-older-than 30d"; | options = "--delete-older-than 30d"; | ||
}; | }; | ||
</nowiki>}} | |||
If using nix-darwin, use this to run on 0th day of every week: | |||
{{file|configuration.nix|nix|<nowiki> | |||
nix.gc = { | |||
automatic = true; | |||
interval = { Weekday = 0; Hour = 0; Minute = 0; }; | |||
options = "--delete-older-than 30d"; | |||
}; | |||
</nowiki>}} | |||
This can result in redownloads (tarballs fetched with <code>import (builtins.fetchTarball ...)</code> for example are not referenced anywhere and removed on GC), but it frees you from runnning GC manually. | |||
It is also possible to automatically run garbage collection whenever there is not enough space left.<ref group="cf.">{{nix:option|min-free}} and {{nix:option|max-free}}</ref> For example, to free up to 1GiB whenever there is less than 100MiB left: | |||
<syntaxhighlight lang="nix"> | |||
nix.extraOptions = '' | |||
min-free = ${toString (100 * 1024 * 1024)} | |||
max-free = ${toString (1024 * 1024 * 1024)} | |||
''; | |||
</syntaxhighlight> | </syntaxhighlight> | ||
This is particularly useful when the store is on its own partition, see [[#Moving_the_store|below]]. | |||
== Moving the store == | |||
{{ic|/nix}} can reside on another device, which is useful if your root device is very small, and you have another, larger drive available. | |||
{{ic| | If the new partition is on the same device, some benefit can be gained by formatting the partition on which {{ic|nix}} resides with a different file system. For example: on a Raspberry Pi, {{ic|f2fs}} could be used for a gain in I/O throughput. | ||
Regardless of <code>/nix</code>'s filesystem, it can also be mounted with <code>noatime</code> (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan. | |||
This is easiest to set up while installing NixOS, but <code>/nix</code> can be moved on a live system: | |||
''All commands below are executed with root privileges'' | |||
# Create a new partition | |||
# Mount this new partition over <code>/mnt</code> <syntaxhighlight lang="console"> | |||
# mount /dev/disk/by-label/nix /mnt | |||
</syntaxhighlight> | |||
# Copy everything from <code>/nix</code> to <code>/mnt</code> ''Trailing slashes are important'', in that without them, <code>rsync</code> will create an additional directory of the same name at the destination. <syntaxhighlight lang="console"> | |||
# rsync --archive --hardlinks --acls --one-file-system --verbose /nix/store/ /mnt/store | |||
# rsync --archive --hardlinks --acls --one-file-system --verbose /nix/var/ /mnt/var | |||
</syntaxhighlight> | |||
# Mount the new partition as the new <code>/nix</code> <syntaxhighlight lang="console"> | |||
# umount /mnt | |||
# mount /dev/disk/by-label/nix /nix | |||
</syntaxhighlight> | |||
# Restart {{ic|nix-daemon}} <syntaxhighlight lang="console"> | |||
$ systemctl stop nix-daemon.service | |||
$ systemctl restart nix-daemon.socket | |||
$ systemctl start nix-daemon.service | |||
</syntaxhighlight> | |||
# Add the new <code>/nix</code> partition to your <code>/etc/nixos/configuration.nix</code> <syntaxhighlight lang="nix"> | |||
{ | |||
# ... | |||
fileSystems."/nix" = { | |||
device = "/dev/disk/by-label/nix"; | |||
fsType = "ext4"; | |||
neededForBoot = true; | |||
options = [ "noatime" ]; | |||
}; | |||
} | |||
</syntaxhighlight> | |||
# Apply your configuration <syntaxhighlight lang="console"> | |||
# nixos-rebuild switch | |||
</syntaxhighlight> | |||
# Reboot to be sure <code>/nix/store</code> is properly mounted | |||
''Optionally'' | |||
# After reboot, check that <code>/nix</code> is mounted over your partition <syntaxhighlight lang="console"> | |||
# mount | grep "/nix" && echo "Nix store is on a new partition" || echo "Nix is on the old partition" | |||
</syntaxhighlight> | |||
# Once '''you are sure''' everything works, you can delete the old store <syntaxhighlight lang="console"> | |||
# mkdir /tmp/old_root | |||
# mount --bind / /tmp/old_root | |||
# rm --recursive /tmp/old_root/nix | |||
# umount /tmp/old_root | |||
# rmdir /tmp/old_root | |||
</syntaxhighlight> | |||
< | {{Tip|Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>. It is possible to get locked out of a system if one mistakenly mounted an empty drive to {{ic|nix}}. }} | ||
</ | |||
== See also == | == See also == | ||
<references group="cf."/> | <references group="cf."/> |