Storage optimization: Difference between revisions
imported>Petersjt014 note about noatime |
imported>Ianthehenry m minor grammatical tweaks |
||
| Line 8: | Line 8: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Run the optimise command once since this option only applies to new files | Run the optimise command once since this option only applies to new files. | ||
==== Manually ==== | ==== Manually ==== | ||
Run <code>nix-store --optimise</code> | Run <code>nix-store --optimise</code>. This will take some time to complete. | ||
== Garbage collection == | == Garbage collection == | ||
| Line 44: | Line 44: | ||
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom | /run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom | ||
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom</syntaxhighlight> | /run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom</syntaxhighlight> | ||
Information about gc roots can be found in <code>/nix/var/nix/gcroots</code>. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks: | Information about gc roots can be found in <code>/nix/var/nix/gcroots</code>. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks: | ||
| Line 57: | Line 56: | ||
=== Look for <code>result</code> symlinks === | === Look for <code>result</code> symlinks === | ||
If you use <code>nix-build</code>, but not <code>--no-build-output</code>, your | If you use <code>nix-build</code>, but not <code>--no-build-output</code>, your file system will be filled with <code>result</code> symlinks to various derivations. In the example above, note the following symlinks: | ||
<syntaxhighlight lang="bash">/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0 | <syntaxhighlight lang="bash">/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0 | ||
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739 | /home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739 | ||
| Line 82: | Line 81: | ||
=== Pinning === | === Pinning === | ||
When you invoke <code>nix-shell</code> with | When you invoke <code>nix-shell</code> with: | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
$ nix-instantiate shell.nix --indirect --add-root $DIR/.nix-gc-roots/shell.drv ... | $ nix-instantiate shell.nix --indirect --add-root $DIR/.nix-gc-roots/shell.drv ... | ||
</syntaxhighlight> | </syntaxhighlight> | ||
A little problem exists though. GC roots are numbered sequentially, so if you change <code>shell.nix</code> to contain ''' | Then you'll have a persistent environment which won't be garbage collected. This is useful when you don't want to spend time waiting for redownloads every time you enter the shell. | ||
A little problem exists though. GC roots are numbered sequentially, so if you change <code>shell.nix</code> to contain '''fewer''' derivations, such that the name of the last GC root starts with <code>shell.drv-7</code>, then <code>shell.drv-{8,9,10,11,12}*</code> will be dangling and unused. To overcome this problem you should remove GC roots dir periodically (or just before running <code>nix-shell</code>). | |||
Obviously, you should remove the GC roots directory for projects you don't plan to work on. | Obviously, you should remove the GC roots directory for projects you don't plan to work on. | ||
| Line 123: | Line 123: | ||
Regardless of <code>/nix</code>'s filesystem, it can also be mounted with <code>noatime</code> (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan. | Regardless of <code>/nix</code>'s filesystem, it can also be mounted with <code>noatime</code> (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan. | ||
This is easiest to set up while installing NixOS, but <code>/nix</code> can be moved on a live system: | This is easiest to set up while installing NixOS, but <code>/nix</code> can be moved on a live system: | ||
| Line 129: | Line 128: | ||
# Create a new partition and mount it over <code>/mnt</code> | # Create a new partition and mount it over <code>/mnt</code> | ||
# <code>rsync -aAxv</code> everything from <code>/nix</code> to <code>/mnt</code> | # <code>rsync -aAxv</code> everything from <code>/nix</code> to <code>/mnt</code> | ||
# | # Bind <code>/mnt</code> to <code>/nix</code> (e.g. using <code>mount</code>). | ||
# Restart nix daemon with <code>systemctl restart nix-daemon.service </code> <code>systemctl restart nix-daemon.socket </code>. | # Restart nix daemon with <code>systemctl restart nix-daemon.service </code> <code>systemctl restart nix-daemon.socket </code>. | ||
# Rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; neededForBoot = true; options = [ "noatime" ]; };</syntaxhighlight> | # Rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; neededForBoot = true; options = [ "noatime" ]; };</syntaxhighlight> | ||
# | # Reboot to be sure <code>/nix/store</code> is properly mounted. | ||
# Once you are sure everything works, you can delete the old store doing a bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>. | # Once you are sure everything works, you can delete the old store doing a bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>. | ||
Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot! | Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot! | ||
Revision as of 00:34, 22 May 2021
A recurring problem with NixOS is lack of space on /. Even if you are using Nix only occasionally, it is easy for /nix/store to go beyond 50GiB. Here are generic notes on how to not run out of space too often.
Optimizing the store
The option and command below save space by hardlinking store files:
Automatically
nix.autoOptimiseStore = true;
Run the optimise command once since this option only applies to new files.
Manually
Run nix-store --optimise. This will take some time to complete.
Garbage collection
The Nix store sometimes contains entries which are no longer useful.[cf. 1] They can be deleted with nix-collect-garbage -d [cf. 2] or nix-store --gc.[cf. 3]
Note that if a result file still exists in the file system, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:
$ nix-store --gc --print-roots
/home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4
/home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv
/home/danbst/dev/test-shell/.shell.drv-2-doc -> /nix/store/14gnv1q1w0n9qwa3q23idsqvn51354y8-bash-4.3-p42-doc
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/danbst/testing/.nix-gc-roots/shell.drv -> /nix/store/v3vqf48awjjzjivrx15kfqdh1d7cg4mq-sshpass-1.05
...
/home/danbst/testing/.nix-gc-roots/shell.drv-12 -> /nix/store/a2li4sl9pxh9aflqia2gp7w88ayvjwci-bash-4.3-p42
/home/danbst/testing/.nix-gc-roots/shell.drv-12-doc -> /nix/store/kcswyb1d8zimkym0pjfi2fj1dly1w34w-bash-4.3-p42-doc
/home/danbst/testing/.nix-gc-roots/shell.drv-12-info -> /nix/store/njb817fwiafswzwvj9skw7w7k6b3fnbi-bash-4.3-p42-info
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/analyt/system-3-link -> /nix/store/snrj72189wh9va23fawl3v80v92xnxlm-nixos-system-iron-16.03.1291.efe2d64
/nix/var/nix/profiles/per-container/d-live/system-6-link -> /nix/store/cp2c58hnczsjk5h69ksajq5xfhsyhl6v-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/d-test/system-4-link -> /nix/store/n1w7ywjg65x8iimchznxcyygbgmyfh55-nixos-system-iron-16.03.1287.6ac7ffd
/nix/var/nix/profiles/per-container/dashboard/system-41-link -> /nix/store/7qk19pkwgq0h3a1q9dcql3nks40rr75s-nixos-system-iron-16.03.1340.5a090dd
...
/nix/var/nix/profiles/per-container/ttt/system-1-link -> /nix/store/1kj9qs5gl3421jlkl3jfc2kqdsl8akwr-nixos-system-ttt-16.03.977.1da05df
/nix/var/nix/profiles/per-user/danbst/channels-1-link -> /nix/store/s0qay9qyqrn92zayldbvvj3zrfcl7a72-user-environment
/nix/var/nix/profiles/per-user/danbst/profile-28-link -> /nix/store/69ds606146dqml04sm0fbpqwnv2w8i3q-user-environment
/nix/var/nix/profiles/per-user/ec2-user/profile-7-link -> /nix/store/y2hc7zsnkzys9ba6xaijvjhff03rcgpy-user-environment
/nix/var/nix/profiles/per-user/root/channels-4-link -> /nix/store/254b6pkhhnjywvj5c0lp2vdai8nz4p0g-user-environment
/nix/var/nix/profiles/system-398-link -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
Information about gc roots can be found in /nix/var/nix/gcroots. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks:
find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo'
This acts a simpler (but faster) version of --print-roots and could be implemented as a bash alias for convenience.
Run as root
nix-collect-garbage -d operates only for the current user. To clear system profiles, run it with root privileges.
Look for result symlinks
If you use nix-build, but not --no-build-output, your file system will be filled with result symlinks to various derivations. In the example above, note the following symlinks:
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
How much space do these (apparently) abandoned derivations use?
$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)
...
3.4G total
Not all of the derivations are garbage in this case, but quite a few are:
# rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result
# nix-collect-garbage -d
...
690 store paths deleted, 1817.99 MiB freed
Look for system derivations in particular. Those are created on many occasions, for example when running nixos-rebuild build-vm
Reboot
As you see, the reference in /run/booted-system is a GC root, so it won't be cleared until reboot. If you don't want to reboot, just rm /run/booted-system that link and rerun sudo nix-collect-garbage.
Pinning
When you invoke nix-shell with:
$ nix-instantiate shell.nix --indirect --add-root $DIR/.nix-gc-roots/shell.drv ...
Then you'll have a persistent environment which won't be garbage collected. This is useful when you don't want to spend time waiting for redownloads every time you enter the shell.
A little problem exists though. GC roots are numbered sequentially, so if you change shell.nix to contain fewer derivations, such that the name of the last GC root starts with shell.drv-7, then shell.drv-{8,9,10,11,12}* will be dangling and unused. To overcome this problem you should remove GC roots dir periodically (or just before running nix-shell).
Obviously, you should remove the GC roots directory for projects you don't plan to work on.
Automation
It is possible to enable periodic automatic GC,[cf. 4] for example like this:
nix.gc = {
automatic = true;
dates = "weekly";
options = "--delete-older-than 30d";
};
This can result in redownloads (tarballs fetched with import (builtins.fetchTarball ...) for example are not referenced anywhere and removed on GC), but it frees you from runnning GC manually.
It is also possible to automatically run garbage collection whenever there is not enough space left.[cf. 5] For example, to free up to 1GiB whenever there is less than 100MiB left:
nix.extraOptions = ''
min-free = ${toString (100 * 1024 * 1024)}
max-free = ${toString (1024 * 1024 * 1024)}
'';
This is particularly useful when the store is on its own partition, see below.
Moving the store
/nix can reside on another device. This is useful if your root device is very small, and that you have another, larger drive at hand.
If the second mountpoint is on the same device, some benefit can still be gained by formatting the partition it points to with a different file system. For example: on a Raspberry Pi, f2fs could possibly be used for a gain in I/O throughput.
Regardless of /nix's filesystem, it can also be mounted with noatime (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan.
This is easiest to set up while installing NixOS, but /nix can be moved on a live system:
- Create a new partition and mount it over
/mnt rsync -aAxveverything from/nixto/mnt- Bind
/mntto/nix(e.g. usingmount). - Restart nix daemon with
systemctl restart nix-daemon.servicesystemctl restart nix-daemon.socket. - Rerun
nixos-rebuild switchwith something likefileSystems."/nix" = { device = "/dev/disk/by-label/nix"; neededForBoot = true; options = [ "noatime" ]; };
- Reboot to be sure
/nix/storeis properly mounted. - Once you are sure everything works, you can delete the old store doing a bind mount
/to/old_root, and remove/old_root/nix.
Keep in mind that all commands like mount and bash point to some executable in /nix/store, so never mount an empty disk over /nix or /nix/store, otherwise you will be locked out until reboot!