Storage optimization: Difference between revisions

(11 intermediate revisions by 10 users not shown)

Line 1:

A recurring problem with NixOS is lack of space on <code>/</code>. Even if you only ~~ocasionally~~ use Nix, it is easy for <code>/nix/store</code> to grow beyond ~~50GiB~~. ~~Here~~ are generic notes on how to ~~not run out~~ of ~~space too often~~.

A recurring problem with NixOS is lack of space on <code>/</code>. Even if you only occasionally use Nix, it is easy for <code>/nix/store</code> to grow beyond reasonable sizes. What follows are generic notes on how to reduce the growth of the [[Nix store]].

== ~~Optimizing~~ the store ==

== Optimising the store ==

~~The option and command below~~ save space by hardlinking store files:

Here, we demonstrate how to configure <code>nix</code> to save space via hardlinking store files.

==== ~~Automatically~~ ====

=== Automatic ===

<~~syntaxhighlight lang="~~nix">

To turn on periodic optimisation of the nix store, set the following option in <code>/etc/nixos/configuration.nix</code>:

nix.~~autoOptimiseStore~~ = true;

{{file|configuration.nix|nix|<nowiki>

</~~syntaxhighlight~~>

nix.optimise.automatic = true;

nix.optimise.dates = [ "03:45" ]; # Optional; allows customizing optimisation schedule

</nowiki>}}

Alternatively, the store can be optimised during ''every'' build. This may slow down builds, as discussed [https://github.com/NixOS/nix/issues/6033 here]. To enable this behavior, set the following option:

{{file|configuration.nix|nix|<nowiki>

nix.settings.auto-optimise-store = true;

</nowiki>}}

~~But this~~ option only applies to new files~~: you'll still need to~~ manually ~~optimise~~ your store ~~once, after you enable~~ this option.

{{Tip|This option only applies to new files, so we recommend manually optimising your nix store when first setting this option.}}

===~~= Manually =~~===

=== Manual ===

Run ~~<code>~~nix-store --optimise~~</code>~~. This ~~will take some time to complete~~.

Run

{{ic|# nix-store --optimise}}.

This is a potentially long operation.

== Garbage collection ==

The Nix store ~~sometimes contains~~ entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage -d}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>

The Nix store accumulates entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>

Note that if a result file still exists in the file system, and your Nix configuration has both <code>keep-outputs = true</code> and <code>keep-derivations = true</code>, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:

Line 50:

Line 59:

GC roots can be found in <code>/nix/var/nix/gcroots</code>. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks:

find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo'

$ find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo'

</syntaxhighlight>

This acts a simpler (but faster) version of <code>--print-roots</code> and could be implemented as a bash alias for convenience.

~~=== Run as root ===~~

{{Tip|

<code>nix-collect-garbage -d</code> operates only for the current user. To clear system profiles, run it with root privileges.

}}

=== Look for <code>result</code> symlinks ===

Line 87:

Line 97:

Look for system derivations in particular. Those are created on many occasions, for example when running <code>nixos-rebuild build-vm</code>

=== Removing old generations ===

NixOS keeps old configurations of your system around so that you can always rollback to a previous configuration if something goes wrong. You can also select which generation to boot into via GRUB.

However these previous generations are GC roots that can keep around old, unnecessary software in your nix store. You can check what system generations you have with

$ sudo nix-env -p /nix/var/nix/profiles/system --list-generations

...

58 2021-09-04 02:56:54

59 2021-09-05 07:12:43

60 2021-09-05 22:12:13 (current)

</syntaxhighlight>

You can remove all but the current generation with

$ sudo nix-collect-garbage -d

...

4394 store paths deleted, 3467.28 MiB freed

</syntaxhighlight>

There are also user-specific generations for different things (eg. home-manager). These can be removed with

$ nix-collect-garbage -d

</syntaxhighlight>

=== Reboot ===

Line 112:

Line 146:

=== Automation ===

~~It is possible to enable periodic automatic GC~~,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example ~~like this~~:

Garbage collection can be automated,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example:

<~~syntaxhighlight lang="nix"~~>

{{file|configuration.nix|nix|<nowiki>

nix.gc = {

automatic = true;

Line 120:

Line 154:

options = "--delete-older-than 30d";

};

</~~syntaxhighlight~~>

</nowiki>}}

If using nix-darwin, use this to run on 0th day of every week:

{{file|configuration.nix|nix|<nowiki>

nix.gc = {

automatic = true;

interval = { Weekday = 0; Hour = 0; Minute = 0; };

options = "--delete-older-than 30d";

};

</nowiki>}}

This can result in redownloads (tarballs fetched with <code>import (builtins.fetchTarball ...)</code> for example are not referenced anywhere and removed on GC), but it frees you from runnning GC manually.

Line 136:

Line 178:

== Moving the store ==

{{ic|/nix}} can reside on another device~~. This~~ is useful if your root device is very small, and you have another, larger drive available.

{{ic|/nix}} can reside on another device, which is useful if your root device is very small, and you have another, larger drive available.

If the ~~second mountpoint~~ is on the same device, some benefit can ~~still~~ be gained by formatting the partition ~~it points to~~ with a different file system. For example: on a Raspberry Pi, f2fs could ~~possibly~~ be used for a gain in I/O throughput.

If the new partition is on the same device, some benefit can be gained by formatting the partition on which {{ic|nix}} resides with a different file system. For example: on a Raspberry Pi, {{ic|f2fs}} could be used for a gain in I/O throughput.

Regardless of <code>/nix</code>'s filesystem, it can also be mounted with <code>noatime</code> (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan.

Line 147:

Line 189:

# Create a new partition

# Mount this new partition over <code>/mnt</code> <syntaxhighlight lang="~~bash~~">

# Mount this new partition over <code>/mnt</code> <syntaxhighlight lang="console">

mount /dev/disk/by-label/nix /mnt

# mount -o defaults,noatime /dev/disk/by-label/nix /mnt/nix

</syntaxhighlight>

# Copy everything from <code>/nix</code> to <code>/mnt</code> ''Trailing slashes are important'', in that without them, <code>rsync</code> will create an additional directory of the same name at the destination. <syntaxhighlight lang="~~bash~~">

# Copy everything from <code>/nix</code> to <code>/mnt</code> ''Trailing slashes are important'', in that without them, <code>rsync</code> will create an additional directory of the same name at the destination. <syntaxhighlight lang="console">

rsync --archive --~~acls --one-file-system --verbose /nix/store/ /mnt/store~~

# rsync --archive --hard-links --acls --one-file-system --verbose /nix/{store,var} /mnt/nix

~~rsync -~~-~~archive~~ --acls --one-file-system --verbose /nix/var/ /mnt/~~var~~

</syntaxhighlight>

# ~~Use~~ the new partition as new <code>/nix</code> <syntaxhighlight lang="~~bash~~">

# Mount the new partition as the new <code>/nix</code> <syntaxhighlight lang="console">

umount /mnt

# umount /mnt/nix

mount /dev/disk/by-label/nix /nix

# mount /dev/disk/by-label/nix /nix

</syntaxhighlight>

# Restart nix daemon <syntaxhighlight lang="~~bash~~">

# Restart {{ic|nix-daemon}} <syntaxhighlight lang="console">

systemctl stop nix-daemon.service

$ systemctl stop nix-daemon.service

systemctl restart nix-daemon.socket

$ systemctl restart nix-daemon.socket

systemctl start nix-daemon.service

$ systemctl start nix-daemon.service

</syntaxhighlight>

# Add ~~mounting~~ the <code>/nix</code> partition to your <code>/etc/nixos/configuration.nix</code> <syntaxhighlight lang="nix">

# Add the new <code>/nix</code> partition to your <code>/etc/nixos/configuration.nix</code> <syntaxhighlight lang="nix">

{

# ...

Line 172:

Line 213:

options = [ "noatime" ];

};

~~# ...~~

}

</syntaxhighlight>

# Apply your configuration <syntaxhighlight lang="~~bash~~">

# Apply your configuration <syntaxhighlight lang="console">

nixos-rebuild switch

# nixos-rebuild switch

</syntaxhighlight>

# Reboot to be sure <code>/nix/store</code> is properly mounted

''Optionally''

# After reboot, check that <code>/nix</code> is mounted over your partition <syntaxhighlight lang="~~bash~~">

# After reboot, check that <code>/nix</code> is mounted over your partition <syntaxhighlight lang="console">

mount | grep "/nix" && echo "Nix ~~seems to use your~~ new partition" || echo "~~It seems that something bad happened~~"

# mount | grep "/nix" && echo "Nix store is on a new partition" || echo "Nix is on the old partition"

</syntaxhighlight>

# Once '''you are sure''' everything works, you can delete the old store <syntaxhighlight lang="~~bash~~">

# Once '''you are sure''' everything works, you can delete the old store <syntaxhighlight lang="console">

mkdir /tmp/old_root

# mkdir /tmp/old_root

mount --bind / /tmp/old_root

# mount --bind / /tmp/old_root

rm --recursive /tmp/old_root/nix

# rm --recursive /tmp/old_root/nix

umount /tmp/old_root

# umount /tmp/old_root

rmdir /tmp/old_root

# rmdir /tmp/old_root

</syntaxhighlight>

Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>~~, so never mount~~ an empty ~~disk over <code>/nix</code> or <code>/~~nix~~/store</code>, otherwise you will be locked out until reboot!~~

{{Tip|Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>. It is possible to get locked out of a system if one mistakenly mounted an empty drive to {{ic|nix}}. }}

== See also ==

@@ Line 1: / Line 1: @@
-A recurring problem with NixOS is lack of space on <code>/</code>. Even if you only ocasionally use Nix, it is easy for <code>/nix/store</code> to grow beyond 50GiB. Here are generic notes on how to not run out of space too often.
+A recurring problem with NixOS is lack of space on <code>/</code>. Even if you only occasionally use Nix, it is easy for <code>/nix/store</code> to grow beyond reasonable sizes. What follows are generic notes on how to reduce the growth of the [[Nix store]].
-== Optimizing the store ==
+== Optimising the store ==
-The option and command below save space by hardlinking store files:
+Here, we demonstrate how to configure <code>nix</code> to save space via hardlinking store files.
-==== Automatically ====
+=== Automatic ===
-<syntaxhighlight lang="nix">
+To turn on periodic optimisation of the nix store, set the following option in <code>/etc/nixos/configuration.nix</code>:
-nix.autoOptimiseStore = true;
+{{file|configuration.nix|nix|<nowiki>
-</syntaxhighlight>
+nix.optimise.automatic = true;
+nix.optimise.dates = [ "03:45" ]; # Optional; allows customizing optimisation schedule
+</nowiki>}}
+Alternatively, the store can be optimised during ''every'' build. This may slow down builds, as discussed [https://github.com/NixOS/nix/issues/6033 here]. To enable this behavior, set the following option:
+{{file|configuration.nix|nix|<nowiki>
+nix.settings.auto-optimise-store = true;
+</nowiki>}}
-But this option only applies to new files: you'll still need to manually optimise your store once, after you enable this option.
+{{Tip|This option only applies to new files, so we recommend manually optimising your nix store when first setting this option.}}
-==== Manually ====
+=== Manual ===
-Run <code>nix-store --optimise</code>. This will take some time to complete.
+Run
+{{ic|# nix-store --optimise}}.
+This is a potentially long operation.
 == Garbage collection ==
-The Nix store sometimes contains entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage -d}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>
+The Nix store accumulates entries which are no longer useful.<ref group="cf.">[https://nixos.org/nix/manual/#sec-garbage-collection Nix Manual, 11. Garbage Collection]</ref> They can be deleted with {{ic|nix-collect-garbage}} <ref group="cf.">{{man|nix-collect-garbage|sec=1}}</ref> or {{ic|nix-store --gc}}.<ref group="cf.">{{man|nix-store|sec=1}}, under {{ic|OPERATION --GC}}</ref>
 Note that if a result file still exists in the file system, and your Nix configuration has both <code>keep-outputs = true</code> and <code>keep-derivations = true</code>, all the dependencies used to build it will be kept. To see which result files prevent garbage collection, run:
@@ Line 50: / Line 59: @@
 GC roots can be found in <code>/nix/var/nix/gcroots</code>. The following script demonstrates how this directory can be used to (for example) query the state of manually made result symlinks:
-<syntaxhighlight lang="bash">
+<syntaxhighlight lang="console">
-find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo'
+$ find -H /nix/var/nix/gcroots/auto -type l | xargs -I {} sh -c 'readlink {}; realpath {}; echo'
 </syntaxhighlight>
 This acts a simpler (but faster) version of <code>--print-roots</code> and could be implemented as a bash alias for convenience.
-=== Run as root ===
+{{Tip|
 <code>nix-collect-garbage -d</code> operates only for the current user. To clear system profiles, run it with root privileges.
+}}
 === Look for <code>result</code> symlinks ===
@@ Line 87: / Line 97: @@
 Look for system derivations in particular. Those are created on many occasions, for example when running <code>nixos-rebuild  build-vm</code>
+=== Removing old generations ===
+NixOS keeps old configurations of your system around so that you can always rollback to a previous configuration if something goes wrong. You can also select which generation to boot into via GRUB.
+However these previous generations are GC roots that can keep around old, unnecessary software in your nix store. You can check what system generations you have with
+<syntaxhighlight lang="console">
+$ sudo nix-env -p /nix/var/nix/profiles/system --list-generations
+...
+   2021-09-04 02:56:54
+   2021-09-05 07:12:43
+   2021-09-05 22:12:13   (current)
+</syntaxhighlight>
+You can remove all but the current generation with
+<syntaxhighlight lang="console">
+$ sudo nix-collect-garbage -d
+...
+store paths deleted, 3467.28 MiB freed
+</syntaxhighlight>
+There are also user-specific generations for different things (eg. home-manager). These can be removed with
+<syntaxhighlight lang="console">
+$ nix-collect-garbage -d
+</syntaxhighlight>
 === Reboot ===
@@ Line 112: / Line 146: @@
 === Automation ===
-It is possible to enable periodic automatic GC,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example like this:
+Garbage collection can be automated,<ref group="cf.">{{nixos:option|nix.gc}}</ref> for example:
-<syntaxhighlight lang="nix">
+{{file|configuration.nix|nix|<nowiki>
 nix.gc = {
    automatic = true;
@@ Line 120: / Line 154: @@
    options = "--delete-older-than 30d";
 };
-</syntaxhighlight>
+</nowiki>}}
+If using nix-darwin, use this to run on 0th day of every week:
+{{file|configuration.nix|nix|<nowiki>
+nix.gc = {
+  automatic = true;
+  interval = { Weekday = 0; Hour = 0; Minute = 0; };
+  options = "--delete-older-than 30d";
+};
+</nowiki>}}
 This can result in redownloads (tarballs fetched with <code>import (builtins.fetchTarball ...)</code> for example are not referenced anywhere and removed on GC), but it frees you from runnning GC manually.
@@ Line 136: / Line 178: @@
 == Moving the store ==
-{{ic|/nix}} can reside on another device. This is useful if your root device is very small, and you have another, larger drive available.
+{{ic|/nix}} can reside on another device, which is useful if your root device is very small, and you have another, larger drive available.
-If the second mountpoint is on the same device, some benefit can still be gained by formatting the partition it points to with a different file system. For example: on a Raspberry Pi, f2fs could possibly be used for a gain in I/O throughput.
+If the new partition is on the same device, some benefit can be gained by formatting the partition on which {{ic|nix}} resides with a different file system. For example: on a Raspberry Pi, {{ic|f2fs}} could be used for a gain in I/O throughput.
 Regardless of <code>/nix</code>'s filesystem, it can also be mounted with <code>noatime</code> (as seen in the example below). This will reduce metadata writes, improving I/O and the device's lifespan.
@@ Line 147: / Line 189: @@
 # Create a new partition
-# Mount this new partition over <code>/mnt</code> <syntaxhighlight lang="bash">
+# Mount this new partition over <code>/mnt</code> <syntaxhighlight lang="console">
-mount /dev/disk/by-label/nix /mnt
+# mount -o defaults,noatime /dev/disk/by-label/nix /mnt/nix
 </syntaxhighlight>
-# Copy everything from <code>/nix</code> to <code>/mnt</code> ''Trailing slashes are important'', in that without them, <code>rsync</code> will create an additional directory of the same name at the destination.  <syntaxhighlight lang="bash">
+# Copy everything from <code>/nix</code> to <code>/mnt</code> ''Trailing slashes are important'', in that without them, <code>rsync</code> will create an additional directory of the same name at the destination. <syntaxhighlight lang="console">
-rsync --archive --acls --one-file-system --verbose /nix/store/ /mnt/store
+# rsync --archive --hard-links --acls --one-file-system --verbose /nix/{store,var} /mnt/nix
-rsync --archive --acls --one-file-system --verbose /nix/var/ /mnt/var
 </syntaxhighlight>
-# Use the new partition as new <code>/nix</code> <syntaxhighlight lang="bash">
+# Mount the new partition as the new <code>/nix</code> <syntaxhighlight lang="console">
-umount /mnt
+# umount /mnt/nix
-mount /dev/disk/by-label/nix /nix
+# mount /dev/disk/by-label/nix /nix
 </syntaxhighlight>
-# Restart nix daemon <syntaxhighlight lang="bash">
+# Restart {{ic|nix-daemon}} <syntaxhighlight lang="console">
-systemctl stop nix-daemon.service
+$ systemctl stop nix-daemon.service
-systemctl restart nix-daemon.socket
+$ systemctl restart nix-daemon.socket
-systemctl start nix-daemon.service
+$ systemctl start nix-daemon.service
 </syntaxhighlight>
-# Add mounting the <code>/nix</code> partition to your <code>/etc/nixos/configuration.nix</code> <syntaxhighlight lang="nix">
+# Add the new <code>/nix</code> partition to your <code>/etc/nixos/configuration.nix</code> <syntaxhighlight lang="nix">
 {
     # ...
@@ Line 172: / Line 213: @@
       options = [ "noatime" ];
     };
-   # ...
 }
 </syntaxhighlight>
-# Apply your configuration <syntaxhighlight lang="bash">
+# Apply your configuration <syntaxhighlight lang="console">
-nixos-rebuild switch
+# nixos-rebuild switch
 </syntaxhighlight>
 # Reboot to be sure <code>/nix/store</code> is properly mounted
 ''Optionally''
-# After reboot, check that <code>/nix</code> is mounted over your partition <syntaxhighlight lang="bash">
+# After reboot, check that <code>/nix</code> is mounted over your partition <syntaxhighlight lang="console">
-mount | grep "/nix" && echo "Nix seems to use your new partition" || echo "It seems that something bad happened"
+# mount | grep "/nix" && echo "Nix store is on a new partition" || echo "Nix is on the old partition"
 </syntaxhighlight>
-# Once '''you are sure''' everything works, you can delete the old store <syntaxhighlight lang="bash">
+# Once '''you are sure''' everything works, you can delete the old store <syntaxhighlight lang="console">
-mkdir /tmp/old_root
+# mkdir /tmp/old_root
-mount --bind / /tmp/old_root
+# mount --bind / /tmp/old_root
-rm --recursive /tmp/old_root/nix
+# rm --recursive /tmp/old_root/nix
-umount /tmp/old_root
+# umount /tmp/old_root
-rmdir /tmp/old_root
+# rmdir /tmp/old_root
 </syntaxhighlight>
-Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot!
+{{Tip|Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>. It is possible to get locked out of a system if one mistakenly mounted an empty drive to {{ic|nix}}. }}
 == See also ==
 <references group="cf."/>