Storage optimization: Difference between revisions

Line 1:

A problem with NixOS~~, I've run into multiple time,~~ is lack of space on <code>/</code> ~~device~~. Even if you are using Nix only occasionally, it is easy for <code>/nix/store</code> to go beyond 50G.

A recurring problem with NixOS is lack of space on <code>/</code>. Even if you are using Nix only occasionally, it is easy for <code>/nix/store</code> to go beyond 50G.

Here are generic notes on how to not run out of space too often. I assume you already know about <code>nix-collect-garbage</code> and do care about used disk space.

== Split <code>/nix</code> to separate LVM volume ==

== Split <code>/nix</code> to a separate LVM volume ==

If your root device is fixed and small-sized, you should move <code>/nix</code> to separate device. ~~I have~~ it ~~running~~ on LVM. ~~Best to design that out of starts~~, otherwise you have to: - first you should know, that commands <code>mount</code> and <code>bash</code> belong to <code>/nix/store</code> so never mount empty disk over <code>/nix</code> or <code>/nix/store</code>, you will be locked out until reboot - create new ext4 FS and mount it over <code>/mnt</code> - <code>rsync -a</code> everything from <code>/nix</code> to <code>/mnt</code> - bind mount <code>/mnt</code> over <code>/nix</code> and rerun <code>nixos-rebuild switch</code> with something like

If your root device is fixed and small-sized, you should move <code>/nix</code> to a separate device. For example, you can run it on LVM. Optimally, you would set this up when installing NixOS, otherwise you might have to go through these steps:

<syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };</syntaxhighlight>

# Create a new ext4 partition and mount it over <code>/mnt</code>

* reboot to be sure <code>/nix/store</code> is properly mounted

# <code>rsync -a</code> everything from <code>/nix</code> to <code>/mnt</code>

* bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>

# bind <code>/mnt</code> to <code>/nix</code> (e.g. using <code>mount</code>) and rerun <code>nixos-rebuild switch</code> with something like <syntaxhighlight lang="bash">fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };</syntaxhighlight>

# reboot to be sure <code>/nix/store</code> is properly mounted

# bind mount <code>/</code> to <code>/old_root</code>, and remove <code>/old_root/nix</code>

Keep in mind that all commands like <code>mount</code> and <code>bash</code> point to some executable in <code>/nix/store</code>, so never mount an empty disk over <code>/nix</code> or <code>/nix/store</code>, otherwise you will be locked out until reboot!

That's it! You moved <code>/nix/store</code>!

== Enable periodic GC ==

== Enable periodic Garbage Collection ==

Next you may enable periodic auto GC, like this:

Next you may enable periodic auto GC, for example like this:

Line 21:

Line 25:

nix.gc.dates = "weekly";

nix.gc.options = "--delete-older-than 30d";</syntaxhighlight>

~~In my case this results~~ into some redownloads (if you ever use <code>import (builtins.fetchFromTarball ...)</code> all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.

This can result into some redownloads (if you ever use <code>import (builtins.fetchFromTarball ...)</code> all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.

== You will never be free of manual GC! ==

One of least mentioned problems with <code>nix-collect-garbage -d</code> is it doesn't collect all the garbage! The key to understand how's that, is to run command:

One of least mentioned problems with <code>nix-collect-garbage -d</code> is it doesn't collect all the garbage! The key to understand how's that, is to run this command:

<syntaxhighlight lang="bash">$ nix-store --gc --print-roots

Line 52:

Line 56:

/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom

/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom</syntaxhighlight>

=== Don't forget to run <code>nix-collect-garbage -d</code> ~~with <code>sudo</code>~~ ===

=== Don't forget to run <code>nix-collect-garbage -d</code> as root ===

~~Otherwise~~ you are clearing only ~~current~~ user ~~profile~~, not system-wide.

Run <code>nix-collect-garbage -d</code> with <code>sudo</code> or <code>su</code>, or logged in as root, otherwise you are clearing only your old user profiles, not system-wide profiles.

=== Look for <code>result</code> symlinks ===

If you use <code>nix-build</code>, but don't use <code>--no-build-output</code>, your FS will be filled with <code>result</code> symlinks to various derivations. In example above ~~there are~~

If you use <code>nix-build</code>, but don't use <code>--no-build-output</code>, your FS will be filled with <code>result</code> symlinks to various derivations. In the example above, note the following symlinks:

<syntaxhighlight lang="bash">/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0

/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739

/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git</syntaxhighlight>

How much do these (apparently) abandoned derivations ~~take space~~?

How much space do these (apparently) abandoned derivations use?

<syntaxhighlight lang="bash">$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)

...

3.4G total</syntaxhighlight>

~~Wow,~~ garbage~~? Not so much~~, but:

Not all of the derivations are garbage in this case, but quite a few are:

<syntaxhighlight lang="bash"># rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result