Storage optimization

A problem with NixOS, I've run into multiple time, is lack of space on / device. Even if you are using Nix only occasionally, it is easy for /nix/store to go beyond 50G.

Here are generic notes on how to not run out of space too often. I assume you already know about nix-collect-garbage and do care about used disk space.

Split /nix to separate LVM volume

If your root device is fixed and small-sized, you should move /nix to separate device. I have it running on LVM. Best to design that out of starts, otherwise you have to: - first you should know, that commands mount and bash belong to /nix/store so never mount empty disk over /nix or /nix/store, you will be locked out until reboot - create new ext4 FS and mount it over /mnt - rsync -a everything from /nix to /mnt - bind mount /mnt over /nix and rerun nixos-rebuild switch with something like

fileSystems."/nix" = { device = "/dev/disk/by-label/nix"; options = [ "noatime" ]; };

• reboot to be sure /nix/store is properly mounted
• bind mount / to /old_root, and remove /old_root/nix

That's it! You moved /nix/store!

Enable periodic GC

Next you may enable periodic auto GC, like this:

  nix.gc.automatic = true;
  nix.gc.dates = "weekly";
  nix.gc.options = "--delete-older-than 30d";

In my case this results into some redownloads (if you ever use import (builtins.fetchFromTarball ...) all these fetched tarballs are not referenced anywhere and removed on GC), but overall it frees you from runnning GC manually often.

You will never be free of manual GC!

One of least mentioned problems with nix-collect-garbage -d is it doesn't collect all the garbage! The key to understand how's that, is to run command:

$ nix-store --gc --print-roots
/home/danbst/dev/test-shell/.shell.drv -> /nix/store/4diqwczyjipdqyi7aj34wfagblbhfjr9-nixops-1.4
/home/danbst/dev/test-shell/.shell.drv-2 -> /nix/store/62h3c4d6rdnlxichixqg8h9jxi8nhxk0-stdenv
/home/danbst/dev/test-shell/.shell.drv-2-doc -> /nix/store/14gnv1q1w0n9qwa3q23idsqvn51354y8-bash-4.3-p42-doc
/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/danbst/testing/.nix-gc-roots/shell.drv -> /nix/store/v3vqf48awjjzjivrx15kfqdh1d7cg4mq-sshpass-1.05
...
/home/danbst/testing/.nix-gc-roots/shell.drv-12 -> /nix/store/a2li4sl9pxh9aflqia2gp7w88ayvjwci-bash-4.3-p42
/home/danbst/testing/.nix-gc-roots/shell.drv-12-doc -> /nix/store/kcswyb1d8zimkym0pjfi2fj1dly1w34w-bash-4.3-p42-doc
/home/danbst/testing/.nix-gc-roots/shell.drv-12-info -> /nix/store/njb817fwiafswzwvj9skw7w7k6b3fnbi-bash-4.3-p42-info
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/analyt/system-3-link -> /nix/store/snrj72189wh9va23fawl3v80v92xnxlm-nixos-system-iron-16.03.1291.efe2d64
/nix/var/nix/profiles/per-container/d-live/system-6-link -> /nix/store/cp2c58hnczsjk5h69ksajq5xfhsyhl6v-nixos-system-iron-16.03.1299.a8e0739
/nix/var/nix/profiles/per-container/d-test/system-4-link -> /nix/store/n1w7ywjg65x8iimchznxcyygbgmyfh55-nixos-system-iron-16.03.1287.6ac7ffd
/nix/var/nix/profiles/per-container/dashboard/system-41-link -> /nix/store/7qk19pkwgq0h3a1q9dcql3nks40rr75s-nixos-system-iron-16.03.1340.5a090dd
...
/nix/var/nix/profiles/per-container/ttt/system-1-link -> /nix/store/1kj9qs5gl3421jlkl3jfc2kqdsl8akwr-nixos-system-ttt-16.03.977.1da05df
/nix/var/nix/profiles/per-user/danbst/channels-1-link -> /nix/store/s0qay9qyqrn92zayldbvvj3zrfcl7a72-user-environment
/nix/var/nix/profiles/per-user/danbst/profile-28-link -> /nix/store/69ds606146dqml04sm0fbpqwnv2w8i3q-user-environment
/nix/var/nix/profiles/per-user/ec2-user/profile-7-link -> /nix/store/y2hc7zsnkzys9ba6xaijvjhff03rcgpy-user-environment
/nix/var/nix/profiles/per-user/root/channels-4-link -> /nix/store/254b6pkhhnjywvj5c0lp2vdai8nz4p0g-user-environment
/nix/var/nix/profiles/system-398-link -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git
/run/booted-system -> /nix/store/8jkrl9jyq7hqxb6xpwcaghpdm26gq98j-nixos-system-iron-16.0916.09pre.custom
/run/current-system -> /nix/store/wmndyzzrbc9fyjw844jmvzwgwgcinq7s-nixos-system-iron-16.0916.09pre.custom

Don't forget to run nix-collect-garbage -d with sudo

Otherwise you are clearing only current user profile, not system-wide.

Look for result symlinks

If you use nix-build, but don't use --no-build-output, your FS will be filled with result symlinks to various derivations. In example above there are

/home/danbst/stack/new/website/server/result -> /nix/store/1jhmp6vl364p32r8bjigk65qh1xa562f-server-0.1.0.0
/home/ec2-user/result -> /nix/store/q35aq2sh5dbyka6g6f6qb7b8msxwds5m-nixos-system-iron-16.03.1299.a8e0739
/root/forkstat/result -> /nix/store/i5glmg3wk2a48x52rhd92zip1cmc0kq9-forkstat-git

How much do these (apparently) abandoned derivations take space?

$ du -sch $(nix-store -qR /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result)
...
3.4G    total

Wow, garbage? Not so much, but:

# rm /root/forkstat/result /home/ec2-user/result /home/danbst/stack/new/website/server/result
# nix-collect-garbage -d
...
690 store paths deleted, 1817.99 MiB freed

Especially look for system derivations. Those are created in many cases, for example, when running nixos-rebuild build-vm

Don't forget to reboot

As you see, the reference in /run/booted-system is a GC root, so it won't be cleared until reboot. If you don't want to reboot, just rm /run/booted-system that link and rerun sudo nix-collect-garbage.

Proper nix-shell GC pinning

When you invoke nix-shell with

  nix-shell --indirect --add-root $DIR/.nix-gc-roots/shell.drv ...

then you'll have a persistent environment which won't be garbage collected. It is useful when you don't want to spend time waiting for redownloads every time you enter the shell.

A little problem exists though. GC roots are numbered sequentially, so if you change shell.nix to contain less derivations, and name of last GC root will start with shell.drv-7, then shell.drv-{8,9,10,11,12}* will be dangling and unused. To overcome this problem you should remove GC roots dir periodically (or just before nix-shell)

Obviously, you should remove the GC roots directory for projects you don't plan to work on.

Last resort - --optimise

Run nix-store --optimise in case you really need some a bit of space. It is a lengthy operation, and after multiple-outputs PR landing, it became less useful.

# nix-store --optimise
985.41 MiB freed by hard-linking 251942 files