Nix Cookbook: Difference between revisions

(16 intermediate revisions by 7 users not shown)

Line 4:

=== Reclaim space on Nix install? ===

{{META Box|TL&DR:|<code>nix-collect-garbage --delete-older-than 7d</code>}}

==== Remove old generations ====

When you make changes to your system, Nix creates a new system [[~~Generation~~]]. All of the changes to the system since the previous generation are stored there. Old generations can add up and will not be removed automatically by default. You can see your generations with:

When you make changes to your system, Nix creates a new system [[generation]]. All of the changes to the system since the previous generation are stored there. Old generations can add up and will not be removed automatically by default. You can see your saved system generations with:

<~~syntaxHighlight~~ lang=shell>

$ nix-env --list-generations

# nix-env --profile /nix/var/nix/profiles/system --list-generations

</~~syntaxHighlight~~>

</syntaxhighlight>

To keep just your current generation and the two older than it:

<~~syntaxHighlight~~ lang=shell>

$ nix-env --delete-generations +3

# nix-env --profile … --delete-generations +3

</~~syntaxHighlight~~>

</syntaxhighlight>

To remove all but your current generation:

<~~syntaxHighlight~~ lang=shell>

$ nix-env --delete-generations old

# nix-env --profile … --delete-generations old

</~~syntaxHighlight~~>

</syntaxhighlight>Apart from the system profile in <code>/nix/var/nix/profiles/system</code>, every user has profiles for their user environment and channels. The operations above may be repeated for those profiles. By default, they are located at <code>~/.nix-profile</code> and <code>~/.nix-defexpr/channels</code>. For more information on profile locations, see [https://nix.dev/manual/nix/stable/command-ref/nix-env#files Nix Manual - nix-env - Files].

===== Generation trimmer script =====

Line 56:

Line 57:

$ nix optimise~~-store~~

$ nix store optimise

</syntaxHighlight>

Line 75:

Line 76:

==== Nix manual references ====

* [https:/~~/nixos.org/manual~~/nix~~/stable/#idm140737322626416 Introduction - Garbage Collection]~~

* [https://nix.dev/manual/nix/stable/quick-start Nix Manual - Quick Start]

* [https://nixos.~~org~~/manual/nix/stable/~~#chap-~~quick-start Quick Start]

* [https://nix.dev/manual/nix/stable/package-management/garbage-collection.html Nix Manual - Garbage Collection]

* [https://~~nixos~~.~~org~~/manual/nix/~~#sec~~-garbage-collection Garbage Collection ~~Chapter~~]

* [https://nix.dev/manual/nix/stable/command-ref/nix-store/optimise Nix Manual - nix-store --optimize]

* [https://~~nixos~~.~~org~~/manual/nix/stable/~~#ssec~~-gc-~~roots Garbage Collector Roots]~~

* [https://nixos.org/manual/nixos/#sec-nix-gc NixOS Manual - Cleaning the Nix store]

* [https://nixos.org/manual/nix~~/stable/#operation~~-~~optimise Optimize~~ store]

* [https://nixos.org/manual/nixos/#sec-nix-gc NixOS ~~manual~~ - Cleaning Nix store]

==== Deeper cleaning ====

Line 86:

Line 85:

* [[Storage optimization]] goes into more depth on these options

* [[Cleaning the nix store]] has more specialized tips and further links to helper tools.

== Environment tasks ==

Line 128:

Line 126:

</syntaxHighlight>

== Deprecating a specific input parameter in mkDerivation-style packages ==

Sometimes we want to rename some input parameter in .

E.G. an option `withX` that enables the X11 GUI for a certain app:

{

/*. . .*/

withX ? true,

/*. . .*/

}:

stdenv.mkDerivation { /* . . . */}

</syntaxHighlight>

Suppose that a new version of this package features a more agnostic GUI that can be linked to X11, GTK, Qt etc.

Because of it, `withX` is no longer a descriptive name for this functionality.

However, renaming the parameter is dangerous, because other functions that call this function expect this parameter.

The problem becomes more pronounced when in conjunction with custom, third-party overlays.

The solution is, roughly, to emit a warning about the old parameter being used, reporting the user to the new parameter:

{

/*. . .*/

withX ? null,

withGui ?

if (withX != null) then

lib.warn ''

withX is deprecated and will be removed in the next release;

use withGui instead.

'' withX

else

true

/*. . .*/

}

</syntaxHighlight>

With this warning, consumers will have time to patch their codes.

== Bulk pre-download all dependencies of a package ==

Sometimes we need to download all source dependencies of a package.

E.G. a long build is being planned, so we first download all needed files, so that after that we only need to worry about local (non-Internet) issues.

Here is a one-liner for downloading all the source dependencies of a package (thanks Eelco Dolstra!):

$> nix-store -r $(grep -l outputHash $(nix-store -qR $(nix-instantiate '<nixpkgs>' -A bochs) | grep '.drv$'))

</syntaxHighlight>

Let's dissect this:

## instantiate bochs into `.drv` files and print the filenames;

instantiate=$(nix-instantiate '<nixpkgs>' -A bochs)

## print all references/requirements, filtering the .drv files (which is where static derivations live)

requirements=$(nix-store -qR $instantiate | grep '.drv$')

## keep only the source derivations, since those will have a predefined hash of the output

sources=$(grep -l outputHash $requirements)

## realize those derivations, downloading all sources and storing them in the nix store

nix-store -r $sources

</syntaxhighlight>

After that, all sources will be locally stored!

Source: [https://web.archive.org/web/20160829181620/http://lists.science.uu.nl/pipermail/nix-dev/2013-January/010438.html nix-dev thread]

== Wrapping packages ==

Line 217:

Line 290:

==== Bad configuration option: gssapikexalgorithms ====

Found when using an SSH binary from Nix on typically RPM-based distros like CentOS, Fedora, Scientific Linux, Redhat, etc. '''The quick fix:''' Just comment out the configuration option in the ssh config file, you ~~probably~~ don't need it.

Found when using an SSH binary from Nix on typically RPM-based distros like CentOS, Fedora, Scientific Linux, Redhat, etc. Possible fixes, from least to most invasive:

# '''The quick fix:''' Just comment out the configuration option in the ssh config file

# If you want to keep the option in but don't need it to work (e.g., you're sharing a config across systems, but only use GSSAPI/Kerberos on another system): add {{ic|IgnoreUnknown GSSAPI*}} to your ssh configuration

# Install the {{ic|openssh_gssapi}} package instead of {{ic|openssh}}. This will fix ssh used directly, but some dependencies may still use the non-GSSAPI package.

# Force specific other packages to build with the GSSAPI version: for example, you might add <code>(git-repo.override { openssh = openssh_gssapi; })</code> to your {{ic|environment.systemPackages}} list (if git-repo is the problematic package), or use [[overlays]] like: <syntaxhighlight lang="nix">

nixpkgs.overlays = [

(final: prev: {

mosh = prev.mosh.override { openssh = prev.openssh_gssapi; };

})

];

</syntaxhighlight>(which will fix [[mosh]] used as a dependency too)

# Force all packages that depend on openssh to use openssh_gssapi instead: <syntaxhighlight lang="nix">

nixpkgs.overlays = [

(final: prev: { openssh = prev.openssh_gssapi; } )

];

</syntaxhighlight>

==== Desktop environment does not find .desktop files ====

Line 252:

Line 341:

which will override the other value, and force yours to have priority.

== Auditing ==

=== License stance ===

Example on how to check if a given list of packages (as returned by the ''pkgs.nix'' derivation) conforms to permitted licenses criteria:

with rec {

# Incomplete list, customize to your policies.

permissiveLicense = v: v.license == "bsd3" || v.license == "mit" || v.license == "bsd2" || v.license == "publicDomain" || v.license == "asl20" || v.license == "zlib" || v.license == "bsdOriginal" || v.license == "openssl";

# Omit some false-positive buildInputs like bash and perl.. those should be nativeBuildInputs rather?

saneDep = d: d ? meta.license

&& builtins.substring 0 5 d.name != "bash-"

&& builtins.substring 0 5 d.name != "perl-";

# Keep if the license is not allowed, or if has any (transitive) dep with a license that is not allowed.

keepBadDeps = ds: builtins.filter (n: !(permissiveLicense n) || n.baddeps != []) (map derivToNode (builtins.filter saneDep ds));

derivToNode = d:

{ license = if builtins.typeOf d.meta.license == "string"

then d.meta.license

else if builtins.typeOf d.meta.license == "list" # can happen sometimes, could concat.. but have a look rather

then "MULTI"

else d.meta.license.shortName;

name = d.name;

baddeps = keepBadDeps (builtins.filter saneDep d.buildInputs);

};

let ps = import ./pkgs.nix; # pkgs.nix should result in a list of derivations to check

in keepBadDeps ps

</syntaxHighlight>

Then exercise it in '''nix repl''', using :p to force the result so we can actually see it:

nix-repl> xs = import ./lic.nix

nix-repl> :p xs

</syntaxHighlight>

This will print a (somewhat unreadable) nested tree of derivation names and their licences, where (at least) at the roots there are not-allowed licenses.

Be sure to manually check them for being false positives - navigate to the derivation in the nixpkgs repo and eyeball the license info (it is updated every now and then), also cross-check with the original source to make sure.

=== Vulnerabilities ===

See [https://github.com/nix-community/vulnix vulnix].

[[Category:nix]]

[[Category:Cookbook]]

@@ Line 4: / Line 4: @@
 === Reclaim space on Nix install? ===
+{{META Box|TL&DR:|<code>nix-collect-garbage --delete-older-than 7d</code>}}
 ==== Remove old generations ====
-When you make changes to your system, Nix creates a new system [[Generation]]. All of the changes to the system since the previous generation are stored there. Old generations can add up and will not be removed automatically by default. You can see your generations with:
+When you make changes to your system, Nix creates a new system [[generation]]. All of the changes to the system since the previous generation are stored there. Old generations can add up and will not be removed automatically by default. You can see your saved system generations with:
-<syntaxHighlight lang=shell>
+<syntaxhighlight lang="shell-session">
-   $ nix-env --list-generations
+   # nix-env --profile /nix/var/nix/profiles/system --list-generations
-</syntaxHighlight>
+</syntaxhighlight>
 To keep just your current generation and the two older than it:
-<syntaxHighlight lang=shell>
+<syntaxhighlight lang="shell-session">
-   $ nix-env --delete-generations +3
+   # nix-env --profile … --delete-generations +3
-</syntaxHighlight>
+</syntaxhighlight>
 To remove all but your current generation:
-<syntaxHighlight lang=shell>
+<syntaxhighlight lang="shell-session">
-   $ nix-env --delete-generations old
+   # nix-env --profile … --delete-generations old
-</syntaxHighlight>
+</syntaxhighlight>Apart from the system profile in <code>/nix/var/nix/profiles/system</code>, every user has profiles for their user environment and channels. The operations above may be repeated for those profiles. By default, they are located at <code>~/.nix-profile</code> and <code>~/.nix-defexpr/channels</code>. For more information on profile locations, see [https://nix.dev/manual/nix/stable/command-ref/nix-env#files Nix Manual - nix-env - Files].
 ===== Generation trimmer script =====
@@ Line 56: / Line 57: @@
 <syntaxHighlight lang=shell>
-   $ nix optimise-store
+   $ nix store optimise
 </syntaxHighlight>
@@ Line 75: / Line 76: @@
 ==== Nix manual references ====
-* [https://nixos.org/manual/nix/stable/#idm140737322626416 Introduction - Garbage Collection]
+* [https://nix.dev/manual/nix/stable/quick-start Nix Manual - Quick Start]
-* [https://nixos.org/manual/nix/stable/#chap-quick-start Quick Start]
+* [https://nix.dev/manual/nix/stable/package-management/garbage-collection.html Nix Manual - Garbage Collection]
-* [https://nixos.org/manual/nix/#sec-garbage-collection Garbage Collection Chapter]
+* [https://nix.dev/manual/nix/stable/command-ref/nix-store/optimise Nix Manual - nix-store --optimize]
-* [https://nixos.org/manual/nix/stable/#ssec-gc-roots Garbage Collector Roots]
+* [https://nixos.org/manual/nixos/#sec-nix-gc NixOS Manual - Cleaning the Nix store]
-* [https://nixos.org/manual/nix/stable/#operation-optimise Optimize store]
-* [https://nixos.org/manual/nixos/#sec-nix-gc NixOS manual - Cleaning Nix store]
 ==== Deeper cleaning ====
@@ Line 86: / Line 85: @@
 * [[Storage optimization]] goes into more depth on these options
 * [[Cleaning the nix store]] has more specialized tips and further links to helper tools.
 == Environment tasks ==
@@ Line 128: / Line 126: @@
 </syntaxHighlight>
+== Deprecating a specific input parameter in mkDerivation-style packages ==
+Sometimes we want to rename some input parameter in .
+E.G. an option `withX` that enables the X11 GUI for a certain app:
+<syntaxHighlight lang="nix">
+{
+  /*. . .*/
+  withX ? true,
+  /*. . .*/
+}:
+stdenv.mkDerivation { /* . . . */}
+</syntaxHighlight>
+Suppose that a new version of this package features a more agnostic GUI that can be linked to X11, GTK, Qt etc.
+Because of it, `withX` is no longer a descriptive name for this functionality.
+However, renaming the parameter is dangerous, because other functions that call this function expect this parameter.
+The problem becomes more pronounced when in conjunction with custom, third-party overlays.
+The solution is, roughly, to emit a warning about the old parameter being used, reporting the user to the new parameter:
+<syntaxHighlight lang="nix">
+{
+  /*. . .*/
+  withX ? null,
+  withGui ?
+  if (withX != null) then
+    lib.warn ''
+      withX is deprecated and will be removed in the next release;
+      use withGui instead.
+    '' withX
+  else
+    true
+  /*. . .*/
+}
+</syntaxHighlight>
+With this warning, consumers will have time to patch their codes.
+== Bulk pre-download all dependencies of a package ==
+Sometimes we need to download all source dependencies of a package.
+E.G. a long build is being planned, so we first download all needed files, so that after that we only need to worry about local (non-Internet) issues.
+Here is a one-liner for downloading all the source dependencies of a package (thanks Eelco Dolstra!):
+<syntaxHighlight lang="shell">
+$> nix-store -r $(grep -l outputHash $(nix-store -qR $(nix-instantiate '<nixpkgs>' -A bochs) | grep '.drv$'))
+</syntaxHighlight>
+Let's dissect this:
+<syntaxhighlight lang="sh" line="1">
+## instantiate bochs into `.drv` files and print the filenames;
+instantiate=$(nix-instantiate '<nixpkgs>' -A bochs)
+## print all references/requirements, filtering the .drv files (which is where static derivations live)
+requirements=$(nix-store -qR $instantiate | grep '.drv$')
+## keep only the source derivations, since those will have a predefined hash of the output
+sources=$(grep -l outputHash $requirements)
+## realize those derivations, downloading all sources and storing them in the nix store
+nix-store -r $sources
+</syntaxhighlight>
+After that, all sources will be locally stored!
+Source: [https://web.archive.org/web/20160829181620/http://lists.science.uu.nl/pipermail/nix-dev/2013-January/010438.html nix-dev thread]
 == Wrapping packages ==
@@ Line 217: / Line 290: @@
 ==== Bad configuration option: gssapikexalgorithms ====
-Found when using an SSH binary from Nix on typically RPM-based distros like CentOS, Fedora, Scientific Linux, Redhat, etc. '''The quick fix:''' Just comment out the configuration option in the ssh config file, you probably don't need it.
+Found when using an SSH binary from Nix on typically RPM-based distros like CentOS, Fedora, Scientific Linux, Redhat, etc. Possible fixes, from least to most invasive:
+# '''The quick fix:''' Just comment out the configuration option in the ssh config file
+# If you want to keep the option in but don't need it to work (e.g., you're sharing a config across systems, but only use GSSAPI/Kerberos on another system): add {{ic|IgnoreUnknown GSSAPI*}} to your ssh configuration
+# Install the {{ic|openssh_gssapi}} package instead of {{ic|openssh}}. This will fix ssh used directly, but some dependencies may still use the non-GSSAPI package.
+# Force specific other packages to build with the GSSAPI version: for example, you might add <code>(git-repo.override { openssh = openssh_gssapi; })</code> to your {{ic|environment.systemPackages}} list (if git-repo is the problematic package), or use [[overlays]] like: <syntaxhighlight lang="nix">
+  nixpkgs.overlays = [
+    (final: prev: {
+      mosh = prev.mosh.override { openssh = prev.openssh_gssapi; };
+    })
+  ];
+</syntaxhighlight>(which will fix [[mosh]] used as a dependency too)
+# Force all packages that depend on openssh to use openssh_gssapi instead: <syntaxhighlight lang="nix">
+ nixpkgs.overlays = [
+   (final: prev: { openssh = prev.openssh_gssapi; } )
+ ];
+</syntaxhighlight>
 ==== Desktop environment does not find .desktop files ====
@@ Line 252: / Line 341: @@
 which will override the other value, and force yours to have priority.
+== Auditing ==
+=== License stance ===
+Example on how to check if a given list of packages (as returned by the ''pkgs.nix'' derivation) conforms to permitted licenses criteria:
+<syntaxHighlight lang="nix">
+with rec {
+  # Incomplete list, customize to your policies.
+  permissiveLicense = v: v.license == "bsd3" || v.license == "mit" || v.license == "bsd2" || v.license == "publicDomain" || v.license == "asl20" || v.license == "zlib" || v.license == "bsdOriginal" || v.license == "openssl";
+  # Omit some false-positive buildInputs like bash and perl.. those should be nativeBuildInputs rather?
+  saneDep = d: d ? meta.license
+      && builtins.substring 0 5 d.name != "bash-"
+      && builtins.substring 0 5 d.name != "perl-";
+  # Keep if the license is not allowed, or if has any (transitive) dep with a license that is not allowed.
+  keepBadDeps = ds: builtins.filter (n: !(permissiveLicense n) || n.baddeps != []) (map derivToNode (builtins.filter saneDep ds));
+  derivToNode = d:
+    { license = if builtins.typeOf d.meta.license == "string"
+                then d.meta.license
+                else if builtins.typeOf d.meta.license == "list"  # can happen sometimes, could concat.. but have a look rather
+                     then "MULTI"
+                     else d.meta.license.shortName;
+      name = d.name;
+      baddeps = keepBadDeps (builtins.filter saneDep d.buildInputs);
+    };
+};
+let ps = import ./pkgs.nix;  # pkgs.nix should result in a list of derivations to check
+in keepBadDeps ps
+</syntaxHighlight>
+Then exercise it in '''nix repl''', using :p to force the result so we can actually see it:
+<syntaxHighlight lang="nix">
+nix-repl> xs = import ./lic.nix
+nix-repl> :p xs
+</syntaxHighlight>
+This will print a (somewhat unreadable) nested tree of derivation names and their licences, where (at least) at the roots there are not-allowed licenses.
+Be sure to manually check them for being false positives - navigate to the derivation in the nixpkgs repo and eyeball the license info (it is updated every now and then), also cross-check with the original source to make sure.
+=== Vulnerabilities ===
+See [https://github.com/nix-community/vulnix vulnix].
+[[Category:nix]]
+[[Category:Cookbook]]