Nix Hash: Difference between revisions
imported>Sternenseemann (Add in depth comparison between nix hash{-path,-file}, nix to-{sri,base32,base16,base64} and nix-hash and correct mistake about nix-hash not supporting sha512) |
Artoria2e5 (talk | contribs) (→Updating Packages: Sentence case, demote to <h3>) |
||
| (19 intermediate revisions by 13 users not shown) | |||
| Line 1: | Line 1: | ||
[https://en.wikipedia.org/wiki/Cryptographic_hash_function Cryptographic hashes] play an essential role in a lot of places in the Nix ecosystem. When using a hash somewhere, two criteria are essential to do so properly: the '''algorithm''' used and the '''encoding''' (and, to some | [https://en.wikipedia.org/wiki/Cryptographic_hash_function Cryptographic hashes] play an essential role in a lot of places in the Nix ecosystem. When using a hash somewhere, two criteria are essential to do so properly: the '''algorithm''' used and the '''encoding''' (and, to some extent, ''what'' is hashed). | ||
Supported algorithms are <code>md5</code>, <code>sha1</code>, <code>sha256</code>, <code>sha512</code>. The first two are deprecated and should not be used anymore, but you may still stumble upon them in existing code. | Supported algorithms are <code>md5</code>, <code>sha1</code>, <code>sha256</code>, <code>sha512</code>. The first two are deprecated and should not be used anymore, but you may still stumble upon them in existing code. | ||
| Line 7: | Line 6: | ||
A hash – which is simply a sequence of bytes – is usually encoded in order to be representable as string. Common encodings are <code>base16</code> (commonly called "hex"), <code>base32</code> and <code>base64</code>. Note that the base32 is a [https://github.com/NixOS/nix/blob/master/src/libutil/hash.cc#L83-L107 '''custom one'''] that is not documented nor standardized in any way! If possible, use the provided hashing tools to convert hashes to it (see below). base32 is used by Nix in a lot of places because it is shorter than hex but can still safely be part of a file path (as it contains no slashes). | A hash – which is simply a sequence of bytes – is usually encoded in order to be representable as string. Common encodings are <code>base16</code> (commonly called "hex"), <code>base32</code> and <code>base64</code>. Note that the base32 is a [https://github.com/NixOS/nix/blob/master/src/libutil/hash.cc#L83-L107 '''custom one'''] that is not documented nor standardized in any way! If possible, use the provided hashing tools to convert hashes to it (see below). base32 is used by Nix in a lot of places because it is shorter than hex but can still safely be part of a file path (as it contains no slashes). | ||
== Usage == | |||
Many derivations are so-called ''fixed-output'' derivations, meaning that you need to know and specify the hash of the output in advance. As an example, let's look at <code>fetchurl</code>: | Many derivations are so-called ''fixed-output'' derivations, meaning that you need to know and specify the hash of the output in advance. As an example, let's look at nixpkgs function <code>fetchurl</code>: | ||
<syntaxHighlight lang=nix> | <syntaxHighlight lang=nix> | ||
src = fetchurl { | src = fetchurl { | ||
url = "https://example.org/downloads/source-code.zip"; | url = "https://example.org/downloads/source-code.zip"; | ||
sha256 = | hash = "sha256-IdU23rswdtT26QRL2e8VyMWLKfnL1K1AawWDEKVl3rw="; | ||
}; | }; | ||
</syntaxHighlight> | </syntaxHighlight> | ||
The format of the hash follows the [https://www.w3.org/TR/SRI/#introduction SRI (Subresource Integrity)] specification. | |||
=== Updating packages === | |||
[https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers-caveats Using TOFU to get the new hash] | |||
== What exactly is hashed == | |||
Some content can either be hashed "flat" or "recursively". "flat" (sometimes also called "file") is simply taking the hash of the file, byte by byte, and will give you the same result as for example `sha256sum -b myfile.zip`. "recursive" (or sometimes "path") hashing takes multiple files, path names and metadata (attributes) into consideration. It works by | Some content can either be hashed "flat" or "recursively". "flat" (sometimes also called "file") is simply taking the hash of the file, byte by byte, and will give you the same result as for example `sha256sum -b myfile.zip`. "recursive" (or sometimes "path") hashing takes multiple files, path names and metadata (attributes) into consideration. It works by <abbr title="nix archive (file format)">NAR</abbr>ing the input before hashing. | ||
For <code>fetchurl</code>, the option to switch between both is called <code>recursiveHash</code> and defaults to <code>false</code>. | For <code>fetchurl</code>, the option to switch between both is called <code>recursiveHash</code> and defaults to <code>false</code>. | ||
<code>fetchzip</code> on the other hand will download the file, unzip it and then recursively hash the output. There's no option. | <code>fetchzip</code> on the other hand will download the file, unzip it and then recursively hash the output. There's no option. The motivation behind this is that sometimes, the content is always the same, but the archive may change. This is because zip files are inherently non-deterministic, and might be generated automatically. If they are regenerated, they'll have a different hash, although the content is the same. <code>recursiveHash</code> works around that. | ||
The | |||
== Tools == | |||
* [https://nixos.org/manual/nix/stable/command-ref/nix-hash.html nix-hash] | |||
* [https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-hash nix hash] ([[nix command]]) | |||
When dealing with remote files, <code>nix-prefetch-url</code> offers a handy shortcut for downloading the file into the Nix store and printing out its hash. (<code>nix-prefetch-url --unpack</code> is its <code>fetchzip</code> equivalent.) | When dealing with remote files, <code>nix-prefetch-url</code> offers a handy shortcut for downloading the file into the Nix store and printing out its hash. (<code>nix-prefetch-url --unpack</code> is its <code>fetchzip</code> equivalent.) | ||
== Libraries == | |||
* [https://github.com/NixOS/nix/blob/master/src/libutil/hash.cc#L83 Original C++ implementation] | |||
* [https://github.com/kolloch/nix-base32 Rust implementation] | * [https://github.com/kolloch/nix-base32 Rust implementation] | ||
* [https://gist.github.com/colemickens/27ad622adfaf8c980f3b5066b21604ba Go implementation] | * [https://gist.github.com/colemickens/27ad622adfaf8c980f3b5066b21604ba Go implementation] | ||
* [https://github.com/dritter/nix-hasher/blob/master/src/Hasher.php#L18-L49 PHP Implementation] | |||
== Further reading == | == Further reading == | ||
* Eelco Dolstra's [https://edolstra.github.io/pubs/phd-thesis.pdf | * Eelco Dolstra's [https://edolstra.github.io/pubs/phd-thesis.pdf PhD thesis], section 5.1. | ||
* [https://github.com/NixOS/nix/issues/806 Github Issue] about which encoding is used where, and what pitfalls can arise from it. | * [https://github.com/NixOS/nix/issues/806 Github Issue] about which encoding is used where, and what pitfalls can arise from it. | ||
[[Category: Nix]] | |||
Latest revision as of 06:37, 15 May 2024
Cryptographic hashes play an essential role in a lot of places in the Nix ecosystem. When using a hash somewhere, two criteria are essential to do so properly: the algorithm used and the encoding (and, to some extent, what is hashed).
Supported algorithms are md5, sha1, sha256, sha512. The first two are deprecated and should not be used anymore, but you may still stumble upon them in existing code.
A hash – which is simply a sequence of bytes – is usually encoded in order to be representable as string. Common encodings are base16 (commonly called "hex"), base32 and base64. Note that the base32 is a custom one that is not documented nor standardized in any way! If possible, use the provided hashing tools to convert hashes to it (see below). base32 is used by Nix in a lot of places because it is shorter than hex but can still safely be part of a file path (as it contains no slashes).
Usage
Many derivations are so-called fixed-output derivations, meaning that you need to know and specify the hash of the output in advance. As an example, let's look at nixpkgs function fetchurl:
src = fetchurl {
url = "https://example.org/downloads/source-code.zip";
hash = "sha256-IdU23rswdtT26QRL2e8VyMWLKfnL1K1AawWDEKVl3rw=";
};
The format of the hash follows the SRI (Subresource Integrity) specification.
Updating packages
Using TOFU to get the new hash
What exactly is hashed
Some content can either be hashed "flat" or "recursively". "flat" (sometimes also called "file") is simply taking the hash of the file, byte by byte, and will give you the same result as for example `sha256sum -b myfile.zip`. "recursive" (or sometimes "path") hashing takes multiple files, path names and metadata (attributes) into consideration. It works by NARing the input before hashing.
For fetchurl, the option to switch between both is called recursiveHash and defaults to false.
fetchzip on the other hand will download the file, unzip it and then recursively hash the output. There's no option. The motivation behind this is that sometimes, the content is always the same, but the archive may change. This is because zip files are inherently non-deterministic, and might be generated automatically. If they are regenerated, they'll have a different hash, although the content is the same. recursiveHash works around that.
Tools
When dealing with remote files, nix-prefetch-url offers a handy shortcut for downloading the file into the Nix store and printing out its hash. (nix-prefetch-url --unpack is its fetchzip equivalent.)
Libraries
Further reading
- Eelco Dolstra's PhD thesis, section 5.1.
- Github Issue about which encoding is used where, and what pitfalls can arise from it.