Official NixOS Wiki

Nixpkgs/Patching Nixpkgs: Difference between revisions

← Older edit
Perchun (talk | contribs)
4 edits
VisualWikitext
N0099 (talk | contribs)
3 edits
replace importing nixpkgs: https://discourse.nixos.org/t/1000-instances-of-nixpkgs/17347 with existing NixOS module param `pkgs`, and replace fetchurl with fetchpatch2 that will do normalization on generated patch
Perchun (talk | contribs)
4 edits
m nixpkgs -> Nixpkgs
 
(One intermediate revision by the same user not shown)
Line 2: Line 2:
existing PR that has yet to be merged, and so want to leverage those changes prior to them being merged.
existing PR that has yet to be merged, and so want to leverage those changes prior to them being merged.


For the sake of example, let's say you are using the software package Ghidra, and the latest version available on
There are many ways to patch Nixpkgs, each with its own advantages and disadvantages. We will go through the most popular ones:
Nixpkgs unstable is [https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=ghidra Ghidra 11.1.2].
{| class="wikitable" style="text-align: center; width: 500px;"
You know that the Ghidra developers have recently released 11.2, and want to use it before it's in Nixpkgs. You could try to create an overlay and manually update the package, but maybe in the process you realize it is not a
|-
! Name            !! Method          !! IFD !! Requires Flakes
|-
| applyPatch      || .patch files    ||  X  ||
|-
| nixpkgs-patcher || .patch files    ||  X  ||  X
|-
| nix-patcher    || .patch files    ||    ||  X
|-
| gh-cherry-pick  || git cherry-pick ||    ||
|}
 
Most of the solutions work on the <code>.patch</code> files, but those have a huge disadvantage - they quickly go out of date and have to be constantly rebased. This is especially true for solutions that use Git to apply patches. Another disadvantage may be [https://nixos.org/manual/nix/unstable/language/import-from-derivation IFD], which may result in [https://nixcademy.com/posts/what-is-ifd-ups-and-downs/ performance and runtime issues].
 
== Usage ==
 
For the sake of example, let's say you are using the software package [https://ghidra-sre.org/ Ghidra], and the latest version available on
Nixpkgs unstable is [https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=ghidra 11.1.2].
You know that the Ghidra developers have recently released 11.2, and want to use it before it's in Nixpkgs. You could try to create an overlay and manually update the package, but maybe in the process, you realize it is not a
trivial update. You may find there is already a pending PR to Nixpkgs with the updated version and
trivial update. You may find there is already a pending PR to Nixpkgs with the updated version and
changes you want, such as [https://github.com/NixOS/nixpkgs/pull/344917 this] PR introducing Ghidra 11.2. If you didn't
changes you want, such as [https://github.com/NixOS/nixpkgs/pull/344917 this PR]. If you didn't
want to wait for this PR to be merged into Nixpkgs, you could apply the PR patch directly to your Nixpkgs instead. Which we will do below.
want to wait for that PR to be merged into Nixpkgs, you could apply the PR patch directly to your Nixpkgs instead. There are, of course, other reasons you may wish to use a patched Nixpkgs, and the method applies to any of those cases.


There are of course other reasons you may wish to use a patched Nixpkgs and the method applies to any of those cases.
To get your patch, you will usually need to find a list of commits from the desired PR and fetch them. Try to avoid fetching something like https://github.com/NixOS/nixpkgs/pull/344917.diff, as it can change if the PR is still open. If you are fetching from GitHub, don't forget to add <code>?full_index=1</code> at the end of the URL for better reproducibility.


<syntaxhighlight lang="nix">
# for https://github.com/NixOS/nixpkgs/pull/344917
pkgs.fetchpatch2 {
  # note the `.diff` at the end!
  url = "https://github.com/NixOS/nixpkgs/commit/55367b381af23045c93f7b85171db850e628ef7d.diff?full_index=1";
  hash = "sha256-/XnEHRQahmHUNstgTVkQC+LGp7Z0wmf+qPVyKntgZG0=";
}
</syntaxhighlight>


First you will need to apply the Nixpkgs PR patch to some copy of Nixpkgs. In this example we will assume a NixOS modules has parameter <code>pkgs</code> pointing to <code>nixos-unstable</code>.
Or you can also fetch a range, if you want to fetch multiple commits in one go
We will use the [https://noogle.dev/f/pkgs/applyPatches <code>applyPatches</code>] function to apply the unmerged PR changes to
<code>nixpkgs-unstable</code>. First we need to covert the PR to a patch, which can be done by appending
<code>.patch</code> to the github PR
url. This will redirect to another link, for example https://github.com/NixOS/nixpkgs/pull/344917.patch will
redirect to https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/344917.patch. Then we can use <code>[https://nixos.org/manual/nixpkgs/stable/#fetchpatch fetchpatch]</code> or <code>[https://noogle.dev/f/pkgs/fetchpatch2 fetchpatch2]</code> to normalize this patch that generated by GitHub:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ pkgs, ... }:
pkgs.fetchpatch2 {
  # 4c030cf is a parent of the first PR commit
  url = "https://github.com/NixOS/nixpkgs/compare/4c030cf309bffa9cd87336705e96ce941ce977d9...05071d58a87c832f6366fff6c8ec328cb28a2a66.diff?full_index=1";
  hash = "sha256-gH8NHJ+i1snTN/ehFopQko8BfeO0PxQaFRCQCoLzVfg=";
}
</syntaxhighlight>
 
=== Using <code>[https://github.com/NixOS/nixpkgs/blob/f6e07eb0e1e42c38f8e0b6ffc221b7bf73bece4b/pkgs/build-support/trivial-builders/default.nix#L981 applyPatches]</code> function ===
 
<code>applyPatches</code> is a Nixpkgs function that applies a list of patches to a source directory. Internally, it just creates a new derivation via <code>stdenv.mkDerivation</code> and passes all of the patches to the usual <code>[https://github.com/NixOS/nixpkgs/blob/12c894cb74171bcec130756cc0f77720fca6587d/pkgs/stdenv/generic/setup.sh#L1371-L1404 patchPhase]</code>.


<syntaxhighlight lang="nix">
let
let
  # to access the `applyPatches` function, you will need
  # to import unpatched Nixpkgs first
  pkgs = import <nixpkgs> {
    # with flakes, you may also need to hardcode the system
    # system = "x86_64-linux";
  };
   nixpkgs-344917-drv =
   nixpkgs-344917-drv =
     pkgs.applyPatches {
     pkgs.applyPatches {
Line 28: Line 65:
       patches = [
       patches = [
         (pkgs.fetchpatch2 {
         (pkgs.fetchpatch2 {
           url = "https://github.com/NixOS/nixpkgs/pull/344917.patch";
           url = "https://github.com/NixOS/nixpkgs/compare/4c030cf309bffa9cd87336705e96ce941ce977d9...05071d58a87c832f6366fff6c8ec328cb28a2a66.diff?full_index=1";
           sha256 = "sha256-aws9J5ZNUyz4Z2RqPVEovBTNng4AdhzS03Bqg8jejWQ=";
           hash = "sha256-gH8NHJ+i1snTN/ehFopQko8BfeO0PxQaFRCQCoLzVfg=";
         })
         })
       ];
       ];
Line 35: Line 72:
   nixpkgs-344917 = import nixpkgs-344917-drv { inherit (pkgs.stdenv) system; };
   nixpkgs-344917 = import nixpkgs-344917-drv { inherit (pkgs.stdenv) system; };
in
in
# ...
nixpkgs-344917.ghidra.version # == 11.2 :tada:
</syntaxhighlight>
 
In the above example, we create a derivation with the patch applied, called <code>nixpkgs-344917-drv</code>. Pay attention, we take the <code>src</code> attribute to get back the original (unpatched) Nixpkgs. We then import that new derivation, which we assign to <code>nixpkgs-344917</code>. Now we can use <code>nixpkgs-344917</code> to access the Ghidra 11.2 package, as well as any other packages normally available in Nixpkgs.
 
=== Using [https://github.com/gepbird/nixpkgs-patcher nixpkgs-patcher] ===
 
nixpkgs-patcher is a nice wrapper around <code>[https://wiki.nixos.org/wiki/Nixpkgs/Patching_Nixpkgs#Using_applyPatches_function applyPatches]</code> which makes integration with NixOS flake's system definition very easy:
 
<syntaxhighlight lang="nix">
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-patcher.url = "github:gepbird/nixpkgs-patcher";
 
    nixpkgs-patch-ghidra-11-2 = {
      url = "https://github.com/NixOS/nixpkgs/commit/55367b381af23045c93f7b85171db850e628ef7d.diff?full_index=1";
      flake = false;
    };
    nixpkgs-patch-ghidra-11-2-ret-sync = {
      url = "https://github.com/NixOS/nixpkgs/commit/05071d58a87c832f6366fff6c8ec328cb28a2a66.diff?full_index=1";
      flake = false;
    };
  };
 
  outputs =
    { nixpkgs-patcher, ... }@inputs:
    {
      nixosConfigurations.yourHostname = nixpkgs-patcher.lib.nixosSystem {
        modules = [
          ./configuration.nix
          ./hardware-configuration.nix
        ];
        specialArgs = inputs;
      };
    };
}
</syntaxhighlight>
 
=== Using [https://github.com/katrinafyi/nix-patcher nix-patcher] ===
 
nix-patcher is a script that automatically gets a list of patches from your <code>flake.nix</code> and applies them to your Nixpkgs fork sequentially.
 
<syntaxhighlight lang="nix"># flake.nix
{
  inputs = {
    nixpkgs = "github:MyName/nixpkgs/patched-branch";
    nixpkgs-upstream = "github:NixOS/nixpkgs/nixos-unstable";
 
    nixpkgs-patch-10 = {
      url = "https://github.com/NixOS/nixpkgs/commit/55367b381af23045c93f7b85171db850e628ef7d.diff?full_index=1";
      flake = false;
    };
    nixpkgs-patch-20 = {
      url = "https://github.com/NixOS/nixpkgs/commit/05071d58a87c832f6366fff6c8ec328cb28a2a66.diff?full_index=1";
      flake = false;
    };
  };
}</syntaxhighlight>
 
nix-patcher will notice the special "-upstream" and "-patch-" suffixes and match these with "nixpkgs-upstream". When run, the repo and branch to update will be taken from the "nixpkgs" inputs. The upstream, patches, and fork inputs are linked together by their common prefix (here, "nixpkgs").
 
=== Using [https://github.com/PerchunPak/gh-cherry-pick gh-cherry-pick] ===
 
gh-cherry-pick interacts directly with the GitHub API to cherry-pick or merge branches. This tool is unique in that it doesn't depend on or use Nix; it is a generic script that fits perfectly for our use case.
 
<syntaxhighlight lang="bash">
# This tool requires a classic token with `repo` and `workflow` permissions
# or a fine-grained token with `contents` and `workflows` permissions
GITHUB_TOKEN=ghp_... gh-cherry-pick \
  --target MyOrg/nixpkgs@patched \
  `: # cherry-pick these commits` \
  NixOS/nixpkgs/3f5ba52cc4701bf341457dfe5f6cb58e0cbb7f83 \
  NixOS/nixpkgs/49ba75edefc8dc4fee45482f77a280ddd7121797 \
  `: # or merge the entire branch!` \
  Someone/nixpkgs@pr-branch
</syntaxhighlight>
</syntaxhighlight>


In the above example we create a derivation with the patch applied, called <code>nixpkgs-344917-drv</code>. Pay attention we take the <code>src</code> attribute to get back the original (unpatched) nixpkgs. We then import that new derivation which we assign to <code>nixpkgs-344917</code>. Now we can use <code>nixpkgs-344917</code> to access the
Very similar to [https://wiki.nixos.org/wiki/Nixpkgs/Patching_Nixpkgs#Using_nix-patcher nix-patcher], it maintains a fork of Nixpkgs and applies the patches on it sequentially. The main difference is that it doesn't operate on patch files, but runs <code>[https://git-scm.com/docs/git-cherry-pick git cherry-pick]</code> directly through GitHub API. This is a huge advantage, since Git can automatically rebase commits instead of forcing you to do that.
Ghidra 11.2 package, as well as any other packages normally available in nixpkgs.


== Resources ==
== Resources ==
Line 46: Line 157:


* [https://ertt.ca/nix/patch-nixpkgs/ Patching <nixpkgs>]
* [https://ertt.ca/nix/patch-nixpkgs/ Patching <nixpkgs>]
* [https://discourse.nixos.org/t/patch-nixpkgs-using-cherry-picks-without-local-clone/76925 gh-cherry-pick announcement (and discussion about other tools)]
Retrieved from "https://wiki.nixos.org/wiki/Nixpkgs/Patching_Nixpkgs"