Nixpkgs/Create and debug packages: Difference between revisions

From NixOS Wiki
imported>Fadenb
m Syntaxhighlight
Artturin (talk | contribs)
 
(46 intermediate revisions by 27 users not shown)
Line 1: Line 1:
This article describes how to work with the nix related repositories to add new packages, edit and debug existing packages. For details on the NixOS module system see [[NixOS:Modules]]. [[NixOS:extend_NixOS]] explains how to write, test and debug your own modules.
This article describes how to work with the nix related repositories to add new packages, edit and debug existing packages. For details on the NixOS module system see [[NixOS:Modules]]. [[NixOS:extend_NixOS]] explains how to write, test and debug your own modules.


There is a chapter about hacking packages and modules in the NixOS manual: http://nixos.org/nixos/manual/index.html#ch-development
There is a chapter about hacking packages and modules in the NixOS manual: [http://nixos.org/nixos/manual/index.html#ch-development development]


Writing packages is covered in http://nixos.org/nixpkgs/manual and writing modules in http://nixos.org/nixos/manual
Writing packages is covered in [https://nixos.org/manual/nixpkgs/stable/#chap-quick-start quick start] and writing modules is covered in the [http://nixos.org/nixos/manual manual]


The nix repositories are hosted here: https://github.com/nixos
If you've read the manual and still don't know how to go about creating a package, read on.
 
The nix repositories are hosted at https://github.com/NixOS




== Basics ==
== Basics ==
The code for nix packages is managed in the nixpkgs repository. NixOS services, and other system configuration options are managed in the nixos sub-directory of the nixpkgs repository.
The code for nix packages is managed in the [https://github.com/NixOS/nixpkgs/tree/master/pkgs nixpkgs/pkgs] repository. NixOS services, and other system configuration options are managed in [https://github.com/NixOS/nixpkgs/tree/master/nixos nixpkgs/nixos].


The steps to take for your first change should look something like this:
The steps to take for your first change should look something like this:


# Fork the repo (e.g. click the fork button on https://github.com/nixos/nixpkgs).
# Fork the repo (e.g. click the fork button on https://github.com/nixos/nixpkgs).
# Clone your fork <code>git clone git@github.com:YOUR-GITHUB-ACCOUNT-NAME/nixpkgs.git</code>
# Clone your fork <code><nowiki>git clone --depth 1 https://github.com/YOURNAME/nixpkgs.git</nowiki></code>
# Hack hack hack
# Hack hack hack
# Push your changes to your fork
# Push your changes to your fork
Line 21: Line 23:


This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)
This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)
== Rough process for creating a package ==
There are different steps here depending on whether you're building from source or packaging an existing binary.  There are some common steps too.
=== Package from source code ===
# Read the repo build instructions and CI scripts (for example, on GitHub, these are located in <code>.github/workflows</code>).
# Look in nixpkgs for a package with a similar build process to use as reference. For example, if you're packaging a project written in Go, find a package for an existing Go application. Each language has its own supporting Nix functions and a more or less standard way of dealing with things. For example, [https://nixos.org/manual/nixpkgs/stable/#sec-language-go Go] has <code>buildGoModule</code>. [https://nixos.org/manual/nixpkgs/stable/#rust Rust] has <code>buildRustPackage</code>. [https://nixos.org/manual/nixpkgs/stable/#python Python] has <code>buildPythonApplication</code>. [https://nixos.org/manual/nixpkgs/stable/#node.js Node.js] has <code>node2nix</code>, <code>yarn2nix</code>, etc. There are also specific functions for wrapping e.g. [https://nixos.org/manual/nixpkgs/stable/#sec-language-gnome GNOME] applications (<code>wrapGAppsHook</code>), or [https://nixos.org/manual/nixpkgs/stable/#sec-language-qt Qt] apps (<code>libsForQt5</code>, <code>wrapQtAppsHook</code>). Refer to the [https://nixos.org/manual/nixpkgs/stable/#chap-language-support language support chapter in the nixpkgs manual].
# If there isn't a specific builder for the language, use <code>stdenv.mkDerivation</code> directly, which has built-in support for GNU make (and other build systems, provided you add the necessary <code>nativeBuildInputs</code>).
# Figure out at least some dependencies from the project repo. See if they're available in nixpkgs (<code>nix search some-library</code> or <code>nix-locate --top-level lib/somelibrary.so</code>). If any dependency is missing you'll need to package that as well.
# Create your derivation in <code>default.nix</code> in some empty local directory.
# At the top of the derivation, temporarily add <code>with import <nixpkgs> {};</code>. For now, don't worry too much about declaring every dependency as a parameter<!-- TODO clarify. "parameter of the default build function"? -->, to save time.
# Build the package with <code>nix-build</code>. Iterate on tweaking the derivation and rebuilding until it succeeds.
# For large projects with long compile times, you can use <code>nix-shell</code> instead to [https://nixos.org/manual/nixpkgs/stable/#sec-building-stdenv-package-in-nix-shell run the individual phases].
# At this stage, you may encounter some build quirks of the project. Compile-time errors will hopefully explain what you're missing. For example [https://github.com/NixOS/nixpkgs/blob/643ce4bd0f057bc0b90f0faebeb83a3b14f01674/pkgs/tools/package-management/micromamba/default.nix#L6-L10 micromamba needs a specialized build of libsolv].
# Read on below for further steps.
=== Packages from binaries ===
# There's probably a package for it for some other distro. Use that package definition to figure out the dependencies. For example, if you have a deb package you can view its dependencies by running <code>dpkg -I <package.deb></code>. [https://aur.archlinux.org/packages/ Arch packages] can also be useful to look up for reference (view the package's PKGBUILD):
# Sometimes the definitions for other distros won't be enough by nix's standards. If that's the case, use <code>ldd</code> and/or <code>strace</code> to find the rest of the dependencies. If you're not familiar with <code>ldd</code>/<code>strace</code> see [https://unix.stackexchange.com/questions/120015/how-to-find-out-the-dynamic-libraries-executables-loads-when-run How to find out the dynamic libraries executables loads when run?]
# See how other nix binary packages deal with dependencies. For example [https://github.com/NixOS/nixpkgs/search?q=%22dpkg+-x%22 nix packages based on deb packages].
# If the application contains some helper executable or vendored dlopen'd library you will probably need to give the nix treatment to it as well. For example, [https://github.com/NixOS/nixpkgs/blob/42c154d332eb4eb17c74b587c1d4c2fcc3042ba1/pkgs/applications/editors/jetbrains/default.nix#L196-L200 JetBrains Rider vendors dotnet so it needs to be replaced with the dotnet nix package].
# Because there's no real build step here you'll have to rely more on testing the actual execution of the package.
=== Both source code packages and binary packages ===
# Once you have the package building successfully, test the output. Ensure the build completes using<code>nix-build</code>, then run <code>result/bin/&lt;executableName></code>. Test as much functionality of the application as you can to ensure that it works as intended.
# Now that your package builds and runs, it's time to move it to nixpkgs. Read [https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md nixpkgs/CONTRIBUTING.md], make sure your package is up to the standards e.g. add a suitable [https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes <code>meta</code> section].
# Git clone https://github.com/NixOS/nixpkgs, figure out the best category / directory for the application (within https://github.com/NixOS/nixpkgs/tree/master/pkgs/), create the directory for your application, and move your default.nix there.
# If you used <code>with import <nixpkgs> {};</code> to iterate more quickly, now is the time to replace that with the actual dependencies as an attribute set at the beginning of the file e.g. <code>{ lib, stdenv, fetchFromGitHub }:</code>
# Add the package to the top level declaration of packages. Most of the time this will be https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix .
# If this is your first package in nixpkgs, add yourself in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix in a separate commit.
# Read on about the final steps of branching and sending your PR in https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md .


== How to install from the local repository ==
== How to install from the local repository ==
For the sake of this article, let's set an environment variable which points to the directory where we've cloned our repo.
For expediency just for this article, we'll shallow clone direct from the distribution repo and set an environment variable pointing to it.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
$ export NIXPKGS=/path/to/clone/of/nixpkgs
$ mkdir -p ~/tmpdev && cd ~/tmpdev
$ git clone --depth=1 https://github.com/nixos/nixpkgs
$ export NIXPKGS=~/tmpdev/nixpkgs
$ ls $NIXPKGS
</syntaxhighlight>
</syntaxhighlight>


Line 32: Line 72:


'''example: list all available software''' from the local repository
'''example: list all available software''' from the local repository
$NIXREPOS/nixpkgs
$NIXPKGS
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ nix-env -f $NIXPKGS -qaP '*'
$ nix-env -f $NIXPKGS -qaP '*'
</syntaxhighlight>
</syntaxhighlight>


'''example: install software from local repository'''
'''example: update the system''' based on your local '''$NIXPKGS'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ nix-env -f $NIXPKGS -i python-urlgrabber
$ nixos-rebuild -I nixpkgs=$NIXPKGS switch
</syntaxhighlight>
 
'''example: update the system''' based on your local '''$NIXREPOS'''
<syntaxhighlight lang="bash">
$ nixos-rebuild -I nixos=$NIXPKGS/nixos -I nixpkgs=$NIXPKGS switch
</syntaxhighlight>
</syntaxhighlight>


'''example: build an expression and put the output in to `pwd`/results'''
'''example: build an expression and put the output in to `pwd`/results'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ nix-build $NIXPKGS -A irssi
$ nix-build $NIXPKGS -A irssi
</syntaxhighlight>
</syntaxhighlight>


'''example: get an environment which is used to build irssi (also see nix-shell)'''
'''example: get an environment which is used to build irssi (also see nix-shell)'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ nix-build $NIXPKGS --run-env -A irssi
$ nix-build $NIXPKGS --run-env -A irssi
</syntaxhighlight>
</syntaxhighlight>


'''example: get a persistent environment which is used to build irssi'''
'''example: get a persistent environment which is used to build irssi'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ nix-build $NIXPKGS --run-env -A irssi --add-root
$ nix-build $NIXPKGS --run-env -A irssi --add-root
</syntaxhighlight>
</syntaxhighlight>
Line 65: Line 100:
You have forked the relevant nix repository, but you will want to track changes in the upstream nix repo too. You can add a remote, and a corresponding branch for this.
You have forked the relevant nix repository, but you will want to track changes in the upstream nix repo too. You can add a remote, and a corresponding branch for this.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
git remote add upstream https://github.com/NixOS/nixpkgs.git
$ git remote add upstream https://github.com/NixOS/nixpkgs.git
</syntaxhighlight>
</syntaxhighlight>


You can create a branch to track the upstream master branch:
You can create a branch to track the upstream master branch:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="console">
$ git fetch upstream
$ git checkout -b upstream-master upstream/master
$ git checkout -b upstream-master upstream/master
$ git pull
$ git pull
Line 78: Line 114:
This will put you into a branch with all the latest changes. Hydra, the build farm, regularly creates binaries, but, since people are constantly contributing to the nix repositories, it is usually the case that there are changes in the master branch which have not yet made it into the binary channel. To take advantage of available binaries you can switch to the revision which produced the binaries in your current system and apply your changes from there. You can use `nixos-version` to see the relevant short revision hash:
This will put you into a branch with all the latest changes. Hydra, the build farm, regularly creates binaries, but, since people are constantly contributing to the nix repositories, it is usually the case that there are changes in the master branch which have not yet made it into the binary channel. To take advantage of available binaries you can switch to the revision which produced the binaries in your current system and apply your changes from there. You can use `nixos-version` to see the relevant short revision hash:


  $ nixos-version  
<syntaxhighlight lang="console">
  14.11pre52727.5d97886 (Caterpillar)
$ nixos-version  
  ${NixOS release}.${nixpkgs revision}  
14.11pre52727.5d97886 (Caterpillar)
(since the git-repo called nixos was merged into nixpkgs)
${NixOS release}.${nixpkgs revision}  
 
(since the git-repo called nixos was merged into nixpkgs)
</syntaxhighlight>
    
    
 
<syntaxhighlight lang="console">
  $ nixos-version  
$ nixos-version  
  13.07pre4871_18de9f6-3c35dae (Aardvark)
13.07pre4871_18de9f6-3c35dae (Aardvark)
  ${NixOS release}_${NixOS revision}-${nixpkgs revision}
${NixOS release}_${NixOS revision}-${nixpkgs revision}
</syntaxhighlight>


This string shows the Nixos release number (13.07pre4871) followed by the nixos revision used to produce your current system (18de9f6) followed by the nixpkgs revision (3c35dae).
This string shows the Nixos release number (13.07pre4871) followed by the nixos revision used to produce your current system (18de9f6) followed by the nixpkgs revision (3c35dae).
 
<syntaxhighlight lang="console">
  $ git branch
$ git branch
  upstream-master
upstream-master
  $ git checkout -b nixpkgs-channel 3c35dae
$ git checkout -b nixpkgs-channel 3c35dae
  Switched to a new branch 'nixpkgs-channel'
Switched to a new branch 'nixpkgs-channel'
  $ git checkout -b my-new-pkg
$ git checkout -b my-new-pkg
  Switched to a new branch 'my-new-pkg'
Switched to a new branch 'my-new-pkg'
</syntaxhighlight>


After making some changes you can commit them into your local repo:
After making some changes you can commit them into your local repo:
  $ git add foo
<syntaxhighlight lang="console">
  $ git commit
$ git add foo
$ git commit
</syntaxhighlight>


Then you push your changes to your fork:
Then you push your changes to your fork:
 
<syntaxhighlight lang="console">
  $ git push origin my-new-pkg
$ git push origin my-new-pkg
</syntaxhighlight>


You can use this to open a pull request on github.
You can use this to open a pull request on github.


If some time has passed since you have created your fork, you will want to merge your changes with upstream and test that it still works.
If some time has passed since you have created your fork, you will want to merge your changes with upstream and test that it still works.
  git fetch upstream
<syntaxhighlight lang="console">
  git merge upstream
$ git fetch upstream
$ git merge upstream
</syntaxhighlight>


If your merge then fails because someone else has made the same change (for example, someone else also packaged a library you have just packed for the program you want to get into nixpkgs), then you can do this:
If your merge then fails because someone else has made the same change (for example, someone else also packaged a library you have just packed for the program you want to get into nixpkgs), then you can do this:


  git rebase -i HEAD~10
<syntaxhighlight lang="console">
$ git rebase -i HEAD~10
</syntaxhighlight>


there select the edit mode for your commit and remove the your code which added the library. **Warning: only use 'git rebase' on your commits, which have not been pushed and nobody else is working with already!**
there select the edit mode for your commit and remove the your code which added the library. **Warning: only use 'git rebase' on your commits, which have not been pushed and nobody else is working with already!**
Line 120: Line 166:
Next you have to test if your program works with the library packaged from someone else, then do:
Next you have to test if your program works with the library packaged from someone else, then do:


  git checkout master
<syntaxhighlight lang="console">
  git log --stat
$ git checkout master
$ git log --stat
</syntaxhighlight>


and pick the commit where the library was added. Finally cherry-pick that commit into your branch:
and pick the commit where the library was added. Finally cherry-pick that commit into your branch:


  git checkout my-new-pkg
<syntaxhighlight lang="console">
  git cherry-pick 5d97886a6a545fb20495e0837cc50fa63d2a80e1
$ git checkout my-new-pkg
$ git cherry-pick 5d97886a6a545fb20495e0837cc50fa63d2a80e1
</syntaxhighlight>


Afterwards do your usual tests and if needed also make modifications to the library but keep in mind that this might break the other use-case of that library and if in doubt check that as well.
Afterwards do your usual tests and if needed also make modifications to the library but keep in mind that this might break the other use-case of that library and if in doubt check that as well.
Line 134: Line 184:
nix-shell is a command which drops you into the build environment for a package. This is convenient for writing and debugging nix expressions. Nix-shell requires nix-1.6.x although running nix-build --run-env produces a similar environment.
nix-shell is a command which drops you into the build environment for a package. This is convenient for writing and debugging nix expressions. Nix-shell requires nix-1.6.x although running nix-build --run-env produces a similar environment.


  mkdir -p /tmp/nix-shell-bc
<syntaxhighlight lang="bash">
  cd /tmp/nix-shell-bc
$ mkdir -p ~/tmpdev/bc-build  &&  cd ~/tmpdev/bc-build
  nix-shell $NIXREPOS/nixpkgs  -A bc
$ nix-shell $NIXPKGS -A bc
  export out=/tmp/foo/out
</syntaxhighlight>
 
You can also drop in the build environment for a package not in nixpkgs.
<syntaxhighlight lang="bash">
$ mkdir -p ~/tmpdev/bc-build  &&  cd ~/tmpdev/bc-build
$ nix-shell -E "with import <nixpkgs> {}; callPackage /path/to/package.nix {}"
</syntaxhighlight>


now we have find out which phases are specified for this package:
You would have seen the dependencies downloading, but the ''bc-build'' directory remains empty. The build system would next invoke a builder with some arguments. You can obtain the exact name of the builder (usually '''bash''') and the arguments '''args''' of the builder (typically a shell script) by checking the corresponding value in:
    typeset -f genericBuild | grep 'phases='
<syntaxhighlight lang="bash">
    phases="$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases";
$ nix derivation show $(nix-instantiate .)
</syntaxhighlight>


The phases can be defined either as a string to be eval'ed or as a shell function, [https://github.com/NixOS/nixpkgs/blob/5a0b79/pkgs/stdenv/generic/setup.sh#L818 this is how] Nix invokes it.
However, most of the time (for instance when using '''stdenv''' ) the [https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/default-builder.sh default builder] invokes first '''source $stdenv/setup''' to load the appropriate environment variables, and then '''genericBuild()'''. This is a shell function defined by [https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh stdenv] that you can review like this...
<syntaxhighlight lang="bash">
$ typeset -f genericBuild | less
</syntaxhighlight>
which shows when custom variables '''$buildCommandPath''' or '''$buildCommand''' are defined, those are evaluated exclusively.  Otherwise, if no custom '''$phases''' variable is set, the standard build phase order is used as shown here...
<syntaxhighlight lang="bash">
$ typeset -f genericBuild | grep 'phases='
phases="$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases";
</syntaxhighlight>


so when developing you need to run these phases in a row:
The phases can be defined either as a string to be eval'ed or as a shell function, [https://github.com/NixOS/nixpkgs/blob/5a0b79f955d6c2dc21239f1b0d956ef8dc89a57e/pkgs/stdenv/generic/setup.sh#L818 this is how] Nix invokes it.
  unpackPhase
 
  patchPhase
 
  configurePhase
So to observe a full build, you can do...
  buildPhase
<syntaxhighlight lang="bash">
  checkPhase
$ export out=~/tmpdev/bc-build/out
  installPhase
$ set -x # Optional: it prints all commands, can be practical to debug
  fixPhase
$ set +e # Optional: do not quit the shell on simple errors, Ctrl-C,...
  installCheckPhase
$ export NIX_ENFORCE_PURITY=0 # Optional: nix automatically skip absolute files not in /nix/, /tmp or NIX_BUILD_TOP. When building in a nix-shell this may be an issue as the source won't be  in the above folders and you may get errors like Fatal error: blabla.h: No such file or directory. https://github.com/NixOS/nixpkgs/issues/204036
  installPhase
$ genericBuild
  distPhase
</syntaxhighlight>
 
To only run some specific phases, use runPhase:
 
<syntaxhighlight lang="bash">
# Syntax: runPhase *phase*
$ runPhase unpackPhase
</syntaxhighlight>
 
While developing your own package, you need to run these phases in order:
<syntaxhighlight lang="bash">
unpackPhase
patchPhase
configurePhase
buildPhase
checkPhase
installPhase
fixupPhase
installCheckPhase
distPhase
</syntaxhighlight>
 
Phases can be both bash functions, or environment of identical name (when they are overridden). <code>genericBuild</code> takes care of that distinction for you, invoking them appropriately. You can of course drop down to evaluating them yourself, for example to invoke an overridden phase (an environment variable) using '''eval''' instead:
 
<syntaxhighlight lang="bash">
eval "$checkPhase"
# etc..
</syntaxhighlight>


{{Note|you do not need to run $preConfigurePhase explicitly as it is run, when running configurePhase already.}}
{{Note|you do not need to run $preConfigurePhase explicitly as it is run, when running configurePhase already.}}


list all functions which are declared in '''set''':
To list all functions which are declared in '''set''':
  typeset -F
<syntaxhighlight lang="bash">
  declare -f addCVars
typeset -F
  declare -f addToCrossEnv
declare -f addCVars
  declare -f addToNativeEnv
declare -f addToCrossEnv
  declare -f addToSearchPath
declare -f addToNativeEnv
  declare -f addToSearchPathWithCustomDelimiter
declare -f addToSearchPath
  declare -f buildPhase
declare -f addToSearchPathWithCustomDelimiter
  declare -f checkPhase
declare -f buildPhase
  declare -f closeNest
declare -f checkPhase
  declare -f command_not_found_handle
declare -f closeNest
  declare -f configurePhase
declare -f command_not_found_handle
  declare -f distPhase
declare -f configurePhase
  declare -f dumpVars
declare -f distPhase
  declare -f ensureDir
declare -f dumpVars
  declare -f exitHandler
declare -f ensureDir
  declare -f findInputs
declare -f exitHandler
  declare -f fixLibtool
declare -f findInputs
  declare -f fixupPhase
declare -f fixLibtool
  declare -f genericBuild
declare -f fixupPhase
  declare -f header
declare -f genericBuild
  declare -f installBin
declare -f header
  declare -f installCheckPhase
declare -f installBin
  declare -f installPhase
declare -f installCheckPhase
  declare -f patchELF
declare -f installPhase
  declare -f patchPhase
declare -f patchELF
  declare -f patchShebangs
declare -f patchPhase
  declare -f runHook
declare -f patchShebangs
  declare -f showPhaseHeader
declare -f runHook
  declare -f startNest
declare -f showPhaseHeader
  declare -f stopNest
declare -f startNest
  declare -f stripDirs
declare -f stopNest
  declare -f stripHash
declare -f stripDirs
  declare -f substitute
declare -f stripHash
  declare -f substituteAll
declare -f substitute
  declare -f substituteAllInPlace
declare -f substituteAll
  declare -f substituteInPlace
declare -f substituteAllInPlace
  declare -f unpackFile
declare -f substituteInPlace
  declare -f unpackPhase
declare -f unpackFile
declare -f unpackPhase
</syntaxhighlight>


If the phase has been defined as a function, to list a particular function type:
If the phase has been defined as a function, to list a particular function type:
  typeset -f unpackPhase
<syntaxhighlight lang="bash">
typeset -f unpackPhase
</syntaxhighlight>


Otherwise, if it was a string, simply echo the variable related to it
Otherwise, if it was a string, simply echo the variable related to it


  echo $unpackPhase
<syntaxhighlight lang="bash">
echo "$unpackPhase"
</syntaxhighlight>


In either case, you can see the code that is about to be executed for each phase:
In either case, you can see the code that is about to be executed for each phase:


  typeset -f unpackPhase
<syntaxhighlight lang="bash">
  unpackPhase ()  
typeset -f unpackPhase
  {  
unpackPhase ()
      runHook preUnpack;
{
      if [ -z "$srcs" ]; then
    runHook preUnpack;
          if [ -z "$src" ]; then
    if [ -z "$srcs" ]; then
              echo 'variable $src or $srcs should point to the source';
        if [ -z "$src" ]; then
              exit 1;
            echo 'variable $src or $srcs should point to the source';
          fi;
            exit 1;
          srcs="$src";
        fi;
      fi;
        srcs="$src";
      local dirsBefore="";
    fi;
      for i in *;
    local dirsBefore="";
      do
    for i in *;
          if [ -d "$i" ]; then
    do
              dirsBefore="$dirsBefore $i ";
        if [ -d "$i" ]; then
          fi;
            dirsBefore="$dirsBefore $i ";
      done;
        fi;
      for i in $srcs;
    done;
      do
    for i in $srcs;
          unpackFile $i;
    do
      done;
        unpackFile $i;
      if [ -n "$setSourceRoot" ]; then
    done;
          runHook setSourceRoot;
    if [ -n "$setSourceRoot" ]; then
      else
        runHook setSourceRoot;
          if [ -z "$sourceRoot" ]; then
    else
              sourceRoot=;
        if [ -z "$sourceRoot" ]; then
              for i in *;
            sourceRoot=;
              do
            for i in *;
                  if [ -d "$i" ]; then
            do
                      case $dirsBefore in  
                if [ -d "$i" ]; then
                          *\ $i\ *)
                    case $dirsBefore in
 
                        *\ $i\ *)
                          ;;
 
                          *)
                        ;;
                              if [ -n "$sourceRoot" ]; then
                        *)
                                  echo "unpacker produced multiple directories";
                            if [ -n "$sourceRoot" ]; then
                                  exit 1;
                                echo "unpacker produced multiple directories";
                              fi;
                                exit 1;
                              sourceRoot="$i"
                            fi;
                          ;;
                            sourceRoot="$i"
                      esac;
                        ;;
                  fi;
                    esac;
              done;
                fi;
          fi;
            done;
      fi;
        fi;
      if [ -z "$sourceRoot" ]; then
    fi;
          echo "unpacker appears to have produced no directories";
    if [ -z "$sourceRoot" ]; then
          exit 1;
        echo "unpacker appears to have produced no directories";
      fi;
        exit 1;
      echo "source root is $sourceRoot";
    fi;
      if [ "$dontMakeSourcesWritable" != 1 ]; then
    echo "source root is $sourceRoot";
          chmod -R u+w "$sourceRoot";
    if [ "$dontMakeSourcesWritable" != 1 ]; then
      fi;
        chmod -R u+w "$sourceRoot";
      runHook postUnpack
    fi;
  }
    runHook postUnpack
}
</syntaxhighlight>


you can also modify the configureFlags prefix:
you can also modify the configureFlags prefix:
  export configureFlags="--prefix=$out --with-readline"
<syntaxhighlight lang="bash">
export configureFlags="--prefix=$out --with-readline"
</syntaxhighlight>


Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.
Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.
== Adding custom libraries and dependencies to a package ==
If you are packaging a dependency, such as a library used by applications for them to compile their code, you might have found you'd like to test if the derivation file installs correctly and can be used by other software.
In order to do this, you'll need to make a simple program that references the library, make a derivation for this program, then add the dependency. For example:
Your program to test the library:
<syntaxhighlight lang="nix">
{
  pkgs ? import <nixpkgs> {
    overlays = [
      (final: prev: {
        my-library = prev.callPackage ./my-library.nix { };
      })
    ];
  },
}:
pkgs.callPackage (
  {
    stdenv,
    hello,
    pkg-config,
    my-library,
  }:
  stdenv.mkDerivation {
    pname = "something";
    version = "1";
    strictDeps = true;
    # host/target agnostic programs
    depsBuildBuild = [
      hello
    ];
    # compilers & linkers & dependecy finding programs
    nativeBuildInputs = [
      pkg-config
    ];
    # libraries
    buildInputs = [
      my-library
    ];
  }
) { }
</syntaxhighlight>


== nix channels ==
== nix channels ==
nix channels can be used in parallel with your new local repositories, see its [[install/remove software#nix-channels| nix-channel-documentation]]
nix channels can be used in parallel with your new local repositories, see its [[install/remove software#nix-channels| nix-channel-documentation]]


== Testing Package Updates with nixpkgs-review ==
You can also use [https://github.com/Mic92/nixpkgs-review nixpkgs-review] to compile, review and merge packages and its dependencies. It claims to be faster than nox and provides a nix-shell where you can test the package.
First make sure it is available in your shell:
<syntaxhighlight lang="bash">
nix-shell -p nixpkgs-review
</syntaxhighlight>
You can run nixpkgs-review against uncommitted/staged changes in a cloned nixpkgs repository:
<syntaxhighlight lang="bash">
cd ~/git/nixpkgs
nixpkgs-review wip [--staged]
</syntaxhighlight>


== Testing Package Updates with Nox ==
It is also possible to review a specified commit:
<syntaxhighlight lang="bash">
cd ~/git/nixpkgs
nixpkgs-review rev HEAD
</syntaxhighlight>


If you are updating a package's version, you can use nox to make sure all packages that depend on the updated package still compile correctly.
If you have already committed your changes and created a pull request, you can use the pr command:
<syntaxhighlight lang="bash">
cd ~/git/nixpkgs
nixpkgs-review pr 5341
</syntaxhighlight>


First make sure it is in your environment:
You can post the build result, approve/merge or read the comments of a pull request inside the provided nix-shell:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
nix-env -i nox
# inside the provided shell
nixpkgs-review post-result
nixpkgs-review approve
nixpkgs-review merge
nixpkgs-review comments
</syntaxhighlight>
</syntaxhighlight>


You can run nox against uncommited changes to a nixpkgs repository:
If the pr provides a new package you can start it inside the nix-shell using its package name:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cd ~/.nix-defexpr
# inside the provided shell
nox-review wip
packagename
</syntaxhighlight>
</syntaxhighlight>


If you have already commited your changes and created a pull request, you can use the pr command:
== Formatting Packages with nixfmt ==
It is "good practice" to format packages in a way that following changed will create as minimal diffs as possible. The formatter [https://github.com/NixOS/nixfmt nixfmt] can be used for that.
<syntaxhighlight lang="bash">
nix-shell -p nixfmt-rfc-style --run 'nixfmt path/to/default.nix'
</syntaxhighlight>
 
== Testing Packages with nixpkgs-hammering ==
You can test some "good practices" in a package with [https://github.com/jtojnar/nixpkgs-hammering nixpkgs-hammering]. But before applying the recommendations you should read the given [https://github.com/jtojnar/nixpkgs-hammering/tree/main/explanations explanations].
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
nox-review pr 5341
cd ~/git/nixpkgs
nix run -f https://github.com/jtojnar/nixpkgs-hammering/archive/master.tar.gz -c nixpkgs-hammer packagename # if you are using stable nix
nix run github:jtojnar/nixpkgs-hammering packagename # if you are using nix flake
</syntaxhighlight>
</syntaxhighlight>


Line 297: Line 479:


* [[Generic Algorithm on Doing Packaging]]
* [[Generic Algorithm on Doing Packaging]]
* [https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.html Fundamentals of Stdenv] in Nix Pills
* [https://nixos.org/guides/nix-pills/developing-with-nix-shell.html Developing with nix-shell] in Nix Pills


[[Category:Development]]
[[Category:Development]]

Latest revision as of 19:00, 29 October 2024

This article describes how to work with the nix related repositories to add new packages, edit and debug existing packages. For details on the NixOS module system see NixOS:Modules. NixOS:extend_NixOS explains how to write, test and debug your own modules.

There is a chapter about hacking packages and modules in the NixOS manual: development

Writing packages is covered in quick start and writing modules is covered in the manual

If you've read the manual and still don't know how to go about creating a package, read on.

The nix repositories are hosted at https://github.com/NixOS


Basics

The code for nix packages is managed in the nixpkgs/pkgs repository. NixOS services, and other system configuration options are managed in nixpkgs/nixos.

The steps to take for your first change should look something like this:

  1. Fork the repo (e.g. click the fork button on https://github.com/nixos/nixpkgs).
  2. Clone your fork git clone --depth 1 https://github.com/YOURNAME/nixpkgs.git
  3. Hack hack hack
  4. Push your changes to your fork
  5. Open a pull request
  6. Profit!

This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)

Rough process for creating a package

There are different steps here depending on whether you're building from source or packaging an existing binary. There are some common steps too.

Package from source code

  1. Read the repo build instructions and CI scripts (for example, on GitHub, these are located in .github/workflows).
  2. Look in nixpkgs for a package with a similar build process to use as reference. For example, if you're packaging a project written in Go, find a package for an existing Go application. Each language has its own supporting Nix functions and a more or less standard way of dealing with things. For example, Go has buildGoModule. Rust has buildRustPackage. Python has buildPythonApplication. Node.js has node2nix, yarn2nix, etc. There are also specific functions for wrapping e.g. GNOME applications (wrapGAppsHook), or Qt apps (libsForQt5, wrapQtAppsHook). Refer to the language support chapter in the nixpkgs manual.
  3. If there isn't a specific builder for the language, use stdenv.mkDerivation directly, which has built-in support for GNU make (and other build systems, provided you add the necessary nativeBuildInputs).
  4. Figure out at least some dependencies from the project repo. See if they're available in nixpkgs (nix search some-library or nix-locate --top-level lib/somelibrary.so). If any dependency is missing you'll need to package that as well.
  5. Create your derivation in default.nix in some empty local directory.
  6. At the top of the derivation, temporarily add with import <nixpkgs> {};. For now, don't worry too much about declaring every dependency as a parameter, to save time.
  7. Build the package with nix-build. Iterate on tweaking the derivation and rebuilding until it succeeds.
  8. For large projects with long compile times, you can use nix-shell instead to run the individual phases.
  9. At this stage, you may encounter some build quirks of the project. Compile-time errors will hopefully explain what you're missing. For example micromamba needs a specialized build of libsolv.
  10. Read on below for further steps.

Packages from binaries

  1. There's probably a package for it for some other distro. Use that package definition to figure out the dependencies. For example, if you have a deb package you can view its dependencies by running dpkg -I <package.deb>. Arch packages can also be useful to look up for reference (view the package's PKGBUILD):
  2. Sometimes the definitions for other distros won't be enough by nix's standards. If that's the case, use ldd and/or strace to find the rest of the dependencies. If you're not familiar with ldd/strace see How to find out the dynamic libraries executables loads when run?
  3. See how other nix binary packages deal with dependencies. For example nix packages based on deb packages.
  4. If the application contains some helper executable or vendored dlopen'd library you will probably need to give the nix treatment to it as well. For example, JetBrains Rider vendors dotnet so it needs to be replaced with the dotnet nix package.
  5. Because there's no real build step here you'll have to rely more on testing the actual execution of the package.

Both source code packages and binary packages

  1. Once you have the package building successfully, test the output. Ensure the build completes usingnix-build, then run result/bin/<executableName>. Test as much functionality of the application as you can to ensure that it works as intended.
  2. Now that your package builds and runs, it's time to move it to nixpkgs. Read nixpkgs/CONTRIBUTING.md, make sure your package is up to the standards e.g. add a suitable meta section.
  3. Git clone https://github.com/NixOS/nixpkgs, figure out the best category / directory for the application (within https://github.com/NixOS/nixpkgs/tree/master/pkgs/), create the directory for your application, and move your default.nix there.
  4. If you used with import <nixpkgs> {}; to iterate more quickly, now is the time to replace that with the actual dependencies as an attribute set at the beginning of the file e.g. { lib, stdenv, fetchFromGitHub }:
  5. Add the package to the top level declaration of packages. Most of the time this will be https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix .
  6. If this is your first package in nixpkgs, add yourself in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix in a separate commit.
  7. Read on about the final steps of branching and sending your PR in https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md .

How to install from the local repository

For expediency just for this article, we'll shallow clone direct from the distribution repo and set an environment variable pointing to it.

$ mkdir -p ~/tmpdev && cd ~/tmpdev
$ git clone --depth=1 https://github.com/nixos/nixpkgs
$ export NIXPKGS=~/tmpdev/nixpkgs
$ ls $NIXPKGS

make some changes ...

example: list all available software from the local repository $NIXPKGS

$ nix-env -f $NIXPKGS -qaP '*'

example: update the system based on your local $NIXPKGS

$ nixos-rebuild -I nixpkgs=$NIXPKGS switch

example: build an expression and put the output in to `pwd`/results

$ nix-build $NIXPKGS -A irssi

example: get an environment which is used to build irssi (also see nix-shell)

$ nix-build $NIXPKGS --run-env -A irssi

example: get a persistent environment which is used to build irssi

$ nix-build $NIXPKGS --run-env -A irssi --add-root

Tracking upstream changes and avoiding extra rebuilding

You have forked the relevant nix repository, but you will want to track changes in the upstream nix repo too. You can add a remote, and a corresponding branch for this.

$ git remote add upstream https://github.com/NixOS/nixpkgs.git

You can create a branch to track the upstream master branch:

$ git fetch upstream
$ git checkout -b upstream-master upstream/master
$ git pull

This will put you into a branch with all the latest changes. Hydra, the build farm, regularly creates binaries, but, since people are constantly contributing to the nix repositories, it is usually the case that there are changes in the master branch which have not yet made it into the binary channel. To take advantage of available binaries you can switch to the revision which produced the binaries in your current system and apply your changes from there. You can use `nixos-version` to see the relevant short revision hash:

$ nixos-version 
14.11pre52727.5d97886 (Caterpillar)
${NixOS release}.${nixpkgs revision} 
(since the git-repo called nixos was merged into nixpkgs)
$ nixos-version 
13.07pre4871_18de9f6-3c35dae (Aardvark)
${NixOS release}_${NixOS revision}-${nixpkgs revision}

This string shows the Nixos release number (13.07pre4871) followed by the nixos revision used to produce your current system (18de9f6) followed by the nixpkgs revision (3c35dae).

$ git branch
upstream-master
$ git checkout -b nixpkgs-channel 3c35dae
Switched to a new branch 'nixpkgs-channel'
$ git checkout -b my-new-pkg
Switched to a new branch 'my-new-pkg'

After making some changes you can commit them into your local repo:

$ git add foo
$ git commit

Then you push your changes to your fork:

$ git push origin my-new-pkg

You can use this to open a pull request on github.

If some time has passed since you have created your fork, you will want to merge your changes with upstream and test that it still works.

$ git fetch upstream
$ git merge upstream

If your merge then fails because someone else has made the same change (for example, someone else also packaged a library you have just packed for the program you want to get into nixpkgs), then you can do this:

$ git rebase -i HEAD~10

there select the edit mode for your commit and remove the your code which added the library. **Warning: only use 'git rebase' on your commits, which have not been pushed and nobody else is working with already!**

Next you have to test if your program works with the library packaged from someone else, then do:

$ git checkout master
$ git log --stat

and pick the commit where the library was added. Finally cherry-pick that commit into your branch:

$ git checkout my-new-pkg
$ git cherry-pick 5d97886a6a545fb20495e0837cc50fa63d2a80e1

Afterwards do your usual tests and if needed also make modifications to the library but keep in mind that this might break the other use-case of that library and if in doubt check that as well.

Using nix-shell for package development

nix-shell is a command which drops you into the build environment for a package. This is convenient for writing and debugging nix expressions. Nix-shell requires nix-1.6.x although running nix-build --run-env produces a similar environment.

$ mkdir -p ~/tmpdev/bc-build  &&  cd ~/tmpdev/bc-build
$ nix-shell $NIXPKGS -A bc

You can also drop in the build environment for a package not in nixpkgs.

$ mkdir -p ~/tmpdev/bc-build  &&  cd ~/tmpdev/bc-build
$ nix-shell -E "with import <nixpkgs> {}; callPackage /path/to/package.nix {}"

You would have seen the dependencies downloading, but the bc-build directory remains empty. The build system would next invoke a builder with some arguments. You can obtain the exact name of the builder (usually bash) and the arguments args of the builder (typically a shell script) by checking the corresponding value in:

$ nix derivation show $(nix-instantiate .)

However, most of the time (for instance when using stdenv ) the default builder invokes first source $stdenv/setup to load the appropriate environment variables, and then genericBuild(). This is a shell function defined by stdenv that you can review like this...

$ typeset -f genericBuild | less

which shows when custom variables $buildCommandPath or $buildCommand are defined, those are evaluated exclusively. Otherwise, if no custom $phases variable is set, the standard build phase order is used as shown here...

$ typeset -f genericBuild | grep 'phases='
phases="$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases";

The phases can be defined either as a string to be eval'ed or as a shell function, this is how Nix invokes it.


So to observe a full build, you can do...

$ export out=~/tmpdev/bc-build/out
$ set -x # Optional: it prints all commands, can be practical to debug
$ set +e # Optional: do not quit the shell on simple errors, Ctrl-C,...
$ export NIX_ENFORCE_PURITY=0 # Optional: nix automatically skip absolute files not in /nix/, /tmp or NIX_BUILD_TOP. When building in a nix-shell this may be an issue as the source won't be  in the above folders and you may get errors like Fatal error: blabla.h: No such file or directory. https://github.com/NixOS/nixpkgs/issues/204036
$ genericBuild

To only run some specific phases, use runPhase:

# Syntax: runPhase *phase*
$ runPhase unpackPhase

While developing your own package, you need to run these phases in order:

unpackPhase
patchPhase
configurePhase
buildPhase
checkPhase
installPhase
fixupPhase
installCheckPhase
distPhase

Phases can be both bash functions, or environment of identical name (when they are overridden). genericBuild takes care of that distinction for you, invoking them appropriately. You can of course drop down to evaluating them yourself, for example to invoke an overridden phase (an environment variable) using eval instead:

eval "$checkPhase"
# etc..
Note: you do not need to run $preConfigurePhase explicitly as it is run, when running configurePhase already.

To list all functions which are declared in set:

typeset -F
declare -f addCVars
declare -f addToCrossEnv
declare -f addToNativeEnv
declare -f addToSearchPath
declare -f addToSearchPathWithCustomDelimiter
declare -f buildPhase
declare -f checkPhase
declare -f closeNest
declare -f command_not_found_handle
declare -f configurePhase
declare -f distPhase
declare -f dumpVars
declare -f ensureDir
declare -f exitHandler
declare -f findInputs
declare -f fixLibtool
declare -f fixupPhase
declare -f genericBuild
declare -f header
declare -f installBin
declare -f installCheckPhase
declare -f installPhase
declare -f patchELF
declare -f patchPhase
declare -f patchShebangs
declare -f runHook
declare -f showPhaseHeader
declare -f startNest
declare -f stopNest
declare -f stripDirs
declare -f stripHash
declare -f substitute
declare -f substituteAll
declare -f substituteAllInPlace
declare -f substituteInPlace
declare -f unpackFile
declare -f unpackPhase

If the phase has been defined as a function, to list a particular function type:

typeset -f unpackPhase

Otherwise, if it was a string, simply echo the variable related to it

echo "$unpackPhase"

In either case, you can see the code that is about to be executed for each phase:

typeset -f unpackPhase
unpackPhase ()
{
    runHook preUnpack;
    if [ -z "$srcs" ]; then
        if [ -z "$src" ]; then
            echo 'variable $src or $srcs should point to the source';
            exit 1;
        fi;
        srcs="$src";
    fi;
    local dirsBefore="";
    for i in *;
    do
        if [ -d "$i" ]; then
            dirsBefore="$dirsBefore $i ";
        fi;
    done;
    for i in $srcs;
    do
        unpackFile $i;
    done;
    if [ -n "$setSourceRoot" ]; then
        runHook setSourceRoot;
    else
        if [ -z "$sourceRoot" ]; then
            sourceRoot=;
            for i in *;
            do
                if [ -d "$i" ]; then
                    case $dirsBefore in
                        *\ $i\ *)

                        ;;
                        *)
                            if [ -n "$sourceRoot" ]; then
                                echo "unpacker produced multiple directories";
                                exit 1;
                            fi;
                            sourceRoot="$i"
                        ;;
                    esac;
                fi;
            done;
        fi;
    fi;
    if [ -z "$sourceRoot" ]; then
        echo "unpacker appears to have produced no directories";
        exit 1;
    fi;
    echo "source root is $sourceRoot";
    if [ "$dontMakeSourcesWritable" != 1 ]; then
        chmod -R u+w "$sourceRoot";
    fi;
    runHook postUnpack
}

you can also modify the configureFlags prefix:

export configureFlags="--prefix=$out --with-readline"

Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.

Adding custom libraries and dependencies to a package

If you are packaging a dependency, such as a library used by applications for them to compile their code, you might have found you'd like to test if the derivation file installs correctly and can be used by other software.

In order to do this, you'll need to make a simple program that references the library, make a derivation for this program, then add the dependency. For example:

Your program to test the library:

{
  pkgs ? import <nixpkgs> {
    overlays = [
      (final: prev: {
        my-library = prev.callPackage ./my-library.nix { };
      })
    ];
  },
}:
pkgs.callPackage (
  {
    stdenv,
    hello,
    pkg-config,
    my-library,
  }:
  stdenv.mkDerivation {
    pname = "something";
    version = "1";
    strictDeps = true;
    # host/target agnostic programs
    depsBuildBuild = [
      hello
    ];
    # compilers & linkers & dependecy finding programs
    nativeBuildInputs = [
      pkg-config
    ];
    # libraries
    buildInputs = [
      my-library
    ];
  }
) { }


nix channels

nix channels can be used in parallel with your new local repositories, see its nix-channel-documentation

Testing Package Updates with nixpkgs-review

You can also use nixpkgs-review to compile, review and merge packages and its dependencies. It claims to be faster than nox and provides a nix-shell where you can test the package.

First make sure it is available in your shell:

nix-shell -p nixpkgs-review

You can run nixpkgs-review against uncommitted/staged changes in a cloned nixpkgs repository:

cd ~/git/nixpkgs
nixpkgs-review wip [--staged]

It is also possible to review a specified commit:

cd ~/git/nixpkgs
nixpkgs-review rev HEAD

If you have already committed your changes and created a pull request, you can use the pr command:

cd ~/git/nixpkgs
nixpkgs-review pr 5341

You can post the build result, approve/merge or read the comments of a pull request inside the provided nix-shell:

# inside the provided shell
nixpkgs-review post-result
nixpkgs-review approve
nixpkgs-review merge
nixpkgs-review comments

If the pr provides a new package you can start it inside the nix-shell using its package name:

# inside the provided shell
packagename

Formatting Packages with nixfmt

It is "good practice" to format packages in a way that following changed will create as minimal diffs as possible. The formatter nixfmt can be used for that.

nix-shell -p nixfmt-rfc-style --run 'nixfmt path/to/default.nix'

Testing Packages with nixpkgs-hammering

You can test some "good practices" in a package with nixpkgs-hammering. But before applying the recommendations you should read the given explanations.

cd ~/git/nixpkgs
nix run -f https://github.com/jtojnar/nixpkgs-hammering/archive/master.tar.gz -c nixpkgs-hammer packagename # if you are using stable nix
nix run github:jtojnar/nixpkgs-hammering packagename # if you are using nix flake

See also