Nixpkgs/Create and debug packages
This article describes how to work with the nix related repositories to add new packages, edit and debug existing packages. For details on the NixOS module system see NixOS:Modules. NixOS:extend_NixOS explains how to write, test and debug your own modules.
There is a chapter about hacking packages and modules in the NixOS manual: http://nixos.org/nixos/manual/index.html#ch-development
Writing packages is covered in http://nixos.org/nixpkgs/manual and writing modules in http://nixos.org/nixos/manual
The nix repositories are hosted here: https://github.com/nixos
Basics
The code for nix packages is managed in the nixpkgs repository. NixOS services, and other system configuration options are managed in the nixos sub-directory of the nixpkgs repository.
The steps to take for your first change should look something like this:
- Fork the repo (e.g. click the fork button on https://github.com/nixos/nixpkgs).
- Clone your fork
git clone git@github.com:YOUR-GITHUB-ACCOUNT-NAME/nixpkgs.git - Hack hack hack
- Push your changes to your fork
- Open a pull request
- Profit!
This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)
How to install from the local repository
For expediency just for this article, we'll shallow clone direct from the distribution repo and set an environment variable pointing to it.
$ mkdir -p ~/tmpdev && cd ~/tmpdev
$ git clone --depth=1 https://github.com/nixos/nixpkgs
$ export NIXPKGS=~/tmpdev/nixpkgs
$ ls $NIXPKGS
make some changes ...
example: list all available software from the local repository $NIXPKGS
$ nix-env -f $NIXPKGS -qaP '*'
example: install software from local repository
$ nix-env -f $NIXPKGS -iA python-urlgrabber
example: update the system based on your local $NIXPKGS
$ nixos-rebuild -I nixpkgs=$NIXPKGS switch
example: build an expression and put the output in to `pwd`/results
$ nix-build $NIXPKGS -A irssi
example: get an environment which is used to build irssi (also see nix-shell)
$ nix-build $NIXPKGS --run-env -A irssi
example: get a persistent environment which is used to build irssi
$ nix-build $NIXPKGS --run-env -A irssi --add-root
Tracking upstream changes and avoiding extra rebuilding
You have forked the relevant nix repository, but you will want to track changes in the upstream nix repo too. You can add a remote, and a corresponding branch for this.
$ git remote add upstream https://github.com/NixOS/nixpkgs.git
You can create a branch to track the upstream master branch:
$ git fetch upstream
$ git checkout -b upstream-master upstream/master
$ git pull
This will put you into a branch with all the latest changes. Hydra, the build farm, regularly creates binaries, but, since people are constantly contributing to the nix repositories, it is usually the case that there are changes in the master branch which have not yet made it into the binary channel. To take advantage of available binaries you can switch to the revision which produced the binaries in your current system and apply your changes from there. You can use `nixos-version` to see the relevant short revision hash:
$ nixos-version
14.11pre52727.5d97886 (Caterpillar)
${NixOS release}.${nixpkgs revision}
(since the git-repo called nixos was merged into nixpkgs)
$ nixos-version
13.07pre4871_18de9f6-3c35dae (Aardvark)
${NixOS release}_${NixOS revision}-${nixpkgs revision}
This string shows the Nixos release number (13.07pre4871) followed by the nixos revision used to produce your current system (18de9f6) followed by the nixpkgs revision (3c35dae).
$ git branch
upstream-master
$ git checkout -b nixpkgs-channel 3c35dae
Switched to a new branch 'nixpkgs-channel'
$ git checkout -b my-new-pkg
Switched to a new branch 'my-new-pkg'
After making some changes you can commit them into your local repo:
$ git add foo
$ git commit
Then you push your changes to your fork:
$ git push origin my-new-pkg
You can use this to open a pull request on github.
If some time has passed since you have created your fork, you will want to merge your changes with upstream and test that it still works.
$ git fetch upstream
$ git merge upstream
If your merge then fails because someone else has made the same change (for example, someone else also packaged a library you have just packed for the program you want to get into nixpkgs), then you can do this:
$ git rebase -i HEAD~10
there select the edit mode for your commit and remove the your code which added the library. **Warning: only use 'git rebase' on your commits, which have not been pushed and nobody else is working with already!**
Next you have to test if your program works with the library packaged from someone else, then do:
$ git checkout master
$ git log --stat
and pick the commit where the library was added. Finally cherry-pick that commit into your branch:
$ git checkout my-new-pkg
$ git cherry-pick 5d97886a6a545fb20495e0837cc50fa63d2a80e1
Afterwards do your usual tests and if needed also make modifications to the library but keep in mind that this might break the other use-case of that library and if in doubt check that as well.
Using nix-shell for package development
nix-shell is a command which drops you into the build environment for a package. This is convenient for writing and debugging nix expressions. Nix-shell requires nix-1.6.x although running nix-build --run-env produces a similar environment.
$ mkdir -p ~/tmpdev/bc-build && cd ~/tmpdev/bc-build
$ nix-shell $NIXPKGS -A bc
You can also drop in the build environment for a package not in nixpkgs.
$ mkdir -p ~/tmpdev/bc-build && cd ~/tmpdev/bc-build
$ nix-shell -E "with import <nixpkgs> {}; callPackage /path/to/package.nix {}"
You would have seen the dependencies downloading, but the bc-build directory remains empty. The build system would next invoke genericBuild(). This is a shell function defined by stdenv that you can review like this...
$ typeset -f genericBuild | less
which shows when custom variables $buildCommandPath or $buildCommand are defined, those are evaluated exclusively. Otherwise, if no custom $phases variable is set, the standard build phase order is used as shown here...
$ typeset -f genericBuild | grep 'phases='
phases="$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases";
The phases can be defined either as a string to be eval'ed or as a shell function, this is how Nix invokes it.
So to observe a full build, you can do...
$ export out=~/tmpdev/bc-build/out
$ set -x
$ genericBuild
or while developing your own package, you need to individually run these phases in order:
unpackPhase
patchPhase
configurePhase
buildPhase
checkPhase
installPhase
fixupPhase
installCheckPhase
distPhase
Any overridden phases should be invoked using eval instead:
eval "$checkPhase"
# etc..
To list all functions which are declared in set:
typeset -F
declare -f addCVars
declare -f addToCrossEnv
declare -f addToNativeEnv
declare -f addToSearchPath
declare -f addToSearchPathWithCustomDelimiter
declare -f buildPhase
declare -f checkPhase
declare -f closeNest
declare -f command_not_found_handle
declare -f configurePhase
declare -f distPhase
declare -f dumpVars
declare -f ensureDir
declare -f exitHandler
declare -f findInputs
declare -f fixLibtool
declare -f fixupPhase
declare -f genericBuild
declare -f header
declare -f installBin
declare -f installCheckPhase
declare -f installPhase
declare -f patchELF
declare -f patchPhase
declare -f patchShebangs
declare -f runHook
declare -f showPhaseHeader
declare -f startNest
declare -f stopNest
declare -f stripDirs
declare -f stripHash
declare -f substitute
declare -f substituteAll
declare -f substituteAllInPlace
declare -f substituteInPlace
declare -f unpackFile
declare -f unpackPhase
If the phase has been defined as a function, to list a particular function type:
typeset -f unpackPhase
Otherwise, if it was a string, simply echo the variable related to it
echo "$unpackPhase"
In either case, you can see the code that is about to be executed for each phase:
typeset -f unpackPhase
unpackPhase ()
{
runHook preUnpack;
if [ -z "$srcs" ]; then
if [ -z "$src" ]; then
echo 'variable $src or $srcs should point to the source';
exit 1;
fi;
srcs="$src";
fi;
local dirsBefore="";
for i in *;
do
if [ -d "$i" ]; then
dirsBefore="$dirsBefore $i ";
fi;
done;
for i in $srcs;
do
unpackFile $i;
done;
if [ -n "$setSourceRoot" ]; then
runHook setSourceRoot;
else
if [ -z "$sourceRoot" ]; then
sourceRoot=;
for i in *;
do
if [ -d "$i" ]; then
case $dirsBefore in
*\ $i\ *)
;;
*)
if [ -n "$sourceRoot" ]; then
echo "unpacker produced multiple directories";
exit 1;
fi;
sourceRoot="$i"
;;
esac;
fi;
done;
fi;
fi;
if [ -z "$sourceRoot" ]; then
echo "unpacker appears to have produced no directories";
exit 1;
fi;
echo "source root is $sourceRoot";
if [ "$dontMakeSourcesWritable" != 1 ]; then
chmod -R u+w "$sourceRoot";
fi;
runHook postUnpack
}
you can also modify the configureFlags prefix:
export configureFlags="--prefix=$out --with-readline"
Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.
nix channels
nix channels can be used in parallel with your new local repositories, see its nix-channel-documentation
Testing Package Updates with Nox
If you are updating a package's version, you can use nox to make sure all packages that depend on the updated package still compile correctly.
First make sure it is in your environment:
nix-env -i nox
You can run nox against uncommited changes to a nixpkgs repository:
cd ~/.nix-defexpr
nox-review wip
If you have already commited your changes and created a pull request, you can use the pr command:
nox-review pr 5341