Nixpkgs/Create and debug packages
This article describes how to work with the nix related repositories to add new packages, edit and debug existing packages. For details on the NixOS module system see NixOS:Modules. NixOS:extend_NixOS explains how to write, test and debug your own modules.
The nix repositories are hosted here: https://github.com/nixos
Basics
The code for nix packages is managed in the nixpkgs repository. NixOS services, and other system configuration options are managed in the nixos sub-directory of the nixpkgs repository.
The steps to take for your first change should look something like this:
- Fork the repo (e.g. click the fork button on https://github.com/nixos/nixpkgs).
- Clone your fork
git clone git@github.com:YOUR-GITHUB-ACCOUNT-NAME/nixpkgs.git - Hack hack hack
- Push your changes to your fork
- Open a pull request
- Profit!
This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)
How to install from the local repository
For the sake of this article, let's set an environment variable which points to the directory where we've cloned our repo.
$ export NIXPKGS=/path/to/clone/of/nixpkgs
make some changes ...
example: list all available software from the local repository $NIXREPOS/nixpkgs
$ nix-env -f $NIXPKGS -qaP '*'
example: install software from local repository
$ nix-env -f $NIXPKGS -i python-urlgrabber
example: update the system based on your local $NIXREPOS
$ nixos-rebuild -I nixos=$NIXPKGS/nixos -I nixpkgs=$NIXPKGS switch
example: build an expression and put the output in to `pwd`/results
$ nix-build $NIXPKGS -A irssi
example: get an environment which is used to build irssi (also see nix-shell)
$ nix-build $NIXPKGS --run-env -A irssi
example: get a persistent environment which is used to build irssi
$ nix-build $NIXPKGS --run-env -A irssi --add-root
Tracking upstream changes and avoiding extra rebuilding
You have forked the relevant nix repository, but you will want to track changes in the upstream nix repo too. You can add a remote, and a corresponding branch for this.
git remote add upstream https://github.com/NixOS/nixpkgs.git
You can create a branch to track the upstream master branch:
$ git checkout -b upstream-master upstream/master $ git pull
This will put you into a branch with all the latest changes. Hydra, the build farm, regularly creates binaries, but, since people are constantly contributing to the nix repositories, it is usually the case that there are changes in the master branch which have not yet made it into the binary channel. To take advantage of available binaries you can switch to the revision which produced the binaries in your current system and apply your changes from there. You can use `nixos-version` to see the relevant short revision hash:
$ nixos-version
14.11pre52727.5d97886 (Caterpillar)
${NixOS release}.${nixpkgs revision} (since the git-repo called nixos was merged into nixpkgs)
$ nixos-version
13.07pre4871_18de9f6-3c35dae (Aardvark)
${NixOS release}_${NixOS revision}-${nixpkgs revision}
This string shows the Nixos release number (13.07pre4871) followed by the nixos revision used to produce your current system (18de9f6) followed by the nixpkgs revision (3c35dae).
$ git branch upstream-master $ git checkout -b nixpkgs-channel 3c35dae Switched to a new branch 'nixpkgs-channel' $ git checkout -b my-new-pkg Switched to a new branch 'my-new-pkg'
After making some changes you can commit them into your local repo:
$ git add foo $ git commit
Then you push your changes to your fork:
$ git push origin my-new-pkg
You can use this to open a pull request on github.
If some time has passed since you have created your fork, you will want to merge your changes with upstream and test that it still works.
git fetch upstream git merge upstream
If your merge then fails because someone else has made the same change (for example, someone else also packaged a library you have just packed for the program you want to get into nixpkgs), then you can do this:
git rebase -i HEAD~10
there select the edit mode for your commit and remove the your code which added the library. **Warning: only use 'git rebase' on your commits, which have not been pushed and nobody else is working with already!**
Next you have to test if your program works with the library packaged from someone else, then do:
git checkout master git log --stat
and pick the commit where the library was added. Finally cherry-pick that commit into your branch:
git checkout my-new-pkg git cherry-pick 5d97886a6a545fb20495e0837cc50fa63d2a80e1
Afterwards do your usual tests and if needed also make modifications to the library but keep in mind that this might break the other use-case of that library and if in doubt check that as well.
Using nix-shell for package development
nix-shell is a command which drops you into the build environment for a package. This is convenient for writing and debugging nix expressions. Nix-shell requires nix-1.6.x although running nix-build --run-env produces a similar environment.
mkdir -p /tmp/nix-shell-bc cd /tmp/nix-shell-bc nix-shell $NIXREPOS/nixpkgs -A bc export out=/tmp/foo/out
now we have find out which phases are specified for this package:
typeset -f genericBuild | grep 'phases=' phases="$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases";
The phases can be defined either as a string to be eval'ed or as a shell function, this is how Nix invokes it.
so when developing you need to run these phases in a row:
unpackPhase patchPhase configurePhase buildPhase checkPhase installPhase fixPhase installCheckPhase installPhase distPhase
list all functions which are declared in set:
typeset -F declare -f addCVars declare -f addToCrossEnv declare -f addToNativeEnv declare -f addToSearchPath declare -f addToSearchPathWithCustomDelimiter declare -f buildPhase declare -f checkPhase declare -f closeNest declare -f command_not_found_handle declare -f configurePhase declare -f distPhase declare -f dumpVars declare -f ensureDir declare -f exitHandler declare -f findInputs declare -f fixLibtool declare -f fixupPhase declare -f genericBuild declare -f header declare -f installBin declare -f installCheckPhase declare -f installPhase declare -f patchELF declare -f patchPhase declare -f patchShebangs declare -f runHook declare -f showPhaseHeader declare -f startNest declare -f stopNest declare -f stripDirs declare -f stripHash declare -f substitute declare -f substituteAll declare -f substituteAllInPlace declare -f substituteInPlace declare -f unpackFile declare -f unpackPhase
If the phase has been defined as a function, to list a particular function type:
typeset -f unpackPhase
Otherwise, if it was a string, simply echo the variable related to it
echo $unpackPhase
In either case, you can see the code that is about to be executed for each phase:
typeset -f unpackPhase
unpackPhase ()
{
runHook preUnpack;
if [ -z "$srcs" ]; then
if [ -z "$src" ]; then
echo 'variable $src or $srcs should point to the source';
exit 1;
fi;
srcs="$src";
fi;
local dirsBefore="";
for i in *;
do
if [ -d "$i" ]; then
dirsBefore="$dirsBefore $i ";
fi;
done;
for i in $srcs;
do
unpackFile $i;
done;
if [ -n "$setSourceRoot" ]; then
runHook setSourceRoot;
else
if [ -z "$sourceRoot" ]; then
sourceRoot=;
for i in *;
do
if [ -d "$i" ]; then
case $dirsBefore in
*\ $i\ *)
;;
*)
if [ -n "$sourceRoot" ]; then
echo "unpacker produced multiple directories";
exit 1;
fi;
sourceRoot="$i"
;;
esac;
fi;
done;
fi;
fi;
if [ -z "$sourceRoot" ]; then
echo "unpacker appears to have produced no directories";
exit 1;
fi;
echo "source root is $sourceRoot";
if [ "$dontMakeSourcesWritable" != 1 ]; then
chmod -R u+w "$sourceRoot";
fi;
runHook postUnpack
}
you can also modify the configureFlags prefix:
export configureFlags="--prefix=$out --with-readline"
Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.
nix channels
nix channels can be used in parallel with your new local repositories, see its nix-channel-documentation
Testing Package Updates with Nox
If you are updating a package's version, you can use nox to make sure all packages that depend on the updated package still compile correctly.
First make sure it is in your environment:
nix-env -i nox
You can run nox against uncommited changes to a nixpkgs repository:
cd ~/.nix-defexpr nox-review wip
If you have already commited your changes and created a pull request, you can use the pr command:
nox-review pr 5341