Nixpkgs/Create and debug packages

This article describes how to work with the nix related repositories to add new packages, edit and debug existing packages. For details on the NixOS module system see NixOS:Modules. NixOS:extend_NixOS explains how to write, test and debug your own modules.

There is a chapter about hacking packages and modules in the NixOS manual: http://nixos.org/nixos/manual/index.html#ch-development

Writing packages is covered in http://nixos.org/nixpkgs/manual and writing modules in http://nixos.org/nixos/manual

The nix repositories are hosted here: https://github.com/nixos

Basics

The code for nix packages is managed in the nixpkgs repository. NixOS services, and other system configuration options are managed in the nixos sub-directory of the nixpkgs repository.

The steps to take for your first change should look something like this:

Fork the repo (e.g. click the fork button on https://github.com/nixos/nixpkgs).
Clone your fork git clone git@github.com:YOUR-GITHUB-ACCOUNT-NAME/nixpkgs.git
Hack hack hack
Push your changes to your fork
Open a pull request
Profit!

This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)

How to install from the local repository

For the sake of this article, let's set an environment variable which points to the directory where we've cloned our repo.

$ export NIXPKGS=/path/to/clone/of/nixpkgs

make some changes ...

example: list all available software from the local repository $NIXREPOS/nixpkgs

$ nix-env -f $NIXPKGS -qaP '*'

example: install software from local repository

$ nix-env -f $NIXPKGS -i python-urlgrabber

example: update the system based on your local $NIXREPOS

$ nixos-rebuild -I nixos=$NIXPKGS/nixos -I nixpkgs=$NIXPKGS switch

example: build an expression and put the output in to `pwd`/results

$ nix-build $NIXPKGS -A irssi

example: get an environment which is used to build irssi (also see nix-shell)

$ nix-build $NIXPKGS --run-env -A irssi

example: get a persistent environment which is used to build irssi

$ nix-build $NIXPKGS --run-env -A irssi --add-root

Tracking upstream changes and avoiding extra rebuilding

You have forked the relevant nix repository, but you will want to track changes in the upstream nix repo too. You can add a remote, and a corresponding branch for this.

git remote add upstream https://github.com/NixOS/nixpkgs.git

You can create a branch to track the upstream master branch:

$ git checkout -b upstream-master upstream/master
$ git pull

This will put you into a branch with all the latest changes. Hydra, the build farm, regularly creates binaries, but, since people are constantly contributing to the nix repositories, it is usually the case that there are changes in the master branch which have not yet made it into the binary channel. To take advantage of available binaries you can switch to the revision which produced the binaries in your current system and apply your changes from there. You can use `nixos-version` to see the relevant short revision hash:

$ nixos-version 
14.11pre52727.5d97886 (Caterpillar)
${NixOS release}.${nixpkgs revision} 
(since the git-repo called nixos was merged into nixpkgs)

$ nixos-version 
13.07pre4871_18de9f6-3c35dae (Aardvark)
${NixOS release}_${NixOS revision}-${nixpkgs revision}

This string shows the Nixos release number (13.07pre4871) followed by the nixos revision used to produce your current system (18de9f6) followed by the nixpkgs revision (3c35dae).

$ git branch
upstream-master
$ git checkout -b nixpkgs-channel 3c35dae
Switched to a new branch 'nixpkgs-channel'
$ git checkout -b my-new-pkg
Switched to a new branch 'my-new-pkg'

After making some changes you can commit them into your local repo:

$ git add foo
$ git commit

Then you push your changes to your fork:

$ git push origin my-new-pkg

You can use this to open a pull request on github.

If some time has passed since you have created your fork, you will want to merge your changes with upstream and test that it still works.

git fetch upstream
git merge upstream

If your merge then fails because someone else has made the same change (for example, someone else also packaged a library you have just packed for the program you want to get into nixpkgs), then you can do this:

git rebase -i HEAD~10

there select the edit mode for your commit and remove the your code which added the library. **Warning: only use 'git rebase' on your commits, which have not been pushed and nobody else is working with already!**

Next you have to test if your program works with the library packaged from someone else, then do:

git checkout master
git log --stat

and pick the commit where the library was added. Finally cherry-pick that commit into your branch:

git checkout my-new-pkg
git cherry-pick 5d97886a6a545fb20495e0837cc50fa63d2a80e1

Afterwards do your usual tests and if needed also make modifications to the library but keep in mind that this might break the other use-case of that library and if in doubt check that as well.

Using nix-shell for package development

nix-shell is a command which drops you into the build environment for a package. This is convenient for writing and debugging nix expressions. Nix-shell requires nix-1.6.x although running nix-build --run-env produces a similar environment.

mkdir -p /tmp/nix-shell-bc
cd /tmp/nix-shell-bc
nix-shell $NIXREPOS/nixpkgs -A bc
export out=/tmp/foo/out

now we have find out which phases are specified for this package:

typeset -f genericBuild | grep 'phases='
phases="$prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase installCheckPhase $preDistPhases distPhase $postPhases";

The phases can be defined either as a string to be eval'ed or as a shell function, this is how Nix invokes it.

so when developing you need to run these phases in a row:

unpackPhase
patchPhase
configurePhase
buildPhase
checkPhase
installPhase
fixPhase
installCheckPhase
installPhase
distPhase

Note: you do not need to run $preConfigurePhase explicitly as it is run, when running configurePhase already.

list all functions which are declared in set:

typeset -F
declare -f addCVars
declare -f addToCrossEnv
declare -f addToNativeEnv
declare -f addToSearchPath
declare -f addToSearchPathWithCustomDelimiter
declare -f buildPhase
declare -f checkPhase
declare -f closeNest
declare -f command_not_found_handle
declare -f configurePhase
declare -f distPhase
declare -f dumpVars
declare -f ensureDir
declare -f exitHandler
declare -f findInputs
declare -f fixLibtool
declare -f fixupPhase
declare -f genericBuild
declare -f header
declare -f installBin
declare -f installCheckPhase
declare -f installPhase
declare -f patchELF
declare -f patchPhase
declare -f patchShebangs
declare -f runHook
declare -f showPhaseHeader
declare -f startNest
declare -f stopNest
declare -f stripDirs
declare -f stripHash
declare -f substitute
declare -f substituteAll
declare -f substituteAllInPlace
declare -f substituteInPlace
declare -f unpackFile
declare -f unpackPhase

If the phase has been defined as a function, to list a particular function type:

typeset -f unpackPhase

Otherwise, if it was a string, simply echo the variable related to it

echo $unpackPhase

In either case, you can see the code that is about to be executed for each phase:

typeset -f unpackPhase
unpackPhase ()
{
    runHook preUnpack;
    if [ -z "$srcs" ]; then
        if [ -z "$src" ]; then
            echo 'variable $src or $srcs should point to the source';
            exit 1;
        fi;
        srcs="$src";
    fi;
    local dirsBefore="";
    for i in *;
    do
        if [ -d "$i" ]; then
            dirsBefore="$dirsBefore $i ";
        fi;
    done;
    for i in $srcs;
    do
        unpackFile $i;
    done;
    if [ -n "$setSourceRoot" ]; then
        runHook setSourceRoot;
    else
        if [ -z "$sourceRoot" ]; then
            sourceRoot=;
            for i in *;
            do
                if [ -d "$i" ]; then
                    case $dirsBefore in
                        *\ $i\ *)

                        ;;
                        *)
                            if [ -n "$sourceRoot" ]; then
                                echo "unpacker produced multiple directories";
                                exit 1;
                            fi;
                            sourceRoot="$i"
                        ;;
                    esac;
                fi;
            done;
        fi;
    fi;
    if [ -z "$sourceRoot" ]; then
        echo "unpacker appears to have produced no directories";
        exit 1;
    fi;
    echo "source root is $sourceRoot";
    if [ "$dontMakeSourcesWritable" != 1 ]; then
        chmod -R u+w "$sourceRoot";
    fi;
    runHook postUnpack
}

you can also modify the configureFlags prefix:

export configureFlags="--prefix=$out --with-readline"

Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.

nix channels

nix channels can be used in parallel with your new local repositories, see its nix-channel-documentation

Testing Package Updates with Nox

If you are updating a package's version, you can use nox to make sure all packages that depend on the updated package still compile correctly.

First make sure it is in your environment:

nix-env -i nox

You can run nox against uncommited changes to a nixpkgs repository:

cd ~/.nix-defexpr
nox-review wip

If you have already commited your changes and created a pull request, you can use the pr command:

nox-review pr 5341