Packaging/Binaries

Downloading a binary on NixOS and trying to run it will almost never work. This is due to hard-coded paths in the executable. Unfortunately almost all unfree and proprietary comes in binary form - the main reason to include binaries is because no source code is available. This tutorial will guide you through packaging a binary executable.

This tutorial is about how to patch the executable with patchelf which is sufficient for most cases. If no source is available for a program patchelf is the preferred way in nixpkgs to add support for the package.

However sometimes this is not enough, more hardcoded paths may be scattered all over the place. For this you may need to set up an FHSUserEnv, a Linux Standard Base style directory structure with a final chroot call to fixate all paths. For a tutorial on how to use this technique, check out Anders Papittos blog post on installing debian packages in nixos.

Starting Point

We want to package a tool called "MasterPDFEditor", the package for debian can be found at [1] (archive.org mirror). This tutorial assumes you run nixos-17.09 or later (for nix-index to work).

Let's download the archive and unpack the archive in a nix shell for testing:

$ nix-shell -p binutils stdenv wget dpkg nix-index \
   stdenv.cc # stdenv.cc is required for setting $NIX_CC env
$ wget https://web.archive.org/web/20170914112947/http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb
# we extract data.tar.xz from the deb package and untar it
$ ar p master-pdf-editor-4.3.10_qt5.amd64.deb data.tar.xz | tar xJ
ls opt/master-pdf-editor-4/
fonts  license.txt       masterpdfeditor4.png  templates
lang   masterpdfeditor4  stamps
# running the executable does not `just work`
$ opt/master-pdf-editor-4/masterpdfeditor4
bash: opt/master-pdf-editor-4/masterpdfeditor4: No such file or directory

The Dynamic Loader

The binary has the dynamic loader ("ELF interpreter") set to a static path which is in general not available under NixOS: /lib64/ld-linux-x86-64.so.2

We can use patchelf to show and set the library path and dynamic linker appropriately:

$ cd opt/master-pdf-editor-4/
# the current interpreter
$ patchelf --print-interpreter masterpdfeditor4
# all the needed libraries
/lib64/ld-linux-x86-64.so.2

We start by patching the interpreter and see if the executable already starts:

$ patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      masterpdfeditor4
$ ./masterpdfeditor4
./masterpdfeditor4: error while loading shared libraries: libsane.so.1: cannot open shared object file: No such file or directory

We went beyond the No such file or directory error, means that setting the interpreter worked! Unfortunately we have new errors which show that some shared objects cannot be found.

Extra Dynamic Libraries

Just like before, the executable simply expects libraries to be available in the Linux Standard Base(LSB) directories. Because Nix tries to avoid impurity and global directories we also have to explicitly set the libraries.

Patchelf can show the required libraries but we still have to set them manually.

$ patchelf --print-needed masterpdfeditor4
libdl.so.2
librt.so.1
libsane.so.1
libQt5Svg.so.5
libQt5PrintSupport.so.5
libQt5Widgets.so.5
libQt5Gui.so.5
libQt5Network.so.5
libQt5Core.so.5
libGL.so.1
libpthread.so.0
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libc.so.6
# now that the interpreter path is patched we can use ldd to find the libraries which are currently not found (check '=> not found'):
$ ldd masterpdfeditor4 | grep 'not found'
	libsane.so.1 => not found
	libQt5Svg.so.5 => not found
	libQt5PrintSupport.so.5 => not found
	libQt5Widgets.so.5 => not found
	libQt5Gui.so.5 => not found
	libQt5Network.so.5 => not found
	libQt5Core.so.5 => not found
	libstdc++.so.6 => not found

All the libraries which are not yet found are the ones we need to add to the runtime search path of the executable (RPATH). Again we can use patchelf to do this. We will be using nix-index for finding the files we are looking for:

# we generate the database index of all files in our channel first
$ nix-index
+ querying available packages
+ generating index: 41977 paths found :: 15957 paths not in binary cache :: 00000 paths in queue 
+ wrote index of 21,621,061 bytes
# we use the power of nix-locate to find the packages which contain the file:
$ nix-locate -1 -w lib/libsane.so.1
(wineStaging.out)
saneBackends.out
saneBackendsGit.out
$ nix-locate -1 -w lib/libQt5Svg.so.5
libsForQt56.qtinstaller.out
qt56.qtsvg.out
qt5.qtsvg.out
qt56.full.out
libsForQt5.qtinstaller.out
$ nix-locate -1 -w lib/libQt5PrintSupport.so.5
robomongo.out
libsForQt56.qtinstaller.out
qt5.qtbase.out
qt56.qtbase.out
qt56.full.out
libsForQt5.qtinstaller.out
$ nix-locate -1 -w lib/libstdc++.so.6
# ...
# libsdtc++.so.6 is `special`, it resides in stdenv.cc.cc.lib (see other packages)

Unfortunately there is no "right" way to choose which package to actually take, you can check out other derivations by grepping in nixpkgs.

Tip

in case there are too many .so libraries, you may save time by automated approach, executing nix-de-generate for target application, which will produce list of nix packages (in nix expression) to satisfy dependencies of target application.

The next step is to create a library path for all these packages. We use nix-repl to resolve the paths:

$ nix-repl
# load all variables from nixpkgs
nix-repl> :l <nixpkgs>
# .out can be omitted because this is the default output for all packages
# makeLibraryPath outputs the correct path for each package to use as rpath
nix-repl> with pkgs; lib.makeLibraryPath [ saneBackends qt5.qtbase qt5.qtsvg stdenv.cc.cc.lib ]
"/nix/store/7lbi3gn351j4hix3dqhis58adxbmvbxa-sane-backends-1.0.25/lib:/nix/store/0990ls1p2nnxq6605mr9lxpps8p7qvw7-qtbase-5.9.1/lib:/nix/store/qzhn2svk71886fz3a79vklps781ah0lb-qtsvg-5.9.1/lib:/nix/store/snc31f0alikhh3a835riyqhbsjm29vki-gcc-6.4.0-lib/lib"

Let's try out the path we generated:

$ patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" --set-rpath /nix/store/7lbi3gn351j4hix3dqhis58adxbmvbxa-sane-backends-1.0.25/lib:/nix/store/0990ls1p2nnxq6605mr9lxpps8p7qvw7-qtbase-5.9.1/lib:/nix/store/qzhn2svk71886fz3a79vklps781ah0lb-qtsvg-5.9.1/lib:/nix/store/snc31f0alikhh3a835riyqhbsjm29vki-gcc-6.4.0-lib/lib   masterpdfeditor4
$ ./masterpdfeditor4
# SUCCESS!!!

Creating the Derivation for upstream Packaging

Packaging is straight forward. We just have to add all the steps we did into a simple derivation file. We call it default.nix and store it in the checked out nixpkgs repository at pkgs/applications/office/master-pdf-editor The content looks like this:

{ stdenv, lib, qt5, saneBackends, makeWrapper, fetchurl }:
stdenv.mkDerivation rec {
  name = "master-pdf-editor-${version}";
  version = "4.3.10";

  src = fetchurl {
    url = "http://get.code-industry.net/public/${name}_qt5.amd64.deb";
    sha256 = "1z26qjhbiyz33rm7mp8ycgl5ka0v3v5lv5i5v0b5mx35arvx2zzy";
  };
  sourceRoot = ".";
  unpackCmd = ''
    ar p "$src" data.tar.xz | tar xJ
  '';

  buildPhase = ":";   # nothing to build

  installPhase = ''
    mkdir -p $out/bin
    cp -R usr/share opt $out/
    # fix the path in the desktop file
    substituteInPlace \
      $out/share/applications/masterpdfeditor4.desktop \
      --replace /opt/ $out/opt/
    # symlink the binary to bin/
    ln -s $out/opt/master-pdf-editor-4/masterpdfeditor4 $out/bin/masterpdfeditor4
  '';
  preFixup = let
    # we prepare our library path in the let clause to avoid it become part of the input of mkDerivation
    libPath = lib.makeLibraryPath [
      qt5.qtbase        # libQt5PrintSupport.so.5
      qt5.qtsvg         # libQt5Svg.so.5
      stdenv.cc.cc.lib  # libstdc++.so.6
      saneBackends      # libsane.so.1
    ];
  in ''
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath "${libPath}" \
      $out/opt/master-pdf-editor-4/masterpdfeditor4
  '';

  meta = with stdenv.lib; {
    homepage = https://code-industry.net/masterpdfeditor/;
    description = "a multifunctional PDF Editor";
    license = licenses.proprietary;
    platforms = platforms.linux;
    maintainers = [ your_name ];
  };
}

Because we created a derivation which is meant to be called by callPackage we can build the package now only via: nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })' --keep-failed --no-out-link

Add the package to nixpkgs

In order to add this new package to nixpkgs and be able to install it via nix-build -A pkgs.master-pdf-editor you need to add it to pkgs/top-level/all-packages.nix:

  ...
  masscan = callPackage ../tools/security/masscan { };

  master-pdf-editor = callPackage ../applications/office/master-pdf-editor {};

  meson = ../development/tools/build-managers/meson { };
  ...

Creating a Pull Request

With this new package you can create a pull request for nixpkgs. Be aware that binary distributions are frowned upon if the source is available.