NixOS Wiki

Packaging/Binaries: Difference between revisions

← Older edit
VisualWikitext
imported>Makefu
link to archive.org
m update url to wayback machine url
 
(34 intermediate revisions by 15 users not shown)
Line 1: Line 1:
Downloading a binary on NixOS and trying to run it will almost never work. This is due to hard-coded paths in the executable. Unfortunately almost all unfree and proprietary comes in binary form - the main reason to include binaries is because no source code is available.
Downloading and attempting to run a binary on NixOS will almost never work. This is due to hard-coded paths in the executable. Unfortunately, almost all unfree and proprietary software comes in binary form - the main reason to include binaries is because no source code is available.
This tutorial will guide you through packaging a binary executable.
This tutorial will guide you through packaging a binary executable.


If a downloaded binary is not packaged for NixOS then running the binary may fail like the following:
<syntaxHighlight lang=bash>
$ ~/.local/share/nvim/mason/packages/rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu  --help
bash: /home/gdforj/.local/share/nvim/mason/packages/rust-analyzer/rust-analyzer-x86_64-unknown-linux-gnu: cannot execute: required file not found
</syntaxHighlight>
Which indicates the interpreter, [[#The Dynamic Loader|the dynamic loader]] in this case, could not be located.
== Using AutoPatchelfHook ==
<code>autoPatchelfHook</code> is the current (as of 19.09) preferred way to package binaries.
<syntaxHighlight lang=nix>
{ stdenv, lib
, fetchurl
, alsaLib
, openssl
, zlib
, pulseaudio
, autoPatchelfHook
}:
stdenv.mkDerivation rec {
  pname = "studio-link";
  version = "21.07.0";
  src = fetchurl {
    url = "https://download.studio.link/releases/v${version}-stable/linux/studio-link-standalone-v${version}.tar.gz";
    hash = "sha256-4CkijAlenhht8tyk3nBULaBPE0GBf6DVII699/RmmWI=";
  };
  nativeBuildInputs = [
    autoPatchelfHook
  ];
  buildInputs = [
    alsaLib
    openssl
    zlib
    pulseaudio
  ];
  sourceRoot = ".";
  installPhase = ''
    runHook preInstall
    install -m755 -D studio-link-standalone-v${version} $out/bin/studio-link
    runHook postInstall
  '';
  meta = with lib; {
    homepage = "https://studio-link.com";
    description = "Voip transfer";
    platforms = platforms.linux;
  };
}
</syntaxHighlight>
See [https://github.com/NixOS/nixpkgs/commit/ea5787ad5291ee1c131326cb9c9fec03d359edff this commit], or the example in method 5 of [https://unix.stackexchange.com/a/522823 this answer], for more details.
=== Additional Library Paths ===
Some packages may put their library files somewhere other than <code>$out/lib/</code> in the install phase, while <code>autoPatchelfHook</code> by default only looks in that one directory for every package in <code>buildInputs</code>.  You can tell <code>autoPatchelfHook</code> to look in additional directories by passing it to the command <code>addAutoPatchelfSearchPath</code> in any phase before <code>autoPatchelfHook</code> is run.
For example,
<pre>
  preBuild = ''
    addAutoPatchelfSearchPath ${jre8}/lib/openjdk/jre/lib/
  '';
</pre>
== Manual Method ==
This sections describes the manual method of packaging a binary. It serves as a reference on the packaging issue with binaries and how these issues can be solved in nix derivations.
This tutorial is about how to patch the executable with <code>patchelf</code> which is sufficient for most cases. If no source is available for a program patchelf is the preferred way in nixpkgs to add support for the package.
This tutorial is about how to patch the executable with <code>patchelf</code> which is sufficient for most cases. If no source is available for a program patchelf is the preferred way in nixpkgs to add support for the package.


However sometimes this is not enough, more hardcoded paths may be scattered all over the place. For this you may need to set up an FHSUserEnv, a Linux Standard Base style directory structure with a final <code>chroot</code> call to fixate all paths. For a tutorial on how to use this technique, check out [https://web.archive.org/web/20170313102700/http://anderspapitto.com/posts/2015-02-28-deb-installation-nixos.html Anders Papittos blog post on installing debian packages in nixos].
However sometimes this is not enough, more hardcoded paths may be scattered all over the place. For this you may need to set up an FHSUserEnv, a Linux Standard Base style directory structure with a final <code>chroot</code> call to fixate all paths. For a tutorial on how to use this technique, check out [https://web.archive.org/web/20240125095859/http://reflexivereflection.com/posts/2015-02-28-deb-installation-nixos.html Anders Papittos blog post on installing debian packages in nixos].
== Starting Point ==
=== Starting Point ===
We want to package a tool called "MasterPDFEditor", the package for debian can be found at [http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb] ([https://web.archive.org/web/20170914112947/http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb archive.org mirror]).
We want to package a tool called "MasterPDFEditor", the package for debian can be found at [http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb] ([https://web.archive.org/web/20170914112947/http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb archive.org mirror]).
This tutorial assumes you run nixos-17.09 or later (for <code>nix-index</code> to work).  
This tutorial assumes you run nixos-17.09 or later (for <code>nix-index</code> to work).  
Line 14: Line 86:
   stdenv.cc # stdenv.cc is required for setting $NIX_CC env
   stdenv.cc # stdenv.cc is required for setting $NIX_CC env
$ wget https://web.archive.org/web/20170914112947/http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb
$ wget https://web.archive.org/web/20170914112947/http://get.code-industry.net/public/master-pdf-editor-4.3.10_qt5.amd64.deb
# we extract data.tar.xz from the deb package and untar it
$ # we extract data.tar.xz from the deb package and untar it
$ ar p master-pdf-editor-4.3.10_qt5.amd64.deb data.tar.xz | tar xJ
$ dpkg-deb -x master-pdf-editor-4.3.10_qt5.amd64.deb .
ls opt/master-pdf-editor-4/
$ ls opt/master-pdf-editor-4/
fonts  license.txt      masterpdfeditor4.png  templates
fonts  license.txt      masterpdfeditor4.png  templates
lang  masterpdfeditor4  stamps
lang  masterpdfeditor4  stamps
# running the executable does not `just work`
$ # running the executable does not `just work`
$ opt/master-pdf-editor-4/masterpdfeditor4
$ opt/master-pdf-editor-4/masterpdfeditor4
bash: opt/master-pdf-editor-4/masterpdfeditor4: No such file or directory
bash: opt/master-pdf-editor-4/masterpdfeditor4: No such file or directory
</syntaxHighlight>
</syntaxHighlight>
== The Dynamic Loader ==
== The Dynamic Loader ==


Line 68: Line 141:
libgcc_s.so.1
libgcc_s.so.1
libc.so.6
libc.so.6
# now that the interpreter path is patched we can use ldd to find the libraries which are currently not found (check '=> not found'):
$ # now that the interpreter path is patched we can use ldd to find the libraries which are currently not found (check '=> not found'):
$ ldd masterpdfeditor4 | grep 'not found'
$ ldd masterpdfeditor4 | grep 'not found'
libsane.so.1 => not found
libsane.so.1 => not found
Line 82: Line 155:
All the libraries which are not yet found are the ones we need to add to the runtime search path of the executable (RPATH). Again we can use <code>patchelf</code> to do this. We will be using <code>nix-index</code> for finding the files we are looking for:
All the libraries which are not yet found are the ones we need to add to the runtime search path of the executable (RPATH). Again we can use <code>patchelf</code> to do this. We will be using <code>nix-index</code> for finding the files we are looking for:
<syntaxHighlight lang="console">
<syntaxHighlight lang="console">
# we generate the database index of all files in our channel first
$ # we generate the database index of all files in our channel first
$ nix-index
$ nix-index
+ querying available packages
+ querying available packages
Line 114: Line 187:
The next step is to create a library path for all these packages. We use nix-repl to resolve the paths:
The next step is to create a library path for all these packages. We use nix-repl to resolve the paths:
<syntaxHighlight lang="console">
<syntaxHighlight lang="console">
$ nix-repl
$ nix repl '<nixpkgs>'
# load all variables from nixpkgs
nix-repl> :l <nixpkgs>
# .out can be omitted because this is the default output for all packages
# .out can be omitted because this is the default output for all packages
# makeLibraryPath outputs the correct path for each package to use as rpath
# makeLibraryPath outputs the correct path for each package to use as rpath
Line 132: Line 203:


== Creating the Derivation for upstream Packaging ==
== Creating the Derivation for upstream Packaging ==
Packaging is straight forward. We just have to add all the steps we did into a simple derivation file. We call it <code>default.nix</code> and store it in the checked out nixpkgs repository at <code>pkgs/applications/office/master-pdf-editor</code>
Packaging is straight forward. We just have to add all the steps we did into a simple derivation file. We call it <code>package.nix</code> and store it in the checked out nixpkgs repository at <code>pkgs/by-name/ma/master-pdf-editor</code>
The content looks like this:
The content looks like this:
<syntaxHighlight lang="nix">
<syntaxHighlight lang="nix">
Line 141: Line 212:


   src = fetchurl {
   src = fetchurl {
     url = "http://get.code-industry.net/public/${name}_qt5.amd64.deb";
     url = "http://get.code-industry.net/public/master-pdf-editor-${version}_qt5.amd64.deb";
     sha256 = "1z26qjhbiyz33rm7mp8ycgl5ka0v3v5lv5i5v0b5mx35arvx2zzy";
     hash = "sha256-1z26qjhbiyz33rm7mp8ycgl5ka0v3v5lv5i5v0b5mx35arvx2zzy";
   };
   };
   sourceRoot = ".";
   sourceRoot = ".";
   unpackCmd = ''
   unpackCmd = "dpkg-deb -x master-pdf-editor-${version}_qt5.amd64.deb .";
    ar p "$src" data.tar.xz | tar xJ
  '';


   buildPhase = ":";  # nothing to build
   dontConfigure = true;
   dontBuild = true;


   installPhase = ''
   installPhase = ''
    runHook preInstall
     mkdir -p $out/bin
     mkdir -p $out/bin
     cp -R usr/share opt $out/
     cp -R usr/share opt $out/
Line 160: Line 232:
     # symlink the binary to bin/
     # symlink the binary to bin/
     ln -s $out/opt/master-pdf-editor-4/masterpdfeditor4 $out/bin/masterpdfeditor4
     ln -s $out/opt/master-pdf-editor-4/masterpdfeditor4 $out/bin/masterpdfeditor4
    runHook postInstall
   '';
   '';
   preFixup = let
   preFixup = let
Line 176: Line 250:
   '';
   '';


   meta = with stdenv.lib; {
   meta = with lib; {
     homepage = https://code-industry.net/masterpdfeditor/;
     homepage = https://code-industry.net/masterpdfeditor/;
     description = "a multifunctional PDF Editor";
     description = "a multifunctional PDF Editor";
     license = licenses.proprietary;
     license = licenses.unfree;
     platforms = platforms.linux;
     platforms = platforms.linux;
     maintainers = [ your_name ];
     maintainers = [ your_name ];
Line 187: Line 261:


Because we created a derivation which is meant to be called by callPackage we can build the package now only via:
Because we created a derivation which is meant to be called by callPackage we can build the package now only via:
<code>nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })' --keep-failed --no-out-link</code>
<code>nix-build -E '((import <nixpkgs> {}).callPackage (import ./package.nix) { })' --keep-failed --no-out-link</code>
 
== Creating a Pull Request ==
 
With this new package you can [[Pull request|create a pull request for nixpkgs]]. Be aware that binary distributions are frowned upon if the source is available.
 
 
== autoPatchelfHook ==
 
<code>autoPatchelfHook</code> can make the manual written patchelf invocations unnecessary. [[Category:Tutorial]]
[[Category:nixpkgs]]
 
== Wrong file paths ==
 
Some programs will try to access hard-coded FHS file paths like <code>/usr/lib</code> or <code>/opt</code>. Mostly though, this will produce silent <code>No such file</code> errors, which can break the program.
 
To make these errors visible, we can use <code>strace</code>


== Add the package to nixpkgs ==
<pre>
In order to add this new package to nixpkgs and be able to install it via <code>nix-build -A pkgs.master-pdf-editor</code> you need to add it to <code>pkgs/top-level/all-packages.nix</code>:
strace --trace=file,process --follow-forks --string-limit=200 \
<syntaxHighlight lang=nix>
  ./result/bin/some-program 2>strace.log
  ...
 
  masscan = callPackage ../tools/security/masscan { };
cat strace.log | grep -e 'No such file' -e 'execve("' \
  | grep -v -E -e '(open|stat|access)(at)?\(.*"/nix/store/' \
    -e resumed -e '/etc/ld-nix.so.preload'
</pre>
 
=== Example ===
 
<pre>
[pid 357679] openat(AT_FDCWD, "/opt/brother/Printers/hll3210cw/inf/lut/0600-k_cache17.bin", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 357679] openat(AT_FDCWD, "0600-k_cache17.bin", O_RDONLY) = -1 ENOENT (No such file or directory)
</pre>
 
This means: process <code>357679</code> is trying to open file <code>0600-k_cache17.bin</code> either from the hard-coded path in <code>/opt/brother/Printers/hll3210cw/inf/lut</code> or from the current workdir.
So, as a quick fix, we could change the working directory of the wrapped binary with [https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/make-wrapper.sh makeWrapper]:
 
<pre>
makeWrapper <executable> <out_path> --run 'cd /nix/store/path/to/workdir'
</pre>


  master-pdf-editor = callPackage ../applications/office/master-pdf-editor {};
=== Finding the failing process ===
How can we find which process is throwing the <code>No such file</code> error? Let's search for the process ID and the <code>exec</code> syscall:


  meson = ../development/tools/build-managers/meson { };
<pre>
  ...
cat strace.log | grep 357679 | grep exec
</syntaxHighlight>


== Creating a Pull Request ==
[pid 357679] execve("/nix/store/ybl1pacslmhci9zy5qv95hshdgz6ihjl-brother-hll3210cw-1.0.2-0/opt/brother/Printers/hll3210cw/lpd/brhll3210cwfilter", ["/nix/store/ybl1pacslmhci9zy5qv95"..., "-pi", "/nix/store/ybl1pacslmhci9zy5qv95"..., "-rc", "/nix/store/ybl1pacslmhci9zy5qv95"...], 0x8ba010 /* 132 vars */ <unfinished ...>
</pre>


With this new package you can [[Pull request|create a pull request for nixpkgs]]. Be aware that binary distributions are frowned upon if the source is available.
== See also ==


[[Category:Tutorial]]
* [[Steam#steam-run|steam-run]] as a quick way to run binaries
[[Category:nixpkgs]]
Retrieved from "https://wiki.nixos.org/wiki/Packaging/Binaries"