Packaging/Quirks and Caveats: Difference between revisions

From NixOS Wiki
imported>Makefu
No edit summary
m update url to wayback machine url
 
(15 intermediate revisions by 6 users not shown)
Line 8: Line 8:
== Build software with Autotools ==
== Build software with Autotools ==


Add <code>autoreconfHook</code> to <code>buildInputs</code> to automatically build software which uses <code>automake</code> and <code>autoconf</code>:
Add <code>autoreconfHook</code> to <code>nativeBuildInputs</code> to automatically build software which uses <code>automake</code> and <code>autoconf</code>:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
buildInputs = [ ...  autoreconfHook ];
nativeBuildInputs = [ ...  autoreconfHook ];
</syntaxhighlight>
</syntaxhighlight>
Examples in nixpkgs: * [https://github.com/NixOS/nixpkgs/blob/f4c253ff2f68fbe3e302f944e8347233d9dc8c9d/pkgs/tools/networking/samplicator/default.nix samplicator]
Examples in nixpkgs:  
* [https://github.com/NixOS/nixpkgs/blob/f4c253ff2f68fbe3e302f944e8347233d9dc8c9d/pkgs/tools/networking/samplicator/default.nix samplicator]


=== Package simple python scripts ===
== Configure Scripts that are using pkg-config ==


For scripts like a single Python file, it is not necessary to specify <code>src</code> in <code>mkDerivation</code>. When you want to use <code>buildPythonPackage</code> the sources need to provide a <code>setup.py</code> file which also is overkill for a lot of projects. The default <code>mkDerivation</code> will attempt to unpack your source code. This can be prevented that by applying <code>unpackPhase = ":";</code> (<code>:</code> is a no-op in shell scripts).
Some configure scripts are using <code>pkg-config</code> to determine the location of libraries and headers. Nixpkgs supports this by adding <code>pkg-config</code> to <code>nativeBuildInputs</code>
<syntaxhighlight lang="nix">
nativeBuildInputs = [ ...  pkg-config ];
</syntaxhighlight>
Examples in nixpkgs:
* [https://github.com/NixOS/nixpkgs/blob/96d41e393da3ca27fbcc7c82b7221a5c923460c0/pkgs/development/libraries/libmms/default.nix#L13 libmms]
 
One typical error when <code>pkg-config</code> is required but not in the <code>nativeBuildInputs</code> is the following during the configure phase:
<syntaxhighlight lang="shell">
./configure: line 20832: syntax error near unexpected token `nss,'
./configure: line 20832: `                      PKG_CHECK_MODULES(nss, nss)'
</syntaxhighlight>
== Package simple python scripts ==
 
For scripts like a single Python file, it is not necessary to specify <code>src</code> in <code>mkDerivation</code>. When you want to use <code>buildPythonPackage</code> the sources need to provide a <code>setup.py</code> file which also is overkill for a lot of projects. The default <code>mkDerivation</code> will attempt to unpack your source code. This can be prevented by applying <code>unpackPhase = ":";</code> (<code>:</code> is a no-op in shell scripts).


<syntaxhighlight lang="nix">myscript-package = pkgs.stdenv.mkDerivation {
<syntaxhighlight lang="nix">myscript-package = pkgs.stdenv.mkDerivation {
Line 37: Line 52:
Source: [http://stackoverflow.com/questions/43837691/how-to-package-a-single-python-script-with-nix/43837692#43837692 nh2 @ StackOverflow]
Source: [http://stackoverflow.com/questions/43837691/how-to-package-a-single-python-script-with-nix/43837692#43837692 nh2 @ StackOverflow]


A more lightweight alternative is to use <code>nix-shell</code> in the shebang line as described in this [http://iam.travishartwell.net/2015/06/17/nix-shell-shebang/ blog post]. This causes the expression to be evaluated and built every time the script is run; this means that the dependencies will always be kept up to date, but since nix-shell only creates a temporary GC root the dependencies may be removed by a garbage collection, so this approach is not advisable for users who don't have an internet connection available all the time.
A more lightweight alternative is to use <code>nix-shell</code> in the shebang line as described in this [https://web.archive.org/web/20230330010914/http://iam.travishartwell.net/2015/06/17/nix-shell-shebang/ blog post]. This causes the expression to be evaluated and built every time the script is run; this means that the dependencies will always be kept up to date, but since nix-shell only creates a temporary GC root the dependencies may be removed by a garbage collection, so this approach is not advisable for users who don't have an internet connection available all the time.


== Caveats ==
== Caveats ==
Line 43: Line 58:
After packaging software and successfully generating an executable some functions of the package might still not work. This is a collection of error and how to fix them:
After packaging software and successfully generating an executable some functions of the package might still not work. This is a collection of error and how to fix them:


=== GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with ===


=== GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with ===
Sometime the error mesage might be also:
 
<syntaxHighlight lang="nix">
GLib-GIO-ERROR **: No GSettings schemas are installed on the system
</syntaxHighlight>


Fixed by adding <code>wrapGAppsHook</code> to buildInputs:
Fixed by adding <code>wrapGAppsHook</code> to buildInputs:


<syntaxHighlight lang="nix">
<syntaxHighlight lang="nix">
buildInputs = [ ...  wrapGAppsHook ];
nativeBuildInputs = [ ...  wrapGAppsHook ];
</syntaxHighlight>
</syntaxHighlight>
Sample PR in nixpkgs:  
Sample PR in nixpkgs:  
* [https://github.com/NixOS/nixpkgs/pull/24617/files networkmanagerapplet]
* [https://github.com/NixOS/nixpkgs/pull/24617/files networkmanagerapplet]
=== Namespace Gdk not available ===
You will need <code>/nix/store/*-gtk+3-*/lib/girepository-1.0</code> in <code>GI_TYPELIB_PATH</code>.
Similar solution as above, solved by:
<syntaxHighlight lang=nix>
  nativeBuildInputs = [ ...  wrapGAppsHook ];
  buildInputs = [ gtk3 ];
</syntaxHighlight>


=== ImportError: libstdc++.so.6: cannot open shared object file: No such file ===
=== ImportError: libstdc++.so.6: cannot open shared object file: No such file ===
Line 59: Line 88:
Solution: add <code>${stdenv.cc.cc.lib}/lib/libstdc++.so.6</code> to the <code>LD_LIBRARY_PATH</code>.
Solution: add <code>${stdenv.cc.cc.lib}/lib/libstdc++.so.6</code> to the <code>LD_LIBRARY_PATH</code>.


[[Category:Guide]]
A sample <code>shell.nix</code>:
<syntaxHighlight lang="nix">
{ pkgs ? (import <nixpkgs> {}).pkgs }:
with pkgs;
mkShell {
  buildInputs = [
    python3Packages.virtualenv # run virtualenv .
    python3Packages.pyqt5 # avoid installing via pip
    python3Packages.pyusb # fixes the pyusb 'No backend available' when installed directly via pip
  ];
  shellHook = ''
    # fixes libstdc++ issues and libgl.so issues
    LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/:/run/opengl-driver/lib/
    # fixes xcb issues :
    QT_PLUGIN_PATH=${qt5.qtbase}/${qt5.qtbase.qtPluginPrefix}
  '';
}
</syntaxHighlight>
 
=== Test cannot access <code>/etc/protocols</code>, <code>/etc/services</code> or expects a special <code>/etc/passwd</code> when building in sandbox ===
 
Sometimes libraries try to fetch protocol specs via <code>socket.getprotobyname('tcp')</code> which fails in sandboxes because /etc/protocols is unaccessible. Override pre- and postCheck phases with this:
<syntaxHighlight lang="nix">
    preCheck = ''
      export NIX_REDIRECTS=/etc/protocols=${pkgs.iana-etc}/etc/protocols \
        LD_PRELOAD=${pkgs.libredirect}/lib/libredirect.so
    '';
    postCheck = ''
      unset NIX_REDIRECTS LD_PRELOAD
    '';
</syntaxHighlight>
 
[[Category:Cookbook]]

Latest revision as of 06:19, 14 September 2024

This article is about the quirks on how to package software where the source code is available.

A good start for packaging your first piece if software is the Quickstart Chapter in the nixpkgs manual Also see the Generic Algorithm on doing Packaging

For packaging executable without building them from source check out the article Packaging Binaries.

Build software with Autotools

Add autoreconfHook to nativeBuildInputs to automatically build software which uses automake and autoconf:

nativeBuildInputs = [ ...  autoreconfHook ];

Examples in nixpkgs:

Configure Scripts that are using pkg-config

Some configure scripts are using pkg-config to determine the location of libraries and headers. Nixpkgs supports this by adding pkg-config to nativeBuildInputs

nativeBuildInputs = [ ...  pkg-config ];

Examples in nixpkgs:

One typical error when pkg-config is required but not in the nativeBuildInputs is the following during the configure phase:

./configure: line 20832: syntax error near unexpected token `nss,'
./configure: line 20832: `                      PKG_CHECK_MODULES(nss, nss)'

Package simple python scripts

For scripts like a single Python file, it is not necessary to specify src in mkDerivation. When you want to use buildPythonPackage the sources need to provide a setup.py file which also is overkill for a lot of projects. The default mkDerivation will attempt to unpack your source code. This can be prevented by applying unpackPhase = ":"; (: is a no-op in shell scripts).

myscript-package = pkgs.stdenv.mkDerivation {
  name = "myscript";
  buildInputs = [
    (pkgs.python36.withPackages (pythonPackages: with pythonPackages; [
      consul
      six
      requests
    ]))
  ];
  unpackPhase = ":";
  installPhase = "install -m755 -D ${./myscript.py} $out/bin/myscript";
};

stdenv's patchShebangs will automatically replace shebangs in the fixup phase, for ex. #!/usr/bin/env python3 with dependencies given in buildInputs. As the derivation got pkgs.python36.withPackages (...) in buildInputs, it will create a virtualenv-like python wrapper. The python wrapper will have all specified dependencies and will be used to call the script.

In NixOS, the package can be put into environment.systemPackages, and myscript will be available as a global command.

Source: nh2 @ StackOverflow

A more lightweight alternative is to use nix-shell in the shebang line as described in this blog post. This causes the expression to be evaluated and built every time the script is run; this means that the dependencies will always be kept up to date, but since nix-shell only creates a temporary GC root the dependencies may be removed by a garbage collection, so this approach is not advisable for users who don't have an internet connection available all the time.

Caveats

After packaging software and successfully generating an executable some functions of the package might still not work. This is a collection of error and how to fix them:

GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with

Sometime the error mesage might be also:

GLib-GIO-ERROR **: No GSettings schemas are installed on the system

Fixed by adding wrapGAppsHook to buildInputs:

nativeBuildInputs = [ ...  wrapGAppsHook ];

Sample PR in nixpkgs:

Namespace Gdk not available

You will need /nix/store/*-gtk+3-*/lib/girepository-1.0 in GI_TYPELIB_PATH.

Similar solution as above, solved by:

  nativeBuildInputs = [ ...  wrapGAppsHook ];
  buildInputs = [ gtk3 ];

ImportError: libstdc++.so.6: cannot open shared object file: No such file

This can happen when importing python libraries: Solution: add ${stdenv.cc.cc.lib}/lib/libstdc++.so.6 to the LD_LIBRARY_PATH.

A sample shell.nix:

{ pkgs ? (import <nixpkgs> {}).pkgs }:
with pkgs;
mkShell {
  buildInputs = [
    python3Packages.virtualenv # run virtualenv .
    python3Packages.pyqt5 # avoid installing via pip
    python3Packages.pyusb # fixes the pyusb 'No backend available' when installed directly via pip
  ];
  shellHook = ''
    # fixes libstdc++ issues and libgl.so issues
    LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/:/run/opengl-driver/lib/
    # fixes xcb issues :
    QT_PLUGIN_PATH=${qt5.qtbase}/${qt5.qtbase.qtPluginPrefix}
  '';
}

Test cannot access /etc/protocols, /etc/services or expects a special /etc/passwd when building in sandbox

Sometimes libraries try to fetch protocol specs via socket.getprotobyname('tcp') which fails in sandboxes because /etc/protocols is unaccessible. Override pre- and postCheck phases with this:

     preCheck = ''
       export NIX_REDIRECTS=/etc/protocols=${pkgs.iana-etc}/etc/protocols \
         LD_PRELOAD=${pkgs.libredirect}/lib/libredirect.so
     '';
     postCheck = ''
       unset NIX_REDIRECTS LD_PRELOAD
     '';