NixOS Wiki

Packaging/Python: Difference between revisions

VisualWikitext
imported>Makefu
init with pip snippet and dev mode
 
Abus (talk | contribs)
1 edit
m Fixed formatting error with note
 
(50 intermediate revisions by 15 users not shown)
Line 1: Line 1:
Packaging of python packages is generally done via pypi2nix
See [[Language-specific package helpers]] for a list of tools to package python packages.
 
== Prepare Packaging ==
== Prepare Packaging ==
When you want to package a new software from a local checkout with the inputs coming from nixpkgs (and not virtualenv+pip) you can use the following <code>shell.nix</code> <ref>[https://nixos.org/nixpkgs/manual/#develop-local-package nixpkgs manual]</ref>:
When you want to package a new software from a local checkout with the inputs coming from nixpkgs (and not virtualenv+pip) you can use the following <code>shell.nix</code> <ref>[https://nixos.org/nixpkgs/manual/#develop-local-package nixpkgs manual]</ref>:
Line 13: Line 14:
</syntaxHighlight>
</syntaxHighlight>
You can now run <code>nix-shell</code> and it will drop you in a shell similar to the <code>python setup.py develop</code> mode which uses the local code in <tt>./path/to/source</tt> as input. <code>propagatedBuildInputs</code> will contain the packages you  need in your project.
You can now run <code>nix-shell</code> and it will drop you in a shell similar to the <code>python setup.py develop</code> mode which uses the local code in <tt>./path/to/source</tt> as input. <code>propagatedBuildInputs</code> will contain the packages you  need in your project.
After you've finished developing you can replace the relative path with <code>fetchFromGitHub { ... }</code> or <code>fetchPypi { ... }</code>.
After you've finished developing you can replace the relative path with <code>fetchFromGitHub { ... }</code> or <code>[https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchpypi/default.nix fetchPypi] { ... }</code>.
 
== Testing out a module in a Python shell ==
Once you have your derivation written, you can create a [[Flakes|flake]] to give yourself a shell and easily test out the module you wrote the derivation for.
 
Create a <code>flake.nix</code> next to your derivation:<syntaxhighlight lang="nix">
{
  description = "Dev shell using external derivation";
 
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        # replace ./derivation.nix with the path (relative to this file) to your derivation
        mypackage = pkgs.python3Packages.callPackage ./derivation.nix {
          lib = pkgs.lib;
        };
      in {
        packages.default = pkgs.python3.withPackages(_: [ mypackage ]);
      });
}
</syntaxhighlight>Replace "mypackage" with your package name. Then, run the flake to get into a python REPL with your package, ready to be imported:<syntaxhighlight lang="shell-session">
$ nix run
Python 3.11.10 (main, Sep  7 2024, 01:03:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage
>>> # test things out...
</syntaxhighlight>


== Pip and Virtualenv enabled nix-shell ==
== Pip and Virtualenv enabled nix-shell ==
It might be the case that you simply need to prototype fast small projects with <code>pip</code> and <code>virtualenv</code> without the need for relying on the dependencies being already packaged in nixpkgs.  
It might be the case that you simply need to prototype fast small projects with <code>pip</code> and <code>virtualenv</code> without the need for relying on the dependencies being already packaged in nixpkgs.
{{Notice|Keep in mind that the virtualenv symlinks will be invalidated if you update your system!||
 
{{Note|Keep in mind that the virtualenv symlinks will be invalidated if you update your system!}}
   
   
For a local working python environment you can use the following <code>shell.nix</code><ref>https://groups.google.com/forum/#!topic/nix-devel/3qPfwCAV3GE</ref>.
For a local working python environment you can use the following <code>shell.nix</code><ref>https://groups.google.com/forum/#!topic/nix-devel/3qPfwCAV3GE</ref>.
<syntaxHighlight lang=nix>
<syntaxHighlight lang=nix>
with import <nixpkgs> {};
{ pkgs ? import <nixpkgs> { } }:
 
let
  pythonEnv = pkgs.python3.withPackages(ps: [ ]);
 
in
pkgs.mkShell {
  packages = [
    pythonEnv
  ];
}
</syntaxHighlight>
When invoked with nix-shell, this environment gives you a readline-enabled Python, plus virtualenv and pip, from which you can create a virtual environment and then proceed to fill it with pip-installed packages from requirements.txt or any other source of packages.
 
And the only other thing you need to do is figure out which non-Python packages your pip-installable packages will need, and include them in buildInputs.
 
== Caveats ==
 
=== ModuleNotFoundError: No module named 'pkg_resources' ===
 
If you see this runtime error
 
<pre>
ModuleNotFoundError: No module named 'pkg_resources'
</pre>


stdenv.mkDerivation {
add <code>setuptools</code> to your derivation
  name = "pip-env";
  buildInputs = [
    # System requirements.
    readline


    # Python requirements (enough to get a virtualenv going).
<syntaxHighlight lang=nix>
    python27Full
buildPythonPackage {
     python27Packages.virtualenv
  # ...
     python27Packages.pip
  propagatedBuildInputs = [
     # ...
     setuptools
   ];
   ];
   src = null;
}
   shellHook = ''
</syntaxHighlight>
     # Allow the use of wheels.
 
     SOURCE_DATE_EPOCH=$(date +%s)
Please report such issues at https://github.com/NixOS/nixpkgs/issues
 
=== HTTP 404 with fetchPypi ===
 
example error:
 
<blockquote>
curl: (22) The requested URL returned error: 404<br>
error: cannot download stt-1.2.0.tar.gz from any mirror
</blockquote>
 
when we look at https://pypi.org/project/stt/#files we see only <code>*.whl</code> files:
 
* <code>stt-1.2.0-cp310-cp310-win_amd64.whl</code>
* <code>stt-1.2.0-cp310-cp310-manylinux_2_24_x86_64.whl</code>
* <code>stt-1.2.0-cp310-cp310-macosx_10_10_universal2.whl</code>
* ...
 
this means, this is a binary release, so we have two options:
 
# build from source
# install binary release
 
==== build from source ====
 
replace this
 
<syntaxHighlight lang=nix>
    buildPythonPackage {
      pname = "TODO";
      version = "TODO";
      src = fetchPypi {
        inherit pname version;
        sha256 = ""; # TODO
      };
</syntaxHighlight>
 
with this
 
<syntaxHighlight lang=nix>
    buildPythonPackage {
      pname = "TODO";
      version = "TODO";
      src = fetchFromGitHub {
        owner = "TODO";
        repo = "TODO";
        rev = "v${version}";
        sha256 = ""; # TODO
      };
</syntaxHighlight>
 
 
==== install binary release ====
 
replace this
 
<syntaxHighlight lang=nix>
    buildPythonPackage {
      pname = "TODO";
      version = "TODO";
      src = fetchPypi {
        inherit pname version;
        sha256 = ""; # TODO
      };
</syntaxHighlight>
 
with this
 
<syntaxHighlight lang=nix>
    buildPythonPackage {
      pname = "TODO";
      version = "TODO";
      format = "wheel";
      src = fetchPypi rec {
        inherit pname version format;
        sha256 = ""; # TODO
        dist = python;
        python = "py3";
        #abi = "none";
        #platform = "any";
      };
</syntaxHighlight>
 
... or use <code>fetchurl</code> to download the <code>*.whl</code> file directly.<br>
 
reference: [https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchpypi/default.nix fetchPypi implementation]
 
== Fix Missing <code>setup.py</code> ==
 
The <code>setup.py</code> file is required for <code>buildPythonPackage</code>,
but it's missing in some packages.
If you get the following error, you need to one of the workarounds below.
 
<pre>
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py'
</pre>
 
If the package has a <code>pyproject.toml</code> file, set
 
<syntaxHighlight lang=nix>
buildPythonPackage {
   format = "pyproject";
}
</syntaxHighlight>
 
If both <code>setup.py</code> and <code>pyproject.toml</code> are missing,
you have to add one of these files.
 
'''Note:''' sometimes you will be able to find <code>pyproject.toml</code> in the source for a package despite it not being present in a <code>.whl</code> file. You can inspect the contents of a <code>.whl</code> file by downloading it from PyPi and then extracting it with <code>nix-shell -p python311Packages.wheel --command wheel unpack path/to/package.whl</code>.
 
For example, you can create the <code>setup.py</code> in the <code>preBuild</code> phase.
 
<syntaxHighlight lang=nix>
buildPythonPackage {
   preBuild = ''
     cat > setup.py << EOF
from setuptools import setup
 
with open('requirements.txt') as f:
     install_requires = f.read().splitlines()


     # Augment the dynamic linker path
setup(
     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
  name='someprogram',
  #packages=['someprogram'],
  version='0.1.0',
  #author='...',
  #description='...',
  install_requires=install_requires,
  scripts=[
    'someprogram.py',
  ],
  entry_points={
     # example: file some_module.py -> function main
     #'console_scripts': ['someprogram=some_module:main']
  },
)
    EOF
   '';
   '';
}
}
</syntaxHighlight>
</syntaxHighlight>
When invoked with nix-shell, this environment gives you a readline-enabled Python, plus virtualenv and pip, from which you can create a virtual environment and then proceed to fill it with pip-installed packages from requirements.txt or any other source of packages.


And the only other thing you need to do is figure out which non-Python packages your pip-installable packages will need, and include them in buildInputs.
More info about the <code>setup.py</code> can be found [https://docs.python.org/3.11/distutils/setupscript.html here]. (<b>note:</b> from python 3.12 onwards, distutils is deprecated see https://docs.python.org/3.11/distutils/index.html)
 
<code>scripts</code> is useful for self-contained python scripts with no local imports.
 
If a python script has local imports,
for example <code>from .some_module import some_function</code>,
either include all files in the <code>scripts</code> array,
or add only the entry function to <code>entry_points</code>.
 
In this example, <code>someprogram.py</code> would be installed as <code>$out/bin/someprogram.py</code>.<br>
To rename the binary, for example to remove the <code>.py</code> file extension, you can use <code>postInstall</code>
 
<syntaxHighlight lang=nix>
buildPythonPackage {
  # ...
  postInstall = ''
    mv -v $out/bin/someprogram.py $out/bin/someprogram
  '';
}
</syntaxHighlight>
 
=== requirements.txt ===
 
<code>requirements.txt</code> in it's simplest form is a list of python packages
 
<pre>
numpy
Requests
Pillow
</pre>
 
<code>buildPythonPackage</code> will check these dependencies, but you still must declare the nix dependencies in <code>buildInputs</code>, <code>propagatedBuildInputs</code>, <code>checkInputs</code>, ...
 
== Automatic packaging ==
{| class="wikitable"
|+
!
!Project
!URL
!Stars
!Status
|-
|TODO
|poetry2nix
|https://github.com/nix-community/poetry2nix
|884+
|unmaintained
|-
|TODO
|pip2nix
|https://github.com/nix-community/pip2nix
|175+
|
|-
|TODO
|<s>pypi2nix</s>
|https://github.com/nix-community/pypi2nix
|194
|archived
|}
 
== Testing via this command is deprecated ==
 
In most cases, tests will pass anyway and you can ignore the warning.
 
In some cases, tests will fail, for example:
 
<blockquote>
running test<br>
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.<br>
[ ... ]<br>
TypeError: some_function() missing 1 required positional argument: 'some_argument'
</blockquote>
 
quick fix: run tests with python's [https://docs.python.org/3/library/unittest.html unittest] module
 
<syntaxHighlight lang=nix>
  checkPhase = ''
    runHook preCheck
    ${python3.interpreter} -m unittest
    runHook postCheck
  '';
</syntaxHighlight>
 
== See also ==
 
* [https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/mk-python-derivation.nix buildPythonPackage implementation]
* [https://nixos.org/manual/nixpkgs/#python Python] in the nixpkgs manual
* [https://github.com/on-nix/python Python on Nix] is an "Extensive collection of Python projects from PyPI"
* [https://nixos.org/manual/nixpkgs/stable/#examples Rust section of Nixpkgs manual] - build Rust code in Python projects
 
== References ==
 
[[Category:Python]]
Retrieved from "https://wiki.nixos.org/wiki/Packaging/Python"