NixOS Wiki

Packaging/Python: Difference between revisions

← Older edit
VisualWikitext
imported>Milahu
update HTTP 404 with fetchPypi
Abus (talk | contribs)
1 edit
m Fixed formatting error with note
 
(29 intermediate revisions by 12 users not shown)
Line 14: Line 14:
</syntaxHighlight>
</syntaxHighlight>
You can now run <code>nix-shell</code> and it will drop you in a shell similar to the <code>python setup.py develop</code> mode which uses the local code in <tt>./path/to/source</tt> as input. <code>propagatedBuildInputs</code> will contain the packages you  need in your project.
You can now run <code>nix-shell</code> and it will drop you in a shell similar to the <code>python setup.py develop</code> mode which uses the local code in <tt>./path/to/source</tt> as input. <code>propagatedBuildInputs</code> will contain the packages you  need in your project.
After you've finished developing you can replace the relative path with <code>fetchFromGitHub { ... }</code> or <code>[https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/fetchpypi.nix fetchPypi] { ... }</code>.
After you've finished developing you can replace the relative path with <code>fetchFromGitHub { ... }</code> or <code>[https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchpypi/default.nix fetchPypi] { ... }</code>.
 
== Testing out a module in a Python shell ==
Once you have your derivation written, you can create a [[Flakes|flake]] to give yourself a shell and easily test out the module you wrote the derivation for.
 
Create a <code>flake.nix</code> next to your derivation:<syntaxhighlight lang="nix">
{
  description = "Dev shell using external derivation";
 
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        # replace ./derivation.nix with the path (relative to this file) to your derivation
        mypackage = pkgs.python3Packages.callPackage ./derivation.nix {
          lib = pkgs.lib;
        };
      in {
        packages.default = pkgs.python3.withPackages(_: [ mypackage ]);
      });
}
</syntaxhighlight>Replace "mypackage" with your package name. Then, run the flake to get into a python REPL with your package, ready to be imported:<syntaxhighlight lang="shell-session">
$ nix run
Python 3.11.10 (main, Sep  7 2024, 01:03:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage
>>> # test things out...
</syntaxhighlight>


== Pip and Virtualenv enabled nix-shell ==
== Pip and Virtualenv enabled nix-shell ==
It might be the case that you simply need to prototype fast small projects with <code>pip</code> and <code>virtualenv</code> without the need for relying on the dependencies being already packaged in nixpkgs.  
It might be the case that you simply need to prototype fast small projects with <code>pip</code> and <code>virtualenv</code> without the need for relying on the dependencies being already packaged in nixpkgs.
{{Notice|Keep in mind that the virtualenv symlinks will be invalidated if you update your system!||
 
{{Note|Keep in mind that the virtualenv symlinks will be invalidated if you update your system!}}
   
   
For a local working python environment you can use the following <code>shell.nix</code><ref>https://groups.google.com/forum/#!topic/nix-devel/3qPfwCAV3GE</ref>.
For a local working python environment you can use the following <code>shell.nix</code><ref>https://groups.google.com/forum/#!topic/nix-devel/3qPfwCAV3GE</ref>.
<syntaxHighlight lang=nix>
<syntaxHighlight lang=nix>
with import <nixpkgs> {};
{ pkgs ? import <nixpkgs> { } }:


stdenv.mkDerivation {
let
   name = "pip-env";
   pythonEnv = pkgs.python3.withPackages(ps: [ ]);
  buildInputs = [
    # System requirements.
    readline


    # Python requirements (enough to get a virtualenv going).
in
    python27Full
pkgs.mkShell {
     python27Packages.virtualenv
  packages = [
    python27Packages.pip
     pythonEnv
   ];
   ];
  src = null;
  shellHook = ''
    # Allow the use of wheels.
    SOURCE_DATE_EPOCH=$(date +%s)
    # Augment the dynamic linker path
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
  '';
}
}
</syntaxHighlight>
</syntaxHighlight>
Line 144: Line 166:
       version = "TODO";
       version = "TODO";
       format = "wheel";
       format = "wheel";
       src = fetchPypi {
       src = fetchPypi rec {
         inherit pname version format;
         inherit pname version format;
         sha256 = ""; # TODO
         sha256 = ""; # TODO
         dist = "py3";
         dist = python;
         python = "py3";
         python = "py3";
         #abi = "none";
         #abi = "none";
Line 155: Line 177:


... or use <code>fetchurl</code> to download the <code>*.whl</code> file directly.<br>
... or use <code>fetchurl</code> to download the <code>*.whl</code> file directly.<br>
examples: [https://grep.app/search?filter%5Brepo%5D%5B0%5D=NixOS/nixpkgs&filter%5Blang%5D%5B0%5D=Nix&filter%5Bpath%5D%5B0%5D=pkgs/development/python-modules/&q=.whl grep for ".whl" in nixpkgs/pkgs/development/python-modules/]


reference: [https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/fetchpypi.nix fetchPypi implementation]
reference: [https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchpypi/default.nix fetchPypi implementation]


== setup.py ==
== Fix Missing <code>setup.py</code> ==


The <code>setup.py</code> file is required by <code>buildPythonPackage</code>,
The <code>setup.py</code> file is required for <code>buildPythonPackage</code>,
but it's missing in some packages
but it's missing in some packages.
If you get the following error, you need to one of the workarounds below.


setup.py example:
<pre>
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py'
</pre>


<syntaxHighlight lang=python>
If the package has a <code>pyproject.toml</code> file, set
 
<syntaxHighlight lang=nix>
buildPythonPackage {
  format = "pyproject";
}
</syntaxHighlight>
 
If both <code>setup.py</code> and <code>pyproject.toml</code> are missing,
you have to add one of these files.
 
'''Note:''' sometimes you will be able to find <code>pyproject.toml</code> in the source for a package despite it not being present in a <code>.whl</code> file. You can inspect the contents of a <code>.whl</code> file by downloading it from PyPi and then extracting it with <code>nix-shell -p python311Packages.wheel --command wheel unpack path/to/package.whl</code>.
 
For example, you can create the <code>setup.py</code> in the <code>preBuild</code> phase.
 
<syntaxHighlight lang=nix>
buildPythonPackage {
  preBuild = ''
    cat > setup.py << EOF
from setuptools import setup
from setuptools import setup


Line 183: Line 225:
   ],
   ],
   entry_points={
   entry_points={
    # example: file some_module.py -> function main
     #'console_scripts': ['someprogram=some_module:main']
     #'console_scripts': ['someprogram=some_module:main']
   },
   },
)
)
    EOF
  '';
}
</syntaxHighlight>
</syntaxHighlight>


In this example, <code>someprogram.py</code> would be installed as <code>$out/bin/someprogram.py</code>
More info about the <code>setup.py</code> can be found [https://docs.python.org/3.11/distutils/setupscript.html here]. (<b>note:</b> from python 3.12 onwards, distutils is deprecated see https://docs.python.org/3.11/distutils/index.html)
 
<code>scripts</code> is useful for self-contained python scripts with no local imports.
 
If a python script has local imports,
for example <code>from .some_module import some_function</code>,
either include all files in the <code>scripts</code> array,
or add only the entry function to <code>entry_points</code>.


In this example, <code>someprogram.py</code> would be installed as <code>$out/bin/someprogram.py</code>.<br>
To rename the binary, for example to remove the <code>.py</code> file extension, you can use <code>postInstall</code>
To rename the binary, for example to remove the <code>.py</code> file extension, you can use <code>postInstall</code>


Line 214: Line 268:


== Automatic packaging ==
== Automatic packaging ==
{| class="wikitable"
|+
!
!Project
!URL
!Stars
!Status
|-
|TODO
|poetry2nix
|https://github.com/nix-community/poetry2nix
|884+
|unmaintained
|-
|TODO
|pip2nix
|https://github.com/nix-community/pip2nix
|175+
|
|-
|TODO
|<s>pypi2nix</s>
|https://github.com/nix-community/pypi2nix
|194
|archived
|}


TODO https://github.com/nix-community/pip2nix
== Testing via this command is deprecated ==


TODO https://github.com/nix-community/poetry2nix
In most cases, tests will pass anyway and you can ignore the warning.


TODO https://pypi.org/project/pypi2nix/
In some cases, tests will fail, for example:
 
<blockquote>
running test<br>
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.<br>
[ ... ]<br>
TypeError: some_function() missing 1 required positional argument: 'some_argument'
</blockquote>
 
quick fix: run tests with python's [https://docs.python.org/3/library/unittest.html unittest] module
 
<syntaxHighlight lang=nix>
  checkPhase = ''
    runHook preCheck
    ${python3.interpreter} -m unittest
    runHook postCheck
  '';
</syntaxHighlight>


== See also ==
== See also ==


* [https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/mk-python-derivation.nix buildPythonPackage implementation]
* [https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/mk-python-derivation.nix buildPythonPackage implementation]
* [https://ryantm.github.io/nixpkgs/languages-frameworks/python/ Python] in the nixpkgs manual
* [https://nixos.org/manual/nixpkgs/#python Python] in the nixpkgs manual
* [https://github.com/on-nix/python Python on Nix] is an "Extensive collection of Python projects from PyPI"
* [https://github.com/on-nix/python Python on Nix] is an "Extensive collection of Python projects from PyPI"
* [https://nixos.org/manual/nixpkgs/stable/#examples Rust section of Nixpkgs manual] - build Rust code in Python projects


== References ==
== References ==
[[Category:Python]]
Retrieved from "https://wiki.nixos.org/wiki/Packaging/Python"