Packaging/Python: Difference between revisions
imported>Milahu fix: python -m unittest |
imported>Milahu setup.py: add `format = "pyproject";`, wording |
||
Line 163: | Line 163: | ||
The <code>setup.py</code> file is required by <code>buildPythonPackage</code>, | The <code>setup.py</code> file is required by <code>buildPythonPackage</code>, | ||
but it's missing in some packages | but it's missing in some packages | ||
<pre> | |||
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py' | |||
</pre> | |||
If the package has a <code>pyproject.toml</code> file, set | |||
<syntaxHighlight lang=nix> | |||
buildPythonPackage { | |||
format = "pyproject"; | |||
} | |||
</syntaxHighlight> | |||
If both <code>setup.py</code> and <code>pyproject.toml</code> are missing, | |||
you have to add one of these files, for example: | |||
<syntaxHighlight lang=nix> | |||
buildPythonPackage { | |||
preBuild = '' | |||
cat >setup.py <<'EOF' | |||
from setuptools import setup | |||
setup( | |||
name='someprogram', | |||
# ... | |||
) | |||
EOF | |||
''; | |||
} | |||
</syntaxHighlight> | |||
setup.py example: | setup.py example: | ||
Line 183: | Line 212: | ||
], | ], | ||
entry_points={ | entry_points={ | ||
# example: file some_module.py -> function main | |||
#'console_scripts': ['someprogram=some_module:main'] | #'console_scripts': ['someprogram=some_module:main'] | ||
}, | }, | ||
Line 188: | Line 218: | ||
</syntaxHighlight> | </syntaxHighlight> | ||
<code>scripts</code> is useful for self-contained python scripts with no local imports. | |||
If a python script has local imports, | |||
for example <code>from .some_module import some_function</code>, | |||
either include all files in the <code>scripts</code> array, | |||
or add only the entry function to <code>entry_points</code>. | |||
In this example, <code>someprogram.py</code> would be installed as <code>$out/bin/someprogram.py</code>.<br> | |||
To rename the binary, for example to remove the <code>.py</code> file extension, you can use <code>postInstall</code> | To rename the binary, for example to remove the <code>.py</code> file extension, you can use <code>postInstall</code> | ||
Revision as of 13:41, 4 April 2022
See Language-specific package helpers for a list of tools to package python packages.
Prepare Packaging
When you want to package a new software from a local checkout with the inputs coming from nixpkgs (and not virtualenv+pip) you can use the following shell.nix
[1]:
with import <nixpkgs> {};
with pkgs.python3Packages;
buildPythonPackage rec {
name = "mypackage";
src = ./path/to/source;
propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
}
You can now run nix-shell
and it will drop you in a shell similar to the python setup.py develop
mode which uses the local code in ./path/to/source as input. propagatedBuildInputs
will contain the packages you need in your project.
After you've finished developing you can replace the relative path with fetchFromGitHub { ... }
or fetchPypi { ... }
.
Pip and Virtualenv enabled nix-shell
It might be the case that you simply need to prototype fast small projects with pip
and virtualenv
without the need for relying on the dependencies being already packaged in nixpkgs.
{{Notice|Keep in mind that the virtualenv symlinks will be invalidated if you update your system!||
For a local working python environment you can use the following shell.nix
[2].
with import <nixpkgs> {};
stdenv.mkDerivation {
name = "pip-env";
buildInputs = [
# System requirements.
readline
# Python requirements (enough to get a virtualenv going).
python27Full
python27Packages.virtualenv
python27Packages.pip
];
src = null;
shellHook = ''
# Allow the use of wheels.
SOURCE_DATE_EPOCH=$(date +%s)
# Augment the dynamic linker path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
'';
}
When invoked with nix-shell, this environment gives you a readline-enabled Python, plus virtualenv and pip, from which you can create a virtual environment and then proceed to fill it with pip-installed packages from requirements.txt or any other source of packages.
And the only other thing you need to do is figure out which non-Python packages your pip-installable packages will need, and include them in buildInputs.
Caveats
ModuleNotFoundError: No module named 'pkg_resources'
If you see this runtime error
ModuleNotFoundError: No module named 'pkg_resources'
add setuptools
to your derivation
buildPythonPackage {
# ...
propagatedBuildInputs = [
# ...
setuptools
];
}
Please report such issues at https://github.com/NixOS/nixpkgs/issues
HTTP 404 with fetchPypi
example error:
curl: (22) The requested URL returned error: 404
error: cannot download stt-1.2.0.tar.gz from any mirror
when we look at https://pypi.org/project/stt/#files we see only *.whl
files:
stt-1.2.0-cp310-cp310-win_amd64.whl
stt-1.2.0-cp310-cp310-manylinux_2_24_x86_64.whl
stt-1.2.0-cp310-cp310-macosx_10_10_universal2.whl
- ...
this means, this is a binary release, so we have two options:
- build from source
- install binary release
build from source
replace this
buildPythonPackage {
pname = "TODO";
version = "TODO";
src = fetchPypi {
inherit pname version;
sha256 = ""; # TODO
};
with this
buildPythonPackage {
pname = "TODO";
version = "TODO";
src = fetchFromGitHub {
owner = "TODO";
repo = "TODO";
rev = "v${version}";
sha256 = ""; # TODO
};
install binary release
replace this
buildPythonPackage {
pname = "TODO";
version = "TODO";
src = fetchPypi {
inherit pname version;
sha256 = ""; # TODO
};
with this
buildPythonPackage {
pname = "TODO";
version = "TODO";
format = "wheel";
src = fetchPypi rec {
inherit pname version format;
sha256 = ""; # TODO
dist = python;
python = "py3";
#abi = "none";
#platform = "any";
};
... or use fetchurl
to download the *.whl
file directly.
examples: grep for ".whl" in nixpkgs/pkgs/development/python-modules/
reference: fetchPypi implementation
setup.py
The setup.py
file is required by buildPythonPackage
,
but it's missing in some packages
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py'
If the package has a pyproject.toml
file, set
buildPythonPackage {
format = "pyproject";
}
If both setup.py
and pyproject.toml
are missing,
you have to add one of these files, for example:
buildPythonPackage {
preBuild = ''
cat >setup.py <<'EOF'
from setuptools import setup
setup(
name='someprogram',
# ...
)
EOF
'';
}
setup.py example:
from setuptools import setup
with open('requirements.txt') as f:
install_requires = f.read().splitlines()
setup(
name='someprogram',
#packages=['someprogram'],
version='0.1.0',
#author='...',
#description='...',
install_requires=install_requires,
scripts=[
'someprogram.py',
],
entry_points={
# example: file some_module.py -> function main
#'console_scripts': ['someprogram=some_module:main']
},
)
scripts
is useful for self-contained python scripts with no local imports.
If a python script has local imports,
for example from .some_module import some_function
,
either include all files in the scripts
array,
or add only the entry function to entry_points
.
In this example, someprogram.py
would be installed as $out/bin/someprogram.py
.
To rename the binary, for example to remove the .py
file extension, you can use postInstall
buildPythonPackage {
# ...
postInstall = ''
mv -v $out/bin/someprogram.py $out/bin/someprogram
'';
}
requirements.txt
requirements.txt
in it's simplest form is a list of python packages
numpy Requests Pillow
buildPythonPackage
will check these dependencies, but you still must declare the nix dependencies in buildInputs
, propagatedBuildInputs
, checkInputs
, ...
Automatic packaging
TODO https://github.com/nix-community/pip2nix
TODO https://github.com/nix-community/poetry2nix
TODO https://pypi.org/project/pypi2nix/
Testing via this command is deprecated
In most cases, tests will pass anyway and you can ignore the warning.
In some cases, tests will fail, for example:
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
[ ... ]
TypeError: some_function() missing 1 required positional argument: 'some_argument'
quick fix: run tests with python's unittest module
checkPhase = ''
runHook preCheck
${python.interpreter} -m unittest
runHook postCheck
'';
See also
- buildPythonPackage implementation
- Python in the nixpkgs manual
- Python on Nix is an "Extensive collection of Python projects from PyPI"
- Rust section of Nixpkgs manual - build Rust code in Python projects