Python: Difference between revisions
imported>Turbotimon fix cuda version according to https://discourse.nixos.org/t/python-venv-help-for-official-wiki/36481 |
fixed broken link for pyproject use-cases |
||
(61 intermediate revisions by 35 users not shown) | |||
Line 1: | Line 1: | ||
== | == Python development environments with Nix == | ||
Nix supports a number of approaches to creating "development environments" for Python programming. These provide functionality analogous to [https://virtualenv.pypa.io/en/latest/ virtualenv] or [https://docs.conda.io/en/latest/ conda]: a shell environment with access to pinned versions of the <code>python</code> executable and Python packages. | |||
=== Using the Nixpkgs Python infrastructure via <code>shell.nix</code> (recommended) === | |||
Nixpkgs has the few last Python versions packaged, as well as a consequent set of Python packages packaged that you can use to quickly create a Python environment. | |||
< | Create a file <code>shell.nix</code> in the project directory, with the following template: | ||
</ | |||
<syntaxhighlight lang="nix" line="1"> | |||
# shell.nix | |||
< | |||
let | let | ||
# We pin to a specific nixpkgs commit for reproducibility. | |||
# Last updated: 2024-04-29. Check for new commits at https://status.nixos.org. | |||
pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/cf8cc1201be8bc71b7cbbbdaf349b22f4f99c7ae.tar.gz") {}; | |||
in pkgs.mkShell { | |||
packages = [ | |||
in | (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [ | ||
# select Python packages here | |||
pandas | |||
requests | |||
])) | |||
]; | ]; | ||
} | } | ||
</ | </syntaxhighlight> | ||
In this example, we create a Python environment with packages <code>pandas</code> and <code>requests</code>. | |||
You can find Python packages that are available in Nixpkgs using [https://search.nixos.org/packages search.nixos.org]. For instance, type a Python package name like <code>numpy</code> in the search bar and click on the search button on the right. You can narrow down results by clicking on eg. "python311Packages" in the "Package sets" section on the left. Note that in the snippet above, on lines 8 and 9, each package is listed in the form <code>python-pkgs.<name></code> where <code><name></code> corresponds to the one found in [https://search.nixos.org/packages search.nixos.org] . See [https://nix.dev/tutorials/nix-language.html Nix language basics] for more information on the <code>python-pkgs</code> attribute set. | |||
Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>) to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>. | |||
Note that with NixOS, this method can be used to install packages at the system level, e.g. <syntaxhighlight lang="nix"> | |||
environment.systemPackages = with pkgs; [ | |||
< | |||
# ... | # ... | ||
( | (python3.withPackages (python-pkgs: with python-pkgs; [ | ||
pandas | |||
requests | |||
])) | |||
]; | ]; | ||
</ | </syntaxhighlight> | ||
=== Using Nix shell (new command line) === | |||
nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])' | |||
If you don't use the channels any more, you can replace <code><nixpkgs></code> by an instance of the <code>NixOS/nixpkgs</code> repository using its absolute path. | |||
==== Using a Python package not in Nixpkgs ==== | |||
Python packages in Nixpkgs are created and updated by Nixpkgs maintainers. Although the community invests a great effort to keep a complete and up-to-date package set, some packages you want may be missing, out of date, or broken. To use your own packages in a Nix environment, you may package it yourself. | |||
The following is a high-level overview. For a complete explanation, see [https://nixos.org/manual/nixpkgs/unstable/#developing-with-python Developing with Python] in the Nixpkgs Manual. | |||
Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1"> | |||
# toolz.nix | |||
{ | |||
lib, | |||
buildPythonPackage, | |||
fetchPypi, | |||
setuptools, | |||
wheel, | |||
}: | |||
== | buildPythonPackage rec { | ||
pname = "toolz"; | |||
version = "0.10.0"; | |||
src = fetchPypi { | |||
inherit pname version; | |||
hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA="; | |||
}; | |||
# do not run tests | |||
doCheck = false; | |||
< | # specific to buildPythonPackage, see its reference | ||
pyproject = true; | |||
build-system = [ | |||
setuptools | |||
wheel | |||
]; | |||
} | |||
</syntaxhighlight>Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1"> | |||
# shell.nix | # shell.nix | ||
let | let | ||
pkgs = import <nixpkgs> {}; | |||
< | |||
python = pkgs.python3.override { | |||
self = python; | |||
packageOverrides = pyfinal: pyprev: { | |||
toolz = pyfinal.callPackage ./toolz.nix { }; | |||
}; | |||
}; | |||
in pkgs.mkShell { | |||
pkgs.mkShell { | |||
packages = [ | packages = [ | ||
# ... | (python.withPackages (python-pkgs: [ | ||
# select Python packages here | |||
python-pkgs.pandas | |||
python-pkgs.requests | |||
python-pkgs.toolz | |||
])) | |||
]; | ]; | ||
} | } | ||
</ | </syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written. | ||
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> === | |||
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session"> | |||
$ python -c 'import grpc' | |||
Traceback (most recent call last): | |||
File "<string>", line 1, in <module> | |||
File "/.../grpc/__init__.py", line 22, in <module> | |||
from grpc import _compression | |||
File "/.../grpc/_compression.py", line 20, in <module> | |||
from grpc._cython import cygrpc | |||
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory | |||
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve. | |||
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>. | |||
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers. | |||
==== Using nix overlay ==== | |||
<syntaxhighlight lang="nix"> | <syntaxhighlight lang="nix"> | ||
with | { | ||
nixpkgs.overlays = [( | |||
self: super: rec { | |||
# https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44 | |||
# > We currently take all libraries from systemd and nix as the default. | |||
] | pythonldlibpath = lib.makeLibraryPath (with super; [ | ||
zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd | |||
]); | |||
# here we are overriding python program to add LD_LIBRARY_PATH to it's env | |||
python = super.stdenv.mkDerivation { | |||
name = "python"; | |||
buildInputs = [ super.makeWrapper ]; | |||
src = super.python311; | |||
installPhase = '' | |||
mkdir -p $out/bin | |||
cp -r $src/* $out/ | |||
wrapProgram $out/bin/python3 --set LD_LIBRARY_PATH ${pythonldlibpath} | |||
wrapProgram $out/bin/python3.11 --set LD_LIBRARY_PATH ${pythonldlibpath} | |||
''; | |||
}; | |||
poetry = super.stdenv.mkDerivation { | |||
name = "poetry"; | |||
buildInputs = [ super.makeWrapper ]; | |||
src = super.poetry; | |||
installPhase = '' | |||
mkdir -p $out/bin | |||
cp -r $src/* $out/ | |||
wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${pythonldlibpath} | |||
''; | |||
}; | |||
} | |||
)]; | |||
environment.systemPackages = with pkgs; [ | |||
python # here python will be taken from the overlay up here | |||
poetry | |||
]; | |||
} | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== | ==== Using [https://github.com/Mic92/nix-ld nix-ld] ==== | ||
<syntaxhighlight lang="nix"> | |||
{ | |||
programs.nix-ld = { | |||
enable = true; | |||
# | libraries = with pkgs; [ | ||
zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd | |||
]; | |||
}; | |||
# https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld | |||
environment.systemPackages = [ | |||
(pkgs.writeShellScriptBin "python" '' | |||
export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH | |||
exec ${pkgs.python3}/bin/python "$@" | |||
'') | |||
]; | |||
# another (dangerous) solution | |||
# environment.systemPackages = with pkgs; [ python3 ]; | |||
# programs.bash = { | |||
# enable = true; | |||
# initExtra = '' | |||
# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH | |||
# ''; | |||
# }; | |||
} | |||
</syntaxhighlight> | |||
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ==== | |||
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}} | |||
Install with: | |||
<syntaxhighlight lang="shell"> | |||
nix profile install github:GuillaumeDesforges/fix-python | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Enter the venv and run: | |||
<syntaxhighlight lang="shell"> | |||
fix-python --venv .venv | |||
</syntaxhighlight> | </syntaxhighlight> | ||
==== Using <code>buildFHSEnv</code> (Recommended) ==== | |||
<syntaxhighlight lang="nix"> | <syntaxhighlight lang="nix"> | ||
{ | #!/usr/bin/env nix-shell | ||
with | { pkgs ? import <nixpkgs> { } }: | ||
( | |||
let base = pkgs.appimageTools.defaultFhsEnvArgs; in | |||
pkgs.buildFHSEnv (base // { | |||
name = "FHS"; | |||
targetPkgs = pkgs: (with pkgs; [ | |||
gcc glibc zlib | |||
]); | |||
runScript = "zsh"; | |||
extraOutputsToInstall = [ "dev" ]; | |||
}) | |||
).env | |||
</syntaxhighlight> | |||
==== Using a custom nix-shell ==== | |||
The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1"> | |||
let | |||
python = pkgs.python311; | |||
# We currently take all libraries from systemd and nix as the default | |||
# https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44 | |||
pythonldlibpath = lib.makeLibraryPath (with pkgs; [ | |||
zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd | |||
]); | |||
patchedpython = (python.overrideAttrs ( | |||
previousAttrs: { | |||
# Add the nix-ld libraries to the LD_LIBRARY_PATH. | |||
# creating a new library path from all desired libraries | |||
postInstall = previousAttrs.postInstall + '' | |||
mv "$out/bin/python3.11" "$out/bin/unpatched_python3.11" | |||
cat << EOF >> "$out/bin/python3.11" | |||
#!/run/current-system/sw/bin/bash | |||
export LD_LIBRARY_PATH="${pythonldlibpath}" | |||
exec "$out/bin/unpatched_python3.11" "\$@" | |||
EOF | |||
chmod +x "$out/bin/python3.11" | |||
''; | |||
} | |||
)); | |||
# if you want poetry | |||
patchedpoetry = ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs ( | |||
previousAttrs: { | |||
# same as above, but for poetry | |||
# not that if you dont keep the blank line bellow, it crashes :( | |||
postInstall = previousAttrs.postInstall + '' | |||
mv "$out/bin/poetry" "$out/bin/unpatched_poetry" | |||
cat << EOF >> "$out/bin/poetry" | |||
#!/run/current-system/sw/bin/bash | |||
export LD_LIBRARY_PATH="${pythonldlibpath}" | |||
exec "$out/bin/unpatched_poetry" "\$@" | |||
EOF | |||
chmod +x "$out/bin/poetry" | |||
''; | |||
} | |||
)); | |||
in | |||
{ | |||
environment.systemPackages = with pkgs; [ | |||
patchedpython | |||
# if you want poetry | |||
patchedpoetry | |||
]; | |||
} | } | ||
</syntaxhighlight> | </syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>. | ||
==== Prefix library paths using wrapProgram ==== | |||
<syntaxhighlight lang="nix" | wrapProgram is a part of the makeWrapper build input<ref>https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/make-wrapper.sh</ref>. By combining it with the symlinkJoin, we can create a wrapper around the Python executable that will always set the required library paths. It’s worth noting that, for this solution to be compatible with Darwin, we need to use a different wrap prefix, as shown in the example below.<syntaxhighlight lang="nixos" line="1"> | ||
let | |||
pkgs. | # We currently take all libraries from systemd and nix as the default | ||
# https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44 | |||
pythonldlibpath = lib.makeLibraryPath (with pkgs; [ | |||
zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd | |||
]); | |||
# Darwin requires a different library path prefix | |||
wrapPrefix = if (!pkgs.stdenv.isDarwin) then "LD_LIBRARY_PATH" else "DYLD_LIBRARY_PATH"; | |||
patchedpython = (pkgs.symlinkJoin { | |||
name = "python"; | |||
paths = [ pkgs.python312 ]; | |||
buildInputs = [ pkgs.makeWrapper ]; | |||
postBuild = '' | |||
wrapProgram "$out/bin/python3.12" --prefix ${wrapPrefix} : "${pythonldlibpath}" | |||
''; | |||
}); | |||
in | |||
{ | |||
environment.systemPackages = with pkgs; [ | |||
patchedpython | |||
]; | |||
} | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Using <code>venv</code> === | |||
< | |||
</ | |||
To create a Python virtual environment with <code>venv</code>:<syntaxhighlight lang="console"> | |||
$ nix-shell -p python3 --command "python -m venv .venv --copies" | |||
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar. | |||
>> | |||
</ | |||
=== | === Using uv === | ||
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote> | |||
uv is written in Rust and does ''not'' need Python as a prerequisite. Use the <code>uv</code> command to initialize Python projects, add Python packages, create or update virtual environments (in <code>.venv</code> folders), etc. as [https://docs.astral.sh/uv/concepts/projects/ described in the uv docs]. Use uv's [https://docs.astral.sh/uv/guides/tools/ tool interface] to install and run Python packages that provide a CLI. | |||
As a system package | |||
<syntaxhighlight lang="nix"> | |||
<syntaxhighlight lang= | environment.systemPackages = with pkgs; [ | ||
uv | |||
]; | |||
</syntaxhighlight> | </syntaxhighlight> | ||
or as a home-manager package | |||
<syntaxhighlight lang="nix"> | <syntaxhighlight lang="nix"> | ||
home.packages = with pkgs; [ | |||
uv | |||
]; | |||
</syntaxhighlight> | </syntaxhighlight> | ||
If you use uv it's recommended that you install the Python versions you need using the <code>uv python install</code> command, e.g. | |||
<syntaxhighlight lang=" | <syntaxhighlight lang="python"> | ||
uv python install 3.14 --preview --default | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== | You may want to set the <code>UV_PYTHON_DOWNLOADS=never</code> environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting <code>environment.localBinInPath = true;</code> is highly recommended, because uv will install binaries in <code>~/.local/bin</code>. | ||
=== | === Using poetry === | ||
<syntaxhighlight lang="nix"> | |||
# shell.nix | |||
let | let | ||
pkgs = import <nixpkgs> {}; | pkgs = import <nixpkgs> {}; | ||
in pkgs.mkShell { | in pkgs.mkShell { | ||
packages = with pkgs; [ | |||
python310 | |||
(poetry.override { python3 = python310; }) | |||
]; | ]; | ||
} | } | ||
</syntaxhighlight> | </syntaxhighlight> | ||
==== poetry2nix ==== | |||
[https://github.com/nix-community/poetry2nix poetry2nix] uses the contents of a <code>poetry.lock</code> and <code>pyproject.toml</code> file to create Nix derivations. It has several functions for generating development environments and python projects. Because some older python projects rely on deprecated build systems (see [https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md edgecase.md] for more info), poetry2nix provides overrides so these packages can still be built. | |||
=== Using micromamba === | |||
</ | |||
Install the <code>micromamba</code> package to create environments and install packages as [https://github.com/mamba-org/mamba#micromamba documented by micromamba]. | |||
Install the <code>micromamba</code> package | |||
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.: | To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.: | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs. | $ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env' | ||
$ eval "$(micromamba shell hook -s bash)" | $ eval "$(micromamba shell hook -s bash)" | ||
$ micromamba activate my-environment | $ micromamba activate my-environment | ||
Line 338: | Line 368: | ||
{ pkgs ? import <nixpkgs> {}}: | { pkgs ? import <nixpkgs> {}}: | ||
let | let | ||
fhs = pkgs. | fhs = pkgs.buildFHSEnv { | ||
name = "my-fhs-environment"; | name = "my-fhs-environment"; | ||
Line 347: | Line 377: | ||
profile = '' | profile = '' | ||
set -e | set -e | ||
eval "$(micromamba shell hook - | eval "$(micromamba shell hook --shell=posix)" | ||
export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba | export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba | ||
micromamba create -q -n my-mamba-environment | if ! test -d $MAMBA_ROOT_PREFIX/envs/my-mamba-environment; then | ||
micromamba create --yes -q -n my-mamba-environment | |||
fi | |||
micromamba activate my-mamba-environment | micromamba activate my-mamba-environment | ||
micromamba install --yes -f conda-requirements.txt -c conda-forge | micromamba install --yes -f conda-requirements.txt -c conda-forge | ||
Line 357: | Line 389: | ||
in fhs.env | in fhs.env | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Using conda === | |||
=== conda === | |||
Install the package <code>conda</code> and run | Install the package <code>conda</code> and run | ||
<syntaxhighlight lang=" | <syntaxhighlight lang="console"> | ||
conda-shell | $ conda-shell | ||
conda-install | $ conda-install | ||
conda env update --file environment.yml | $ conda env update --file environment.yml | ||
</syntaxhighlight> | </syntaxhighlight> | ||
==== Imperative use ==== | ==== Imperative use ==== | ||
It is also possible to use <code>conda-install</code> directly. | It is also possible to use <code>conda-install</code> directly. On first use, run: | ||
On first use, run | <syntaxhighlight lang="console"> | ||
<syntaxhighlight lang=" | $ conda-shell | ||
conda-shell | $ conda-install | ||
conda-install | |||
</syntaxhighlight> | </syntaxhighlight> | ||
to set up conda in <code>~/.conda</code> | to set up conda in <code>~/.conda</code> | ||
=== | === Using pixi === | ||
[https://github.com/nix- | Install the <code>pixi</code> package to create environments and install packages as [https://pixi.sh/latest/ documented by pixi]:<syntaxhighlight lang="console"> | ||
$ pixi init | |||
$ pixi add python | |||
</syntaxhighlight>To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g. a [https://github.com/NixOS/nixpkgs/issues/316443#issuecomment-2151963505 flake.nix] <ref>Suggested flake.nix by [https://github.com/jonas-w @jonas-w] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos"> | |||
{ | |||
description = "pixi env"; | |||
inputs = { | |||
flake-utils.url = "github:numtide/flake-utils"; | |||
}; | |||
outputs = | |||
{ flake-utils, nixpkgs, ... }: | |||
flake-utils.lib.eachDefaultSystem ( | |||
system: | |||
let | |||
pkgs = import nixpkgs { inherit system; }; | |||
fhs = pkgs.buildFHSEnv { | |||
name = "pixi-env"; | |||
targetPkgs = _: [ pkgs.pixi ]; | |||
}; | |||
in | |||
{ | |||
devShell = fhs.env; | |||
} | |||
); | |||
} | |||
</syntaxhighlight>Then using <code>[https://wiki.nixos.org/wiki/Development_environment_with_nix-shell#nix_develop nix develop]</code> the environment can be activated:<syntaxhighlight lang="console"> | |||
$ nix develop | |||
$ pixi s | |||
$ python | |||
>>> import numpy as np | |||
</syntaxhighlight> | |||
==== Using Home Manager ==== | |||
Alternatively an [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] can wrap pixi using [[Home Manager]] e.g<ref>Suggested home manager config by [https://github.com/jennydaman @jennydaman] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos"> | |||
{ lib, pkgs, ... }: | |||
{ | |||
home.packages = [ | |||
(pkgs.buildFHSEnv { | |||
name = "pixi"; | |||
runScript = "pixi"; | |||
targetPkgs = pkgs: with pkgs; [ pixi ]; | |||
}) | |||
]; | |||
} | |||
</syntaxhighlight>Then the environment can be activated:<syntaxhighlight lang="console"> | |||
$ pixi s | |||
$ python | |||
>>> import numpy as np | |||
</syntaxhighlight> | |||
== Package a Python application == | |||
=== With <code>setup.py</code> === | |||
To package a Python application that uses <code>setup.py</code> you can use <code>buildPythonApplication</code>. More details about this and similar functions can be found in [https://nixos.org/manual/nixpkgs/stable/#building-packages-and-applications the nixpkgs manual]. | |||
For example, we can package this simple flask server <code>main.py:</code> | |||
<syntaxhighlight lang="python"> | |||
#!/usr/bin/env python | |||
from flask import Flask | |||
=== | app = Flask(__name__) | ||
@app.route('/') | |||
def hello_world(): | |||
return 'Hello, World!' | |||
if __name__ == '__main__': | |||
app.run(host="0.0.0.0", port=8080) | |||
</syntaxhighlight> | |||
We also need a <code>setup.py</code> file, like this: | |||
<syntaxhighlight lang="python"> | |||
from setuptools import setup, find_packages | |||
setup(name='myFlaskServer', | |||
version='1.0', | |||
# Modules to import from other scripts: | |||
packages=find_packages(), | |||
# Executables | |||
scripts=["main.py"], | |||
) | |||
</syntaxhighlight> | |||
Then, we use the <code>buildPythonApplication</code> in the <code>default.nix</code>: | |||
<syntaxhighlight lang="nix"> | <syntaxhighlight lang="nix"> | ||
{ pkgs ? import <nixpkgs> {} }: | |||
pkgs.python3Packages.buildPythonApplication { | |||
pname = "myFlaskApp"; | |||
version = "0.1.0"; | |||
propagatedBuildInputs = with pkgs.python3Packages; [ | |||
flask | |||
]; | |||
src = ./.; | |||
} | |||
</syntaxhighlight>Finally, build your project using <code>nix-build</code>. The result will be executable in <code>./result/bin/app.py</code>. | |||
=== With <code>pyproject.toml</code> === | |||
When your project is using <code>pyproject.toml</code>you can use [https://github.com/nix-community/pyproject.nix pyproject.nix] to package your application. | |||
First, a simple file structure could look like this:<syntaxhighlight> | |||
├── app/ | |||
└── main.py | |||
├── flake.nix | |||
├── pyproject.toml | |||
└── README.md | |||
</syntaxhighlight>To reuse the example from above, we use the same flask application:<syntaxhighlight lang="python"> | |||
from flask import Flask | |||
app = Flask(__name__) | |||
@app.route('/') | |||
def hello_world(): | |||
return 'Hello, World!' | |||
if __name__ == '__main__': | |||
app.run(host="0.0.0.0", port=8080) | |||
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://pyproject-nix.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml"> | |||
[project] | |||
name = "my-app" | |||
version = "0.1.0" | |||
description = "Simple app" | |||
# define any Python dependencies | |||
dependencies = [ | |||
"flask>3", | |||
] | |||
# define the CLI executable | |||
# Here, we define the entry point to be the 'main()' function in the module 'app/main.py' | |||
[project.scripts] | |||
cli = "app.main:main" | |||
</syntaxhighlight>We package the application by calling the <code>loadPyproject</code> function from <code>pyproject.nix</code>. Again, we only show a minimal example. More information can be found in the [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation]. Note that this example relies on flakes in contrast to some of the others on this page.<syntaxhighlight lang="nix"> | |||
{ | { | ||
description = "A basic flake using pyproject.toml project metadata"; | |||
inputs = { | |||
pyproject-nix = { | |||
url = "github:nix-community/pyproject.nix"; | |||
inputs.nixpkgs.follows = "nixpkgs"; | |||
}; | |||
}; | |||
outputs = { nixpkgs, pyproject-nix, ... }: | |||
let | |||
inherit (nixpkgs) lib; | |||
project = pyproject-nix.lib.project.loadPyproject { | |||
# Read & unmarshal pyproject.toml relative to this project root. | |||
# projectRoot is also used to set `src` for renderers such as buildPythonPackage. | |||
projectRoot = ./.; | |||
}; | |||
# This example is only using x86_64-linux | |||
pkgs = nixpkgs.legacyPackages.x86_64-linux; | |||
python = pkgs.python3; | |||
in | |||
{ | |||
# Build our package using `buildPythonPackage | |||
packages.x86_64-linux.default = | |||
let | |||
# Returns an attribute set that can be passed to `buildPythonPackage`. | |||
attrs = project.renderers.buildPythonPackage { inherit python; }; | |||
in | |||
# Pass attributes to buildPythonPackage. | |||
# Here is a good spot to add on any missing or custom attributes. | |||
python.pkgs.buildPythonPackage (attrs // { | |||
env.CUSTOM_ENVVAR = "hello"; | |||
}); | |||
}; | |||
} | } | ||
</syntaxhighlight>To run the application, call <code>nix run</code>. | |||
You can also launch an IDE under <code>nix develop</code> and get full dependency resolution. For example, the following command opens VS Code in the constructed environment: | |||
<syntaxhighlight lang="shell-session"> | |||
$ nix develop --command code | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Nixpkgs Python contribution guidelines == | |||
=== | === Libraries === | ||
Python | According to the [https://nixos.org/nixpkgs/manual/#contributing-guidelines official guidelines] for Python, new package expressions for libraries should be placed in <syntaxhighlight lang="bash" inline>pkgs/development/python-modules/<name>/default.nix</syntaxhighlight>. | ||
Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> as in | |||
<syntaxhighlight lang="nix"> | <syntaxhighlight lang="nix"> | ||
{ | aenum = callPackage ../development/python-modules/aenum { }; | ||
</syntaxhighlight> | |||
=== Applications === | |||
Applications meant to be executed should be referenced directly from <code>pkgs/top-level/all-packages.nix</code>. | |||
Other Python packages used in the Python package of the application should be taken from the <code>callPackage</code> argument <code>pythonPackages</code> , which guarantees that they belong to the same "pythonPackage" set. For example: | |||
buildPythonApplication | <syntaxhighlight lang="nix" line="1"> | ||
# ... | { | ||
lib, | |||
pythonPackages, | |||
}: | |||
buildPythonApplication { | |||
propagatedBuildInputs = [ pythonPackages.numpy ]; | |||
# ... | |||
} | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Special Modules == | == Special Modules == | ||
Line 436: | Line 647: | ||
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>. | You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>. | ||
== | == Debug Build == | ||
See [https://docs.python.org/3/using/configure.html#python-debug-build python wiki on debug build]. | |||
In order to use a CPython interpreter built using <code>--with-pydebug</code> during configure phase, override any of the python packages passing <code>enableDebug = true</code> argument: | |||
<code>pythonDebug = pkgs.python310.override { enableDebug = true; };</code> | |||
== Installing Multiple Versions == | |||
{{Nixpkgs Manual|name={{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}}|anchor=#function-library-lib.meta.lowPrio}} can be used to install multiple versions without conflicts ({{issue|369997}}): | |||
{{file|/etc/nixos/configuration.nix|nix|3= | |||
environment.systemPackages = with pkgs; [ | |||
python313 | |||
lib.meta.lowPrio python314 | |||
]; | |||
}} | |||
In this case you may run Python 3.13 via {{ic|python}} or {{ic|python3.13}} and Python 3.14 via {{ic|python3.14}}. | |||
{{Note|If you wrapped Python as described in [[#Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix]], call {{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}} for the wrapped Python instead of calling them inside the wrapping.}} | |||
== Performance == | |||
The derivation of CPython that is available via <code>nixpkgs</code> only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options] | |||
Additionally, when compiling something within <code>nix-shell</code> or a derivation security hardening flags are passed to the compiler by default which may have a small performance impact. | |||
At the time of writing certain optimizations cause Python wheels to be non-reproducible and increase install times. For a detailed overview of the trials and tribulations of discovering such performance regressions see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?]. | |||
=== Regression === | === Regression === | ||
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example: | With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example: | ||
Line 454: | Line 685: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
However, synthetic benchmarks are not | However, synthetic benchmarks are not necessarily reflective of real-world performance. In common real-world situations, the performance difference between optimized and non-optimized interpreters is minimal. For example, using <code>pylint</code> with a significant number of custom linters to scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference. Other workflows that were not performance sensitive saw no impact to their run times. | ||
=== Possible Optimizations === | === Possible Optimizations === | ||
If you run code that heavily depends on Python performance | If you run code that heavily depends on Python performance, and you desire the most performant Python interpreter possible, here are some possible things you can do: | ||
* Enable the <code>enableOptimizations</code> flag for your Python derivation. [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example] Do note that this will cause you to compile Python the first time that you run it | * '''Enable the <code>enableOptimizations</code> flag for your Python derivation'''. See [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example]. Do note that this will cause you to compile Python the first time that you run it which will take a few minutes. | ||
* Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement | * '''Switch to a newer version of Python'''. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement, but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster. | ||
* Disable hardening | * '''Disable hardening'''. Beware this only yields a small performance boost and it has impacts beyond Python code. See [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs]. | ||
'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.''' | '''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.''' | ||
== | == Troubleshooting == | ||
=== My module cannot be imported === | === My module cannot be imported === | ||
Line 473: | Line 702: | ||
First, make sure that you installed/added your module to python. Typically you would use something like <code> (python3.withPackages (ps: with ps; [ yourmodule ]))</code> in the list of installed applications. | First, make sure that you installed/added your module to python. Typically you would use something like <code> (python3.withPackages (ps: with ps; [ yourmodule ]))</code> in the list of installed applications. | ||
It is also still possible (e.g. when using nix-shell) that you aren't using the python interpreter you want because another package provides its own <code>python3.withPackages</code> in buildInputs, for example, yosys. In this case, you should either include that package (or all needed packages) in your withPackages list to only have a single Python interpreter. Or you can change the order of your packages, such that the <code>python3.withPackages</code> comes first, and becomes the Python interpreter that you get. | |||
If you packaged yourself your application, make sure to use <code>buildPythonPackage</code> and **not** <code>buildPythonApplication</code> or <code>stdenv.mkDerivation</code>. The reason is that <code>python3.withPackages</code> [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L57 filters] the packages to check that they are built using the appropriate python interpreter: this is done by verifying that the derivation has a <code>pythonModule</code> attribute and only buildPythonPackage [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L43 sets this value] (passthru [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L75 here]) thanks to, notably <code>passthru = { pythonModule = python; }</code>. If you used <code>stdenv.mkDerivation</code> then you can maybe set this value manually, but it's safer to simply use <code>buildPythonPackage {format = "other"; … your derivation …}</code> instead of <code>mkDerivation</code>. | |||
== See also == | == See also == | ||
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual] | |||
* [https:// | |||
[[Category:Languages]] | [[Category:Languages]] | ||
[[Category:Python]] |