Jump to content

Python: Difference between revisions

From NixOS Wiki
imported>2gn
m Upgraded mach-nix version from 3.4.0 to 3.5.0 . Micromamba is now in stable channel. Removed old archive link of nixpkgs (This change may make the document less reproducible.)
Lukeb (talk | contribs)
fixed broken link for pyproject use-cases
 
(85 intermediate revisions by 51 users not shown)
Line 1: Line 1:


The Python packages available to the interpreter must be declared when installing Python.
== Python development environments with Nix ==
Nix supports a number of approaches to creating "development environments" for Python programming. These provide functionality analogous to [https://virtualenv.pypa.io/en/latest/ virtualenv] or [https://docs.conda.io/en/latest/ conda]: a shell environment with access to pinned versions of the <code>python</code> executable and Python packages.


To install, say Python 3 with <code>pandas</code> and <code>requests</code>, define a new package <code>python-with-my-packages</code>:
=== Using the Nixpkgs Python infrastructure via <code>shell.nix</code> (recommended) ===
<syntaxhighlight lang="nix">
Nixpkgs has the few last Python versions packaged, as well as a consequent set of Python packages packaged that you can use to quickly create a Python environment.
with pkgs;
 
Create a file <code>shell.nix</code> in the project directory, with the following template:
 
<syntaxhighlight lang="nix" line="1">
# shell.nix
let
let
   my-python-packages = python-packages: with python-packages; [
   # We pin to a specific nixpkgs commit for reproducibility.
    pandas
  # Last updated: 2024-04-29. Check for new commits at https://status.nixos.org.
    requests
  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/cf8cc1201be8bc71b7cbbbdaf349b22f4f99c7ae.tar.gz") {};
     # other python packages you want
in pkgs.mkShell {
   ];  
  packages = [
  python-with-my-packages = python3.withPackages my-python-packages;
    (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [
in ...
      # select Python packages here
      pandas
      requests
     ]))
   ];
}
</syntaxhighlight>
</syntaxhighlight>


You can put <code>python-with-my-packages</code> into your environment.systemPackages for a system-wide installation, for instance. Be mindful that <code>pythonX.withPackages</code> creates a <code>pythonX-Y.Z.W-env</code> package which is read only, so you can't use <code>pip</code> to install packages in, say, a virtual environment (as described below). Put <code>python</code> in your <code>configuration.nix</code> if you want to use the solution as described in the section "Python Virtual Environment".
In this example, we create a Python environment with packages <code>pandas</code> and <code>requests</code>.


There are several versions of Python available. Replace <code>python3</code> with <code>python2</code> or <code>pypy</code> in the above snippet according to your needs.
You can find Python packages that are available in Nixpkgs using [https://search.nixos.org/packages search.nixos.org]. For instance, type a Python package name like <code>numpy</code> in the search bar and click on the search button on the right. You can narrow down results by clicking on eg. "python311Packages" in the "Package sets" section on the left. Note that in the snippet above, on lines 8 and 9, each package is listed in the form <code>python-pkgs.<name></code> where <code><name></code> corresponds to the one found in [https://search.nixos.org/packages search.nixos.org] . See [https://nix.dev/tutorials/nix-language.html Nix language basics] for more information on the <code>python-pkgs</code> attribute set.


=== Explanation (optional) ===
Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>) to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.


We defined a function <code>my-python-packages</code> which takes as input a set <code>python-packages</code> and returns a list of attributes thereof.
Note that with NixOS, this method can be used to install packages at the system level, e.g. <syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
  # ...
  (python3.withPackages (python-pkgs: with python-pkgs; [
      pandas
      requests
  ]))
];
</syntaxhighlight>


=== Example ===
=== Using Nix shell (new command line) ===
nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])'
If you don't use the channels any more, you can replace <code><nixpkgs></code> by an instance of the <code>NixOS/nixpkgs</code> repository using its absolute path.


One way to define the python package with modules inside the default configuration file is to use the systemPackages list. In such a case, you could have the following under <code>environment.systemPackages = with pkgs;</code> along with the rest of your packages:
==== Using a Python package not in Nixpkgs ====
<syntaxhighlight lang="nix">
Python packages in Nixpkgs are created and updated by Nixpkgs maintainers. Although the community invests a great effort to keep a complete and up-to-date package set, some packages you want may be missing, out of date, or broken. To use your own packages in a Nix environment, you may package it yourself.
(let
 
   my-python-packages = python-packages: with python-packages; [
The following is a high-level overview. For a complete explanation, see [https://nixos.org/manual/nixpkgs/unstable/#developing-with-python Developing with Python] in the Nixpkgs Manual.
    pandas
 
    requests
Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1">
    #other python packages you want
# toolz.nix
   ];
{
   python-with-my-packages = python3.withPackages my-python-packages;
   lib,
in
  buildPythonPackage,
python-with-my-packages)
  fetchPypi,
</syntaxhighlight>
   setuptools,
   wheel,
}:


== Development shell ==
buildPythonPackage rec {
  pname = "toolz";
  version = "0.10.0";


=== withPackages env ===
  src = fetchPypi {
    inherit pname version;
    hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
  };


If you need only python:
  # do not run tests
  doCheck = false;


<syntaxhighlight lang="nix">
  # specific to buildPythonPackage, see its reference
  pyproject = true;
  build-system = [
    setuptools
    wheel
  ];
}
</syntaxhighlight>Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
# shell.nix
# shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
let
   python-with-my-packages = pkgs.python3.withPackages (p: with p; [
  pkgs = import <nixpkgs> {};
    pandas
 
    requests
   python = pkgs.python3.override {
     # other python packages you want
    self = python;
   ]);
    packageOverrides = pyfinal: pyprev: {
in
      toolz = pyfinal.callPackage ./toolz.nix { };
python-with-my-packages.env # replacement for pkgs.mkShell
    };
</syntaxhighlight>
  };
 
in pkgs.mkShell {
  packages = [
    (python.withPackages (python-pkgs: [
      # select Python packages here
      python-pkgs.pandas
      python-pkgs.requests
      python-pkgs.toolz
     ]))
  ];
}
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
 
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> ===
 
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
Traceback (most recent call last):
   File "<string>", line 1, in <module>
  File "/.../grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/.../grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
 
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.


=== mkShell ===
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>.


If you need python and other dependencies:
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers.


==== Using nix overlay ====
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
# shell.nix
{
{ pkgs ? import <nixpkgs> {} }:
  nixpkgs.overlays = [(
let
    self: super: rec {
  my-python = pkgs.python3;
      # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  python-with-my-packages = my-python.withPackages (p: with p; [
      # > We currently take all libraries from systemd and nix as the default.
    pandas
      pythonldlibpath = lib.makeLibraryPath (with super; [
    requests
        zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
     # other python packages you want
      ]);
   ]);
      # here we are overriding python program to add LD_LIBRARY_PATH to it's env
in
      python = super.stdenv.mkDerivation {
pkgs.mkShell {
        name = "python";
  buildInputs = [
        buildInputs = [ super.makeWrapper ];
     python-with-my-packages
        src = super.python311;
     # other dependencies
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/python3 --set LD_LIBRARY_PATH ${pythonldlibpath}
          wrapProgram $out/bin/python3.11 --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
      poetry = super.stdenv.mkDerivation {
        name = "poetry";
        buildInputs = [ super.makeWrapper ];
        src = super.poetry;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
     }
   )];
  environment.systemPackages = with pkgs; [
     python # here python will be taken from the overlay up here
     poetry
   ];
   ];
  shellHook = ''
    PYTHONPATH=${python-with-my-packages}/${python-with-my-packages.sitePackages}
    # maybe set more env-vars
  '';
}
}
</syntaxhighlight>
</syntaxhighlight>


== Using alternative packages ==
==== Using [https://github.com/Mic92/nix-ld nix-ld] ====
 
We saw above how to install Python packages using nixpkgs. Since these are written by hand by nixpkgs maintainers, it isn't uncommon for packages you want to be missing or out of date. To create a custom Python environment with your own package(s), first create a derivation for each python package (look at examples in the <code>python-modules</code> subfolder in Nixpkgs). Then, use those derivations with <code>callPackage</code> as follows:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
with pkgs;
{
let
  programs.nix-ld = {
   my-python-package = ps: ps.callPackage ./my-package.nix {};
    enable = true;
  python-with-my-packages = python3.withPackages(ps: with ps; [
    libraries = with pkgs; [
     (my-python-package ps)
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
   ]);
    ];
in ...
  };
   # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
     (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
    '')
   ];
  # another (dangerous) solution
  # environment.systemPackages = with pkgs; [ python3 ];
  # programs.bash = {
  #  enable = true;
  #  initExtra = ''
  #    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #  '';
  # };
}
</syntaxhighlight>
</syntaxhighlight>


=== Package and development shell for a python project ===
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ====
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}}


It is possible to use <code>buildPythonApplication</code> to package python applications. As explained in the nixpkgs manual, it uses the widely used `setup.py` file in order to package properly the application. We now show how to package a simple python application: a basic flask web server.
Install with:


First, we write the python code, say in a file <code>web_interface.py</code>. Here we create a basic flask web server;
<syntaxhighlight lang="shell">
<syntaxhighlight lang="python">
nix profile install github:GuillaumeDesforges/fix-python
#!/usr/bin/env python
</syntaxhighlight>


from flask import Flask
Enter the venv and run:
app = Flask(__name__)


@app.route('/')
<syntaxhighlight lang="shell">
def hello_world():
fix-python --venv .venv
    return 'Hello, World!'
 
if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>
</syntaxhighlight>


Then, we create the <code>setup.py</code> file, which basically explains which are the executables:
==== Using <code>buildFHSEnv</code> (Recommended) ====
<syntaxhighlight lang="python">
<syntaxhighlight lang="nix">
#!/usr/bin/env python
#!/usr/bin/env nix-shell
 
{ pkgs ? import <nixpkgs> { } }:
from setuptools import setup, find_packages
(
 
  let base = pkgs.appimageTools.defaultFhsEnvArgs; in
setup(name='demo-flask-vuejs-rest',
  pkgs.buildFHSEnv (base // {
      version='1.0',
    name = "FHS";
       # Modules to import from other scripts:
    targetPkgs = pkgs: (with pkgs; [
      packages=find_packages(),
       gcc glibc zlib
      # Executables
    ]);
      scripts=["web_interface.py"],
    runScript = "zsh";
    )
    extraOutputsToInstall = [ "dev" ];
  })
).env
</syntaxhighlight>
</syntaxhighlight>


Finally, our nix derivation is now trivial: the file <code>derivation.nix</code> just needs to provide the python packages (here flask):
==== Using a custom nix-shell ====
<syntaxhighlight lang="nix">
{ lib, python3Packages }:
with python3Packages;
buildPythonApplication {
  pname = "demo-flask-vuejs-rest";
  version = "1.0";


   propagatedBuildInputs = [ flask ];
The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1">
let
   python = pkgs.python311;
  # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
  ]);
  patchedpython = (python.overrideAttrs (
    previousAttrs: {
      # Add the nix-ld libraries to the LD_LIBRARY_PATH.
      # creating a new library path from all desired libraries
      postInstall = previousAttrs.postInstall + ''
        mv  "$out/bin/python3.11" "$out/bin/unpatched_python3.11"
        cat << EOF >> "$out/bin/python3.11"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_python3.11" "\$@"
        EOF
        chmod +x "$out/bin/python3.11"
      '';
    }
  ));
  # if you want poetry
  patchedpoetry =  ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs (
    previousAttrs: {
      # same as above, but for poetry
      # not that if you dont keep the blank line bellow, it crashes :(
      postInstall = previousAttrs.postInstall + ''


  src = ./.;
        mv "$out/bin/poetry" "$out/bin/unpatched_poetry"
        cat << EOF >> "$out/bin/poetry"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_poetry" "\$@"
        EOF
        chmod +x "$out/bin/poetry"
      '';
    }
  ));
in
{
  environment.systemPackages = with pkgs; [
    patchedpython
    # if you want poetry
    patchedpoetry
  ];
}
}
</syntaxhighlight>
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.


and we can now load this derivation from our file <code>default.nix</code>:
==== Prefix library paths using wrapProgram ====
<syntaxhighlight lang="nix">
wrapProgram is a part of the makeWrapper build input<ref>https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/make-wrapper.sh</ref>. By combining it with the symlinkJoin, we can create a wrapper around the Python executable that will always set the required library paths. It’s worth noting that, for this solution to be compatible with Darwin, we need to use a different wrap prefix, as shown in the example below.<syntaxhighlight lang="nixos" line="1">
{ pkgs ? import <nixpkgs> {} }:
let
pkgs.callPackage ./derivation.nix {}
  # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
  ]);
  # Darwin requires a different library path prefix
  wrapPrefix = if (!pkgs.stdenv.isDarwin) then "LD_LIBRARY_PATH" else "DYLD_LIBRARY_PATH";
  patchedpython = (pkgs.symlinkJoin {
    name = "python";
    paths = [ pkgs.python312 ];
    buildInputs = [ pkgs.makeWrapper ];
    postBuild = ''
      wrapProgram "$out/bin/python3.12" --prefix ${wrapPrefix} : "${pythonldlibpath}"
    '';
  });
in
{
  environment.systemPackages = with pkgs; [
    patchedpython
  ];
}
</syntaxhighlight>
</syntaxhighlight>


We can now build with:
=== Using <code>venv</code> ===
<syntaxhighlight>
$ nix-build
[...]
$ ./result/bin/web_interface.py
* Serving Flask app ".web_interface" (lazy loading)
[...]
</syntaxhighlight>
or just enter a nix-shell, and directly execute your program or python if it's easier to develop:
<syntaxhighlight>
$ nix-shell
[...]
[nix-shell]$ chmod +x web_interface.py
[nix-shell]$ ./web_interface.py
* Serving Flask app "web_interface" (lazy loading)
[...]


[nix-shell]$ python
To create a Python virtual environment with <code>venv</code>:<syntaxhighlight lang="console">
Python 3.8.7 (default, Dec 21 2020, 17:18:55)
$ nix-shell -p python3 --command "python -m venv .venv --copies"
[GCC 10.2.0] on linux
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>
</syntaxhighlight>


=== Python virtual environment ===
=== Using uv ===
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>


Starting from Python 3 virtual environment is natively supported.
uv is written in Rust and does ''not'' need Python as a prerequisite. Use the <code>uv</code> command to initialize Python projects, add Python packages, create or update virtual environments (in <code>.venv</code> folders), etc. as [https://docs.astral.sh/uv/concepts/projects/ described in the uv docs]. Use uv's [https://docs.astral.sh/uv/guides/tools/ tool interface] to install and run Python packages that provide a CLI.
The Python 3 venv approach has the benefit of forcing you to choose a specific version of the Python 3 interpreter that should be used to create the virtual environment. This avoids any confusion as to which Python installation the new environment is based on.


Recommended usage:
As a system package
* Python 3.3-3.4 (old): the recommended way to create a virtual environment was to use the pyvenv command-line tool that also comes included with your Python 3 installation by default.
* Python 3.6+: <code>python3 -m venv</code> is the way to go.


Put your packages in a requirements.txt:
<syntaxhighlight lang="nix">
<syntaxhighlight>
environment.systemPackages = with pkgs; [
pandas
    uv
requests
];
</syntaxhighlight>
 
Then setup the virtualenv:
<syntaxhighlight lang="shell">
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
</syntaxhighlight>
</syntaxhighlight>


Installing packages with <code>pip</code> that need to compile code or use C libraries will sometimes fail due to not finding dependencies in the expected places. In that case you can use <code>buildFHSUserEnv</code> to make yourself a sandbox that appears like a more typical Linux install. For example if you were working with machine learning code you could use:
or as a home-manager package


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
home.packages = with pkgs; [
(pkgs.buildFHSUserEnv {
     uv
  name = "pipzone";
];
  targetPkgs = pkgs: (with pkgs; [
     python39
    python39Packages.pip
    python39Packages.virtualenv
    cudaPackages.cudatoolkit_11
  ]);
  runScript = "bash";
}).env
</syntaxhighlight>
</syntaxhighlight>


In <code>pip-shell.nix</code>, and enter the environment with:
If you use uv it's recommended that you install the Python versions you need using the <code>uv python install</code> command, e.g.


<syntaxhighlight lang="shell">
<syntaxhighlight lang="python">
nix-shell pip-shell.nix
uv python install 3.14 --preview --default
virtualenv venv
source venv/bin/activate
</syntaxhighlight>
</syntaxhighlight>


You may want to set the <code>UV_PYTHON_DOWNLOADS=never</code> environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting <code>environment.localBinInPath = true;</code> is highly recommended, because uv will install binaries in <code>~/.local/bin</code>.


=== Emulating virtualenv with nix-shell ===
=== Using poetry ===
In some cases virtualenv fails to install a library because it requires patching on NixOS (example 1, example 2, general issue). In this cases it is better to replace those libraries with ones from Nix.
<syntaxhighlight lang="nix">


Let's say, that nanomsg library fails to install in virtualenv. Then write a <code>shell.nix</code> file:
# shell.nix
 
<syntaxhighlight lang="nix">
let
let
   pkgs = import <nixpkgs> {};
   pkgs = import <nixpkgs> {};
  nanomsg-py = ...build expression for this python library...;
in pkgs.mkShell {
in pkgs.mkShell {
   buildInputs = [
   packages = with pkgs; [
     pkgs.python3
     python310
     pkgs.python3.pkgs.requests
     (poetry.override { python3 = python310; })
    nanomsg-py
   ];
   ];
  shellHook = ''
    # Tells pip to put packages into $PIP_PREFIX instead of the usual locations.
    # See https://pip.pypa.io/en/stable/user_guide/#environment-variables.
    export PIP_PREFIX=$(pwd)/_build/pip_packages
    export PYTHONPATH="$PIP_PREFIX/${pkgs.python3.sitePackages}:$PYTHONPATH"
    export PATH="$PIP_PREFIX/bin:$PATH"
    unset SOURCE_DATE_EPOCH
  '';
}
}
</syntaxhighlight>
</syntaxhighlight>


After entering the environment with `nix-shell`, you can install new python libraries with dump `pip install`, but nanomsg will be detected as installed.
==== poetry2nix ====
 
[https://github.com/nix-community/poetry2nix poetry2nix] uses the contents of a <code>poetry.lock</code> and <code>pyproject.toml</code> file to create Nix derivations. It has several functions for generating development environments and python projects. Because some older python projects rely on deprecated build systems (see [https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md edgecase.md] for more info), poetry2nix provides overrides so these packages can still be built.
Discussion and consequences of this approach are in PR https://github.com/NixOS/nixpkgs/pull/55265.
=== Using micromamba ===
 
=== mach-nix (nixify requirements.txt) ===
Mach-nix is a tool that allows managing python environments with nix based on a `requirements.txt` file.
There are two different ways of how mach-nix can be used, either by installing the mach-nix cmdline tool or by writing a nix expression. The latter is recommended for maximum reproducibility.
 
An example for a nix expression using mach-nix to build a python environment defined by a list of requirements.
<syntaxhighlight lang="nix">
let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    # place version number with the latest one from the github releases page
    ref = "refs/tags/3.5.0";
  }) {};
in
mach-nix.mkPython {
  # contents of a requirements.txt (use builtins.readFile ./requirements.txt alternatively)
  requirements = ''
    pillow
    numpy
    requests
  '';
}
</syntaxhighlight>


Alternatively install the mach-nix cmdline tool
Install the <code>micromamba</code> package to create environments and install packages as [https://github.com/mamba-org/mamba#micromamba documented by micromamba].
<code>nix-env -if https://github.com/DavHau/mach-nix/tarball/3.5.0 -A mach-nix</code>
and run
<syntaxhighlight lang="shell">
mach-nix env ./env -r requirements.txt
# This will generate the python environment into ./env. To activate it, execute:
nix-shell ./env
</syntaxhighlight>
 
Or use a single command based on nix' flakes feature (if you have flake installed, just run the command <code>nix-run ...</code>).
<syntaxhighlight lang="shell">
# nix.extraOptions = ''  experimental-features = nix-command flakes    '';
nix-shell -p nixFlakes --run "nix run github:davhau/mach-nix#gen.python.pillow.numpy.requests --show-trace "
</syntaxhighlight>
 
Please have a look at more [https://github.com/DavHau/mach-nix/blob/master/examples.md examples].
 
=== micromamba ===
 
Install the <code>micromamba</code> package.
You can create environments and install packages as [https://github.com/mamba-org/mamba#micromamba documented by micromamba] e.g.
<syntaxhighlight lang="shell">
micromamba create -n my-environment python=3.9 numpy=1.23.0 -c conda-forge
</syntaxhighlight>


To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="shell">
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ micromamba activate my-environment
Line 315: Line 368:
{ pkgs ? import <nixpkgs> {}}:
{ pkgs ? import <nixpkgs> {}}:
let
let
   fhs = pkgs.buildFHSUserEnv {
   fhs = pkgs.buildFHSEnv {
     name = "my-fhs-environment";
     name = "my-fhs-environment";


Line 324: Line 377:
     profile = ''
     profile = ''
       set -e
       set -e
       eval "$(micromamba shell hook -s bash)"
       eval "$(micromamba shell hook --shell=posix)"
       export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
       export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
       micromamba create -q -n my-mamba-environment
       if ! test -d $MAMBA_ROOT_PREFIX/envs/my-mamba-environment; then
          micromamba create --yes -q -n my-mamba-environment
      fi
       micromamba activate my-mamba-environment
       micromamba activate my-mamba-environment
       micromamba install --yes -f conda-requirements.txt -c conda-forge
       micromamba install --yes -f conda-requirements.txt -c conda-forge
Line 334: Line 389:
in fhs.env
in fhs.env
</syntaxhighlight>
</syntaxhighlight>
 
=== Using conda ===
 
=== conda ===


Install the package <code>conda</code> and run  
Install the package <code>conda</code> and run  
<syntaxhighlight lang="shell">
<syntaxhighlight lang="console">
conda-shell
$ conda-shell
conda-install
$ conda-install
conda env update --file environment.yml
$ conda env update --file environment.yml
</syntaxhighlight>
</syntaxhighlight>


==== Imperative use ====
==== Imperative use ====


It is also possible to use <code>conda-install</code> directly.
It is also possible to use <code>conda-install</code> directly. On first use, run:
On first use, run
<syntaxhighlight lang="console">
<syntaxhighlight lang="shell">
$ conda-shell
conda-shell
$ conda-install
conda-install
</syntaxhighlight>
</syntaxhighlight>
to set up conda in <code>~/.conda</code>
to set up conda in <code>~/.conda</code>


=== pip2nix ===
=== Using pixi ===
[https://github.com/nix-community/pip2nix pip2nix] generate nix expressions for Python packages.
Install the <code>pixi</code> package to create environments and install packages as [https://pixi.sh/latest/ documented by pixi]:<syntaxhighlight lang="console">
$ pixi init
$ pixi add python
</syntaxhighlight>To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g. a [https://github.com/NixOS/nixpkgs/issues/316443#issuecomment-2151963505 flake.nix] <ref>Suggested flake.nix by [https://github.com/jonas-w @jonas-w] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos">
{
  description = "pixi env";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs =
    { flake-utils, nixpkgs, ... }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs { inherit system; };
        fhs = pkgs.buildFHSEnv {
          name = "pixi-env";


Also see the [https://github.com/nix-community/pypi2nix pypi2nix]-project (abandoned in 2019).
          targetPkgs = _: [ pkgs.pixi ];
        };
      in
      {
        devShell = fhs.env;
      }
    );
}
</syntaxhighlight>Then using  <code>[https://wiki.nixos.org/wiki/Development_environment_with_nix-shell#nix_develop nix develop]</code> the environment can be activated:<syntaxhighlight lang="console">
$ nix develop
$ pixi s
$ python
>>> import numpy as np
</syntaxhighlight>
 
==== Using Home Manager ====
Alternatively an [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] can wrap pixi using [[Home Manager]] e.g<ref>Suggested home manager config by [https://github.com/jennydaman @jennydaman] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos">
{ lib, pkgs, ... }:
{
  home.packages = [
    (pkgs.buildFHSEnv {
      name = "pixi";
      runScript = "pixi";
      targetPkgs = pkgs: with pkgs; [ pixi ];
    })
  ];
}
</syntaxhighlight>Then the environment can be activated:<syntaxhighlight lang="console">
$ pixi s
$ python
>>> import numpy as np
</syntaxhighlight>


== Contribution guidelines ==
== Package a Python application ==


=== Libraries ===
=== With <code>setup.py</code> ===
To package a Python application that uses <code>setup.py</code> you can use <code>buildPythonApplication</code>. More details about this and similar functions can be found in [https://nixos.org/manual/nixpkgs/stable/#building-packages-and-applications the nixpkgs manual].
 
For example, we can package this simple flask server <code>main.py:</code>
<syntaxhighlight lang="python">
#!/usr/bin/env python
 
from flask import Flask
 
app = Flask(__name__)
 
@app.route('/')
def hello_world():
    return 'Hello, World!'
 
if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>
 
We also need a <code>setup.py</code> file, like this:
<syntaxhighlight lang="python">
from setuptools import setup, find_packages


According to the [https://nixos.org/nixpkgs/manual/#contributing-guidelines official guidelines] for python new package expressions for libraries should be placed in <syntaxhighlight lang="bash" inline>pkgs/development/python-modules/<name>/default.nix</syntaxhighlight>. Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> like in this example:
setup(name='myFlaskServer',
      version='1.0',
      # Modules to import from other scripts:
      packages=find_packages(),
      # Executables
      scripts=["main.py"],
    )
</syntaxhighlight>


Then, we use the <code>buildPythonApplication</code> in the <code>default.nix</code>:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
pkgs.python3Packages.buildPythonApplication {
  pname = "myFlaskApp";
  version = "0.1.0";
  propagatedBuildInputs = with pkgs.python3Packages; [
    flask
  ];
  src = ./.;
}
</syntaxhighlight>Finally, build your project using <code>nix-build</code>. The result will be executable in <code>./result/bin/app.py</code>.
=== With <code>pyproject.toml</code> ===
When your project is using <code>pyproject.toml</code>you can use [https://github.com/nix-community/pyproject.nix pyproject.nix] to package your application.
First, a simple file structure could look like this:<syntaxhighlight>
├── app/
    └── main.py
├── flake.nix
├── pyproject.toml
└── README.md
</syntaxhighlight>To reuse the example from above, we use the same flask application:<syntaxhighlight lang="python">
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
    return 'Hello, World!'
if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://pyproject-nix.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
[project]
name = "my-app"
version = "0.1.0"
description = "Simple app"
# define any Python dependencies
dependencies = [
  "flask>3",
]
# define the CLI executable
# Here, we define the entry point to be the 'main()' function in the module 'app/main.py'
[project.scripts]
cli = "app.main:main"
</syntaxhighlight>We package the application by calling the <code>loadPyproject</code> function from <code>pyproject.nix</code>. Again, we only show a minimal example. More information can be found in the [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation]. Note that this example relies on flakes in contrast to some of the others on this page.<syntaxhighlight lang="nix">
{
{
   aenum = callPackage ../development/python-modules/aenum { };
   description = "A basic flake using pyproject.toml project metadata";
 
  inputs = {
    pyproject-nix = {
      url = "github:nix-community/pyproject.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
 
  outputs = { nixpkgs, pyproject-nix, ... }:
    let
      inherit (nixpkgs) lib;
 
      project = pyproject-nix.lib.project.loadPyproject {
        # Read & unmarshal pyproject.toml relative to this project root.
        # projectRoot is also used to set `src` for renderers such as buildPythonPackage.
        projectRoot = ./.;
      };
 
      # This example is only using x86_64-linux
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
 
      python = pkgs.python3;
 
    in
    {
      # Build our package using `buildPythonPackage
      packages.x86_64-linux.default =
        let
          # Returns an attribute set that can be passed to `buildPythonPackage`.
          attrs = project.renderers.buildPythonPackage { inherit python; };
        in
        # Pass attributes to buildPythonPackage.
        # Here is a good spot to add on any missing or custom attributes.
        python.pkgs.buildPythonPackage (attrs // {
          env.CUSTOM_ENVVAR = "hello";
        });
    };
}
}
</syntaxhighlight>To run the application, call <code>nix run</code>.
You can also launch an IDE under <code>nix develop</code> and get full dependency resolution. For example, the following command opens VS Code in the constructed environment:
<syntaxhighlight lang="shell-session">
$ nix develop --command code
</syntaxhighlight>
</syntaxhighlight>


The reasoning behind this is the large size of <code>pkgs/top-level/python-packages.nix</code>.
== Nixpkgs Python contribution guidelines ==


=== Applications ===
=== Libraries ===


Python applications instead should be referenced directly from <code>pkgs/top-level/all-packages.nix</code>.
According to the [https://nixos.org/nixpkgs/manual/#contributing-guidelines official guidelines] for Python, new package expressions for libraries should be placed in <syntaxhighlight lang="bash" inline>pkgs/development/python-modules/<name>/default.nix</syntaxhighlight>.


The expression should take <code>pythonPackages</code> as one of the arguments, which guarantees that packages belong to the same set. For example:
Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> as in


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ lib
  aenum = callPackage ../development/python-modules/aenum { };
, pythonPackages
</syntaxhighlight>
}:
 
=== Applications ===
 
Applications meant to be executed should be referenced directly from <code>pkgs/top-level/all-packages.nix</code>.


with pythonPackages;
Other Python packages used in the Python package of the application should be taken from the <code>callPackage</code> argument <code>pythonPackages</code> , which guarantees that they belong to the same "pythonPackage" set. For example:


buildPythonApplication rec {
<syntaxhighlight lang="nix" line="1">
# ...
{
  lib,
  pythonPackages,
}:
buildPythonApplication {
  propagatedBuildInputs = [ pythonPackages.numpy ];
  # ...
}
</syntaxhighlight>
</syntaxhighlight>
== Special Modules ==
== Special Modules ==
Line 399: Line 633:


Or, if you want to use matplotlib interactively:
Or, if you want to use matplotlib interactively:
<syntaxhighlight lang=shell>
<syntaxhighlight lang=console>
$ nix-shell -p gobjectIntrospection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'
$ nix-shell -p gobject-introspection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'
$ ipython
$ ipython
</syntaxhighlight>
</syntaxhighlight>
Line 413: Line 647:
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.


== Performance ==
== Debug Build ==
The derivation of cPython that is available via <code>nixpkgs</code> does not contain optimizations enabled, specifically Profile Guided Optimization (PGO) and Link Time Optimization (LTO). See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options]
See [https://docs.python.org/3/using/configure.html#python-debug-build python wiki on debug build].
Additionally, when you compile something within <code>nix-shell</code> or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.
 
In order to use a CPython interpreter built using <code>--with-pydebug</code> during configure phase, override any of the python packages passing <code>enableDebug = true</code> argument:
 
<code>pythonDebug = pkgs.python310.override { enableDebug = true; };</code>
 
== Installing Multiple Versions ==
 
{{Nixpkgs Manual|name={{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}}|anchor=#function-library-lib.meta.lowPrio}} can be used to install multiple versions without conflicts ({{issue|369997}}):
 
{{file|/etc/nixos/configuration.nix|nix|3=
environment.systemPackages = with pkgs; [
    python313
    lib.meta.lowPrio python314
];
}}
 
In this case you may run Python 3.13 via {{ic|python}} or {{ic|python3.13}} and Python 3.14 via {{ic|python3.14}}.


As of the time of this writing; these optimizations cause Python builds to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.
{{Note|If you wrapped Python as described in [[#Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix]], call {{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}} for the wrapped Python instead of calling them inside the wrapping.}}


== Performance ==
The derivation of CPython that is available via <code>nixpkgs</code> only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options]
Additionally, when compiling something within <code>nix-shell</code> or a derivation security hardening flags are passed to the compiler by default which may have a small performance impact.


At the time of writing certain optimizations cause Python wheels to be non-reproducible and increase install times. For a detailed overview of the trials and tribulations of discovering such performance regressions see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?].
=== Regression ===
=== Regression ===
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:  
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:  
Line 431: Line 685:
</syntaxhighlight>
</syntaxhighlight>


However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.
However, synthetic benchmarks are not necessarily reflective of real-world performance. In common real-world situations, the performance difference between optimized and non-optimized interpreters is minimal. For example, using <code>pylint</code> with a significant number of custom linters to scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference. Other workflows that were not performance sensitive saw no impact to their run times.
=== Possible Optimizations ===
If you run code that heavily depends on Python performance, and you desire the most performant Python interpreter possible, here are some possible things you can do:
 
* '''Enable the <code>enableOptimizations</code> flag for your Python derivation'''. See [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example]. Do note that this will cause you to compile Python the first time that you run it which will take a few minutes.
* '''Switch to a newer version of Python'''. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement, but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
* '''Disable hardening'''. Beware this only yields a small performance boost and it has impacts beyond Python code. See [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs].
 
'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.'''
 
== Troubleshooting ==
=== My module cannot be imported ===


If you are unable to do `import yourmodule` there are a number of reasons that could explain that.


=== Possible Optimizations ===
First, make sure that you installed/added your module to python. Typically you would use something like <code> (python3.withPackages (ps: with ps; [ yourmodule ]))</code> in the list of installed applications.
If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:


* Enable the <code>enableOptimizations</code> flag for your Python derivation. [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example] Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.
It is also still possible (e.g. when using nix-shell) that you aren't using the python interpreter you want because another package provides its own <code>python3.withPackages</code> in buildInputs, for example, yosys. In this case, you should either include that package (or all needed packages) in your withPackages list to only have a single Python interpreter. Or you can change the order of your packages, such that the <code>python3.withPackages</code> comes first, and becomes the Python interpreter that you get.
* Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
* Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs]


'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.'''
If you packaged yourself your application, make sure to use <code>buildPythonPackage</code> and **not** <code>buildPythonApplication</code> or <code>stdenv.mkDerivation</code>. The reason is that <code>python3.withPackages</code> [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L57 filters] the packages to check that they are built using the appropriate python interpreter: this is done by verifying that the derivation has a <code>pythonModule</code> attribute and only buildPythonPackage [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L43 sets this value] (passthru [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L75 here]) thanks to, notably <code>passthru = { pythonModule = python; }</code>. If you used <code>stdenv.mkDerivation</code> then you can maybe set this value manually, but it's safer to simply use <code>buildPythonPackage  {format = "other"; … your derivation …}</code> instead of <code>mkDerivation</code>.


== See also ==
== See also ==


* [[Packaging/Python]]
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
* [https://ryantm.github.io/nixpkgs/languages-frameworks/python/#python Python user guide in nixpkgs manual]
 
[[Category:Languages]]
[[Category:Languages]]
[[Category:Python]]

Latest revision as of 18:59, 12 August 2025

Python development environments with Nix

Nix supports a number of approaches to creating "development environments" for Python programming. These provide functionality analogous to virtualenv or conda: a shell environment with access to pinned versions of the python executable and Python packages.

Using the Nixpkgs Python infrastructure via shell.nix (recommended)

Nixpkgs has the few last Python versions packaged, as well as a consequent set of Python packages packaged that you can use to quickly create a Python environment.

Create a file shell.nix in the project directory, with the following template:

# shell.nix
let
  # We pin to a specific nixpkgs commit for reproducibility.
  # Last updated: 2024-04-29. Check for new commits at https://status.nixos.org.
  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/cf8cc1201be8bc71b7cbbbdaf349b22f4f99c7ae.tar.gz") {};
in pkgs.mkShell {
  packages = [
    (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [
      # select Python packages here
      pandas
      requests
    ]))
  ];
}

In this example, we create a Python environment with packages pandas and requests.

You can find Python packages that are available in Nixpkgs using search.nixos.org. For instance, type a Python package name like numpy in the search bar and click on the search button on the right. You can narrow down results by clicking on eg. "python311Packages" in the "Package sets" section on the left. Note that in the snippet above, on lines 8 and 9, each package is listed in the form python-pkgs.<name> where <name> corresponds to the one found in search.nixos.org . See Nix language basics for more information on the python-pkgs attribute set.

Once you have picked the Python packages you want, run nix-shell (or nix develop -f shell.nix) to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. python --version.

Note that with NixOS, this method can be used to install packages at the system level, e.g.

environment.systemPackages = with pkgs; [
  # ...
  (python3.withPackages (python-pkgs: with python-pkgs; [
      pandas
      requests
  ]))
];

Using Nix shell (new command line)

nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])'

If you don't use the channels any more, you can replace <nixpkgs> by an instance of the NixOS/nixpkgs repository using its absolute path.

Using a Python package not in Nixpkgs

Python packages in Nixpkgs are created and updated by Nixpkgs maintainers. Although the community invests a great effort to keep a complete and up-to-date package set, some packages you want may be missing, out of date, or broken. To use your own packages in a Nix environment, you may package it yourself.

The following is a high-level overview. For a complete explanation, see Developing with Python in the Nixpkgs Manual.

Generally, you may create a file that looks like this:

# toolz.nix
{
  lib,
  buildPythonPackage,
  fetchPypi,
  setuptools,
  wheel,
}:

buildPythonPackage rec {
  pname = "toolz";
  version = "0.10.0";

  src = fetchPypi {
    inherit pname version;
    hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
  };

  # do not run tests
  doCheck = false;

  # specific to buildPythonPackage, see its reference
  pyproject = true;
  build-system = [
    setuptools
    wheel
  ];
}

Given the file above is named toolz.nix and is the same directory as the previous shell.nix , you can edit shell.nix to use the package toolz above like so:

# shell.nix
let
  pkgs = import <nixpkgs> {};

  python = pkgs.python3.override {
    self = python;
    packageOverrides = pyfinal: pyprev: {
      toolz = pyfinal.callPackage ./toolz.nix { };
    };
  };

in pkgs.mkShell {
  packages = [
    (python.withPackages (python-pkgs: [
      # select Python packages here
      python-pkgs.pandas
      python-pkgs.requests
      python-pkgs.toolz
    ]))
  ];
}

Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.

Running Python packages which requires compilation and/or contains libraries precompiled without nix

If you want to use some Python packages containing libraries precompiled without nix as for example grpcio or Numpy, you may encounter the following error :

$ python -c 'import grpc'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/.../grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.

On NixOS, installing packages that need to compile code or use C libraries from outside of the nix package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding /nix/store/... to $LD_LIBRARY_PATH.

After doing one of the following, you should be able to install compiled libraries using venv, poetry, uv, conda or other packages managers.

Using nix overlay

{
  nixpkgs.overlays = [(
    self: super: rec {
      # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
      # > We currently take all libraries from systemd and nix as the default.
      pythonldlibpath = lib.makeLibraryPath (with super; [
        zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
      ]);
      # here we are overriding python program to add LD_LIBRARY_PATH to it's env
      python = super.stdenv.mkDerivation {
        name = "python";
        buildInputs = [ super.makeWrapper ];
        src = super.python311;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/python3 --set LD_LIBRARY_PATH ${pythonldlibpath}
          wrapProgram $out/bin/python3.11 --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
      poetry = super.stdenv.mkDerivation {
        name = "poetry";
        buildInputs = [ super.makeWrapper ];
        src = super.poetry;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
    }
  )];
  environment.systemPackages = with pkgs; [
    python  # here python will be taken from the overlay up here
    poetry
  ];
}

Using nix-ld

{
  programs.nix-ld = {
    enable = true;
    libraries = with pkgs; [
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    ];
  };
  # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
    (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
    '')
  ];
  # another (dangerous) solution
  # environment.systemPackages = with pkgs; [ python3 ];
  # programs.bash = {
  #   enable = true;
  #   initExtra = ''
  #     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #   '';
  # };
}

Using fix-python

Note: fix-python will patch binary files in the virtual environment without following symlinks.

Install with:

nix profile install github:GuillaumeDesforges/fix-python

Enter the venv and run:

fix-python --venv .venv

Using buildFHSEnv (Recommended)

#!/usr/bin/env nix-shell
{ pkgs ? import <nixpkgs> { } }:
(
  let base = pkgs.appimageTools.defaultFhsEnvArgs; in
  pkgs.buildFHSEnv (base // {
    name = "FHS";
    targetPkgs = pkgs: (with pkgs; [
      gcc glibc zlib
    ]);
    runScript = "zsh";
    extraOutputsToInstall = [ "dev" ];
  })
).env

Using a custom nix-shell

The following configuration automatically fix the dependencies:

let
  python = pkgs.python311;
  # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
  ]);
  patchedpython = (python.overrideAttrs (
    previousAttrs: {
      # Add the nix-ld libraries to the LD_LIBRARY_PATH.
      # creating a new library path from all desired libraries
      postInstall = previousAttrs.postInstall + ''
        mv  "$out/bin/python3.11" "$out/bin/unpatched_python3.11"
        cat << EOF >> "$out/bin/python3.11"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_python3.11" "\$@"
        EOF
        chmod +x "$out/bin/python3.11"
      '';
    }
  ));
  # if you want poetry
  patchedpoetry =  ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs (
    previousAttrs: {
      # same as above, but for poetry
      # not that if you dont keep the blank line bellow, it crashes :(
      postInstall = previousAttrs.postInstall + ''

        mv "$out/bin/poetry" "$out/bin/unpatched_poetry"
        cat << EOF >> "$out/bin/poetry"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_poetry" "\$@"
        EOF
        chmod +x "$out/bin/poetry"
      '';
    }
  ));
in
{
  environment.systemPackages = with pkgs; [
    patchedpython
    # if you want poetry
    patchedpoetry
  ];
}

This configuration set the LD_LIBRARY_PATH environment variable before running python using the overrideAttrs[1] function to override the postInstall script of cpython mkDerivation[2].

Prefix library paths using wrapProgram

wrapProgram is a part of the makeWrapper build input[3]. By combining it with the symlinkJoin, we can create a wrapper around the Python executable that will always set the required library paths. It’s worth noting that, for this solution to be compatible with Darwin, we need to use a different wrap prefix, as shown in the example below.

let
  # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
  ]);
  # Darwin requires a different library path prefix
  wrapPrefix = if (!pkgs.stdenv.isDarwin) then "LD_LIBRARY_PATH" else "DYLD_LIBRARY_PATH";
  patchedpython = (pkgs.symlinkJoin {
    name = "python";
    paths = [ pkgs.python312 ];
    buildInputs = [ pkgs.makeWrapper ];
    postBuild = ''
      wrapProgram "$out/bin/python3.12" --prefix ${wrapPrefix} : "${pythonldlibpath}"
    '';
  });
in
{
  environment.systemPackages = with pkgs; [
    patchedpython
  ];
}

Using venv

To create a Python virtual environment with venv:

$ nix-shell -p python3 --command "python -m venv .venv --copies"

You can then activate and use the Python virtual environment as usual and install dependencies with pip and similar.

Using uv

A single tool to replace pip, pip-tools, pipx, poetry, pyenv, virtualenv, and more.

uv is written in Rust and does not need Python as a prerequisite. Use the uv command to initialize Python projects, add Python packages, create or update virtual environments (in .venv folders), etc. as described in the uv docs. Use uv's tool interface to install and run Python packages that provide a CLI.

As a system package

environment.systemPackages = with pkgs; [
    uv
];

or as a home-manager package

home.packages = with pkgs; [
    uv
];

If you use uv it's recommended that you install the Python versions you need using the uv python install command, e.g.

uv python install 3.14 --preview --default

You may want to set the UV_PYTHON_DOWNLOADS=never environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting environment.localBinInPath = true; is highly recommended, because uv will install binaries in ~/.local/bin.

Using poetry

# shell.nix
let
  pkgs = import <nixpkgs> {};
in pkgs.mkShell {
  packages = with pkgs; [
    python310
    (poetry.override { python3 = python310; })
  ];
}

poetry2nix

poetry2nix uses the contents of a poetry.lock and pyproject.toml file to create Nix derivations. It has several functions for generating development environments and python projects. Because some older python projects rely on deprecated build systems (see edgecase.md for more info), poetry2nix provides overrides so these packages can still be built.

Using micromamba

Install the micromamba package to create environments and install packages as documented by micromamba.

To activate an environment you will need a FHS environment e.g.:

$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ python
>>> import numpy as np

Eventually you'll probably want to put this in a shell.nix so you won't have to type all that stuff every time e.g.:

{ pkgs ? import <nixpkgs> {}}:
let
  fhs = pkgs.buildFHSEnv {
    name = "my-fhs-environment";

    targetPkgs = _: [
      pkgs.micromamba
    ];

    profile = ''
      set -e
      eval "$(micromamba shell hook --shell=posix)"
      export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
      if ! test -d $MAMBA_ROOT_PREFIX/envs/my-mamba-environment; then
          micromamba create --yes -q -n my-mamba-environment
      fi
      micromamba activate my-mamba-environment
      micromamba install --yes -f conda-requirements.txt -c conda-forge
      set +e
    '';
  };
in fhs.env

Using conda

Install the package conda and run

$ conda-shell
$ conda-install
$ conda env update --file environment.yml

Imperative use

It is also possible to use conda-install directly. On first use, run:

$ conda-shell
$ conda-install

to set up conda in ~/.conda

Using pixi

Install the pixi package to create environments and install packages as documented by pixi:

$ pixi init
$ pixi add python

To activate an environment you will need a FHS environment e.g. a flake.nix [4]

{
  description = "pixi env";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs =
    { flake-utils, nixpkgs, ... }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs { inherit system; };
        fhs = pkgs.buildFHSEnv {
          name = "pixi-env";

          targetPkgs = _: [ pkgs.pixi ];
        };
      in
      {
        devShell = fhs.env;
      }
    );
}

Then using nix develop the environment can be activated:

$ nix develop
$ pixi s
$ python
>>> import numpy as np

Using Home Manager

Alternatively an FHS environment can wrap pixi using Home Manager e.g[5]

{ lib, pkgs, ... }:
{
  home.packages = [
    (pkgs.buildFHSEnv {
      name = "pixi";
      runScript = "pixi";
      targetPkgs = pkgs: with pkgs; [ pixi ];
    })
  ];
}

Then the environment can be activated:

$ pixi s
$ python
>>> import numpy as np

Package a Python application

With setup.py

To package a Python application that uses setup.py you can use buildPythonApplication. More details about this and similar functions can be found in the nixpkgs manual.

For example, we can package this simple flask server main.py:

#!/usr/bin/env python

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)

We also need a setup.py file, like this:

from setuptools import setup, find_packages

setup(name='myFlaskServer',
      version='1.0',
      # Modules to import from other scripts:
      packages=find_packages(),
      # Executables
      scripts=["main.py"],
     )

Then, we use the buildPythonApplication in the default.nix:

{ pkgs ? import <nixpkgs> {} }:

pkgs.python3Packages.buildPythonApplication {
  pname = "myFlaskApp";
  version = "0.1.0";

  propagatedBuildInputs = with pkgs.python3Packages; [
    flask
  ];

  src = ./.;
}

Finally, build your project using nix-build. The result will be executable in ./result/bin/app.py.

With pyproject.toml

When your project is using pyproject.tomlyou can use pyproject.nix to package your application.

First, a simple file structure could look like this:

├── app/
    └── main.py
├── flake.nix
├── pyproject.toml
└── README.md

To reuse the example from above, we use the same flask application:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)

Also, you need to define the pyproject.toml. Here, we only show some of the important parts. Please refer to pyproject.nix documentation for a full example.

[project]
name = "my-app"
version = "0.1.0"
description = "Simple app"

# define any Python dependencies
dependencies = [
  "flask>3",
]

# define the CLI executable
# Here, we define the entry point to be the 'main()' function in the module 'app/main.py'
[project.scripts]
cli = "app.main:main"

We package the application by calling the loadPyproject function from pyproject.nix. Again, we only show a minimal example. More information can be found in the documentation. Note that this example relies on flakes in contrast to some of the others on this page.

{
  description = "A basic flake using pyproject.toml project metadata";

  inputs = {
    pyproject-nix = {
      url = "github:nix-community/pyproject.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { nixpkgs, pyproject-nix, ... }:
    let
      inherit (nixpkgs) lib;

      project = pyproject-nix.lib.project.loadPyproject {
        # Read & unmarshal pyproject.toml relative to this project root.
        # projectRoot is also used to set `src` for renderers such as buildPythonPackage.
        projectRoot = ./.;
      };

      # This example is only using x86_64-linux
      pkgs = nixpkgs.legacyPackages.x86_64-linux;

      python = pkgs.python3;

    in
    {
      # Build our package using `buildPythonPackage
      packages.x86_64-linux.default =
        let
          # Returns an attribute set that can be passed to `buildPythonPackage`.
          attrs = project.renderers.buildPythonPackage { inherit python; };
        in
        # Pass attributes to buildPythonPackage.
        # Here is a good spot to add on any missing or custom attributes.
        python.pkgs.buildPythonPackage (attrs // {
          env.CUSTOM_ENVVAR = "hello";
        });
    };
}

To run the application, call nix run.

You can also launch an IDE under nix develop and get full dependency resolution. For example, the following command opens VS Code in the constructed environment:

$ nix develop --command code

Nixpkgs Python contribution guidelines

Libraries

According to the official guidelines for Python, new package expressions for libraries should be placed in pkgs/development/python-modules/<name>/default.nix.

Those expressions are then referenced from pkgs/top-level/python-packages.nix as in

  aenum = callPackage ../development/python-modules/aenum { };

Applications

Applications meant to be executed should be referenced directly from pkgs/top-level/all-packages.nix.

Other Python packages used in the Python package of the application should be taken from the callPackage argument pythonPackages , which guarantees that they belong to the same "pythonPackage" set. For example:

{
  lib,
  pythonPackages,
}:
buildPythonApplication {
  propagatedBuildInputs = [ pythonPackages.numpy ];
  # ...
}

Special Modules

GNOME

gobject-introspection based python modules need some environment variables to work correctly. For standalone applications, wrapGAppsHook (see the relevant documentation) wraps the executable with the necessary variables. But this is not fit for development. In this case use a nix-shell with gobject-introspection and all the libraries you are using (gtk and so on) as buildInputs. For example:

$ nix-shell -p gobjectIntrospection gtk3 'python2.withPackages (ps: with ps; [ pygobject3 ])' --run "python -c \"import pygtkcompat; pygtkcompat.enable_gtk(version='3.0')\""

Or, if you want to use matplotlib interactively:

$ nix-shell -p gobject-introspection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'
$ ipython
In [1]: import matplotlib
In [2]: matplotlib.use('gtk3agg')
In [3]: import matplotlib.pyplot as plt
In [4]: plt.ion()
In [5]: plt.plot([1,3,2,4])

You can also set backend : GTK3Agg in your ~/.config/matplotlib/matplotlibrc file to avoid having to call matplotlib.use('gtk3agg').

Debug Build

See python wiki on debug build.

In order to use a CPython interpreter built using --with-pydebug during configure phase, override any of the python packages passing enableDebug = true argument:

pythonDebug = pkgs.python310.override { enableDebug = true; };

Installing Multiple Versions

lib.meta.lowPrio and lib.meta.highPrio can be used to install multiple versions without conflicts (🚩︎#369997):

❄︎ /etc/nixos/configuration.nix
environment.systemPackages = with pkgs; [
    python313
    lib.meta.lowPrio python314
];

In this case you may run Python 3.13 via python or python3.13 and Python 3.14 via python3.14.

Note: If you wrapped Python as described in #Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix, call lib.meta.lowPrio and lib.meta.highPrio for the wrapped Python instead of calling them inside the wrapping.

Performance

The derivation of CPython that is available via nixpkgs only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See Configuring Python 3.1.3. Performance options Additionally, when compiling something within nix-shell or a derivation security hardening flags are passed to the compiler by default which may have a small performance impact.

At the time of writing certain optimizations cause Python wheels to be non-reproducible and increase install times. For a detailed overview of the trials and tribulations of discovering such performance regressions see Why is the nix-compiled Python slower?.

Regression

With the nixpkgs version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:

## Ubuntu's Python 3.8
username:dir$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[7.831622750498354, 7.82998560462147, 7.830805554986, 7.823807033710182, 7.84282516874373]

## nix-shell's Python 3.8
[nix-shell:~/src]$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[10.431915327906609, 10.435049421153963, 10.449542525224388, 10.440207410603762, 10.431304694153368]

However, synthetic benchmarks are not necessarily reflective of real-world performance. In common real-world situations, the performance difference between optimized and non-optimized interpreters is minimal. For example, using pylint with a significant number of custom linters to scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference. Other workflows that were not performance sensitive saw no impact to their run times.

Possible Optimizations

If you run code that heavily depends on Python performance, and you desire the most performant Python interpreter possible, here are some possible things you can do:

  • Enable the enableOptimizations flag for your Python derivation. See Example. Do note that this will cause you to compile Python the first time that you run it which will take a few minutes.
  • Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement, but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
  • Disable hardening. Beware this only yields a small performance boost and it has impacts beyond Python code. See Hardening in Nixpkgs.

Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.

Troubleshooting

My module cannot be imported

If you are unable to do `import yourmodule` there are a number of reasons that could explain that.

First, make sure that you installed/added your module to python. Typically you would use something like (python3.withPackages (ps: with ps; [ yourmodule ])) in the list of installed applications.

It is also still possible (e.g. when using nix-shell) that you aren't using the python interpreter you want because another package provides its own python3.withPackages in buildInputs, for example, yosys. In this case, you should either include that package (or all needed packages) in your withPackages list to only have a single Python interpreter. Or you can change the order of your packages, such that the python3.withPackages comes first, and becomes the Python interpreter that you get.

If you packaged yourself your application, make sure to use buildPythonPackage and **not** buildPythonApplication or stdenv.mkDerivation. The reason is that python3.withPackages filters the packages to check that they are built using the appropriate python interpreter: this is done by verifying that the derivation has a pythonModule attribute and only buildPythonPackage sets this value (passthru here) thanks to, notably passthru = { pythonModule = python; }. If you used stdenv.mkDerivation then you can maybe set this value manually, but it's safer to simply use buildPythonPackage {format = "other"; … your derivation …} instead of mkDerivation.

See also