View source for Python

== Python development environments with Nix ==
Nix supports a number of approaches to creating "development environments" for Python programming. These provide functionality analogous to [https://virtualenv.pypa.io/en/latest/ virtualenv] or [https://docs.conda.io/en/latest/ conda]: a shell environment with access to pinned versions of the <code>python</code> executable and Python packages.

=== Using the Nixpkgs Python infrastructure via <code>shell.nix</code> (recommended) ===
Nixpkgs has the few last Python versions packaged, as well as a consequent set of Python packages packaged that you can use to quickly create a Python environment.

Create a file <code>shell.nix</code> in the project directory, with the following template:

<syntaxhighlight lang="nix" line="1">
# shell.nix
let
  # We pin to a specific nixpkgs commit for reproducibility.
  # Last updated: 2024-04-29. Check for new commits at https://status.nixos.org.
  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/cf8cc1201be8bc71b7cbbbdaf349b22f4f99c7ae.tar.gz") {};
in pkgs.mkShell {
  packages = [
    (pkgs.python3.withPackages (python-pkgs: [
      # select Python packages here
      python-pkgs.pandas
      python-pkgs.requests
    ]))
  ];
}
</syntaxhighlight>

In this example, we create a Python environment with packages <code>pandas</code> and <code>requests</code>.

You can find Python packages that are available in Nixpkgs using [https://search.nixos.org/packages search.nixos.org]. For instance, type a Python package name like <code>numpy</code> in the search bar and click on the search button on the right. You can narrow down results by clicking on eg. "python311Packages" in the "Package sets" section on the left. Note that in the snippet above, on lines 8 and 9, each package is listed in the form <code>python-pkgs.<name></code> where <code><name></code> corresponds to the one found in [https://search.nixos.org/packages search.nixos.org] . See [https://nix.dev/tutorials/nix-language.html Nix language basics] for more information on the <code>python-pkgs</code> attribute set.

Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.

==== Using a Python package not in Nixpkgs ====
Python packages in Nixpkgs are created and updated by Nixpkgs maintainers. Although the community invests a great effort to keep a complete and up-to-date package set, some packages you want may be missing, out of date, or broken. To use your own packages in a Nix environment, you may package it yourself.

The following is a high-level overview. For a complete explanation, see [https://nixos.org/manual/nixpkgs/unstable/#developing-with-python Developing with Python] in the Nixpkgs Manual.

Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1">
# toolz.nix
{ lib
, buildPythonPackage
, fetchPypi
, setuptools
, wheel
}:

buildPythonPackage rec {
  pname = "toolz";
  version = "0.10.0";

  src = fetchPypi {
    inherit pname version;
    hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
  };

  # do not run tests
  doCheck = false;

  # specific to buildPythonPackage, see its reference
  pyproject = true;
  build-system = [
    setuptools
    wheel
  ];
}
</syntaxhighlight>The tool [https://github.com/nix-community/pip2nix pip2nix] can help you generate such files. 

Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
# shell.nix
let
  pkgs = import <nixpkgs> {};
in pkgs.mkShell {
  packages = [
    (pkgs.python3.withPackages (python-pkgs: [
      # select Python packages here
      python-pkgs.pandas
      python-pkgs.requests
      (pkgs.callPackage ./toolz.nix)
    ]))
  ];
}
</syntaxhighlight>Note that the parenthesis (line 10) are required.

Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
==== Using nix-shell alongside pip  ====
When working on a collaborative python project you may want to be able to <code>pip install -r requirements.txt</code> if the project isn't packaged for nix specifically.

The problem is that a lot of python packages won't work out of the box when you pip install them.
To fix this issue, you can create a nix shell that will use <code>pip</code> for the packages that are installing properly, but will fix the environment for the packages that are causing issues.

You can accomplish this by adding these two files to the root of your project:
<syntaxhighlight lang="nix" line="1">
# shell.nix
{ pkgs ? import <nixpkgs> {} }:

let
  # Choose your python version
  myPython = pkgs.python3;
  pythonPackages = pkgs.python3Packages;

  pythonWithPkgs = myPython.withPackages (pythonPkgs: with pythonPkgs; [
    # This list contains tools for Python development.
    # You can also add other tools, like black.
    #
    # Note that even if you add Python packages here like PyTorch or Tensorflow,
    # they will be reinstalled when running `pip -r requirements.txt` because
    # virtualenv is used below in the shellHook.
    ipython
    pip
    setuptools
    virtualenvwrapper
    wheel
    black
  ]);

  extraBuildInputs = with pythonPackages; [
    # this list contains packages that you want to be available at runtime and might not be able to be installed properly via pip
    # pandas
    # requests
  ] ++ (with pkgs; [
    #
  ]);
in
import ./python-shell.nix {
  extraBuildInputs = extraBuildInputs;
  # extraLibPackages = extraLibPackages;
  myPython = myPython;
  pythonWithPkgs = pythonWithPkgs;
  pkgs = pkgs;
}
</syntaxhighlight>

<syntaxhighlight lang="nix" line="1">
# python-shell.nix
{ pkgs ? import <nixpkgs> {}, extraBuildInputs ? [], myPython ? pkgs.python3, extraLibPackages ? [], pythonWithPkgs? myPython }:


let

    buildInputs  = with pkgs; [
        clang
        llvmPackages_16.bintools
        rustup
    ] ++ extraBuildInputs;

  lib-path = with pkgs; lib.makeLibraryPath buildInputs;


shell = pkgs.mkShell {
  buildInputs = [
     # my python and packages
      pythonWithPkgs
      
      # other packages needed for compiling python libs
      pkgs.readline
      pkgs.libffi
      pkgs.openssl

      # unfortunately needed because of messing with LD_LIBRARY_PATH below
      pkgs.git
      pkgs.openssh
      pkgs.rsync
  ] ++ extraBuildInputs;
  shellHook = ''
      # Allow the use of wheels.
      SOURCE_DATE_EPOCH=$(date +%s)
      # Augment the dynamic linker path
      export "LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${lib-path}"
      # Setup the virtual environment if it doesn't already exist.
      VENV=.venv
      if test ! -d $VENV; then
        virtualenv $VENV
      fi
      source ./$VENV/bin/activate
      export PYTHONPATH=$PYTHONPATH:`pwd`/$VENV/${myPython.sitePackages}/
  '';
};

in
shell
</syntaxhighlight>

=== Running compiled libraries ===
If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/.../grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.

On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations.
There are multiple ways to make it work:
* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.

* Create a FHS user env with <code>buildFHSUserEnv</code>.
* Setup <code>nix-ld</code><ref name=":0">https://github.com/Mic92/nix-ld</ref> in your NixOS configuration.

==== Setup nix-ld ====
nix-ld<ref name=":0" /> allow you to <code>Run unpatched dynamic binaries on NixOS.</code>

The following configuration automatically fix the dependencies :<syntaxhighlight lang="nixos" line="1">
let
  python = pkgs.python311;
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  # > We currently take all libraries from systemd and nix as the default.
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib
    zstd
    stdenv.cc.cc
    curl
    openssl
    attr
    libssh
    bzip2
    libxml2
    acl
    libsodium
    util-linux
    xz
    systemd
  ]);
  patchedpython = (python.overrideAttrs (
    previousAttrs: {
      # Add the nix-ld libraries to the LD_LIBRARY_PATH.
      # creating a new library path from all desired libraries
      postInstall = previousAttrs.postInstall + ''
        mv  "$out/bin/python3.11" "$out/bin/unpatched_python3.11"
        cat << EOF >> "$out/bin/python3.11"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_python3.11" "\$@"
        EOF
        chmod +x "$out/bin/python3.11"
      '';
    }
  ));
  # if you want poetry
  patchedpoetry =  ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs (
    previousAttrs: {
      # same as above, but for poetry
      # not that if you dont keep the blank line bellow, it crashes :(
      postInstall = previousAttrs.postInstall + ''

        mv "$out/bin/poetry" "$out/bin/unpatched_poetry"
        cat << EOF >> "$out/bin/poetry"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_poetry" "\$@"
        EOF
        chmod +x "$out/bin/poetry"
      '';
    }
  ));
in
{
  # Some other config...
  
  environment.systemPackages = with pkgs; [
    patchedpython

    # if you want poetry
    patchedpoetry
  ];
}
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.

After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...

=== Using <code>venv</code> ===

To create a Python virtual environment with <code>venv</code>:<syntaxhighlight lang="console">
$ nix-shell -p python3 --command "python -m venv .venv --copies"
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.

=== Using poetry ===
<syntaxhighlight lang="nix">

# shell.nix
let
  pkgs = import <nixpkgs> {};
in pkgs.mkShell {
  packages = with pkgs; [
    python310
    (poetry.override { python3 = python310; })
  ];
}
</syntaxhighlight>

=== Using micromamba ===

Install the <code>micromamba</code> package to create environments and install packages as [https://github.com/mamba-org/mamba#micromamba documented by micromamba].

To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ python
>>> import numpy as np
</syntaxhighlight>

Eventually you'll probably want to put this in a shell.nix so you won't have to type all that stuff every time e.g.:
<syntaxhighlight lang="shell">
{ pkgs ? import <nixpkgs> {}}:
let
  fhs = pkgs.buildFHSUserEnv {
    name = "my-fhs-environment";

    targetPkgs = _: [
      pkgs.micromamba
    ];

    profile = ''
      set -e
      eval "$(micromamba shell hook --shell=posix)"
      export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
      if ! test -d $MAMBA_ROOT_PREFIX/envs/my-mamba-environment; then
          micromamba create --yes -q -n my-mamba-environment
      fi
      micromamba activate my-mamba-environment
      micromamba install --yes -f conda-requirements.txt -c conda-forge
      set +e
    '';
  };
in fhs.env
</syntaxhighlight>
=== Using conda ===

Install the package <code>conda</code> and run 
<syntaxhighlight lang="console">
$ conda-shell
$ conda-install
$ conda env update --file environment.yml
</syntaxhighlight>

==== Imperative use ====

It is also possible to use <code>conda-install</code> directly. On first use, run:
<syntaxhighlight lang="console">
$ conda-shell
$ conda-install
</syntaxhighlight>
to set up conda in <code>~/.conda</code>

== Package a Python application ==
It is possible to use <code>buildPythonApplication</code> to package python applications. As explained in the nixpkgs manual, it uses the widely used <code>setup.py</code> file in order to package properly the application. We now show how to package a simple python application: a basic flask web server.

First, we write the python code, say in a file <code>web_interface.py</code>. Here we create a basic flask web server:
<syntaxhighlight lang="python">
#!/usr/bin/env python

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>

Then, we create the <code>setup.py</code> file, which basically explains which are the executables:
<syntaxhighlight lang="python">
#!/usr/bin/env python

from setuptools import setup, find_packages

setup(name='demo-flask-vuejs-rest',
      version='1.0',
      # Modules to import from other scripts:
      packages=find_packages(),
      # Executables
      scripts=["web_interface.py"],
     )
</syntaxhighlight>

Finally, our nix derivation is now trivial: the file <code>derivation.nix</code> just needs to provide the python packages (here flask):
<syntaxhighlight lang="nix">
{ lib, python3Packages }:
with python3Packages;
buildPythonApplication {
  pname = "demo-flask-vuejs-rest";
  version = "1.0";

  propagatedBuildInputs = [ flask ];

  src = ./.;
}
</syntaxhighlight>

and we can now load this derivation from our file <code>default.nix</code>:
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}
</syntaxhighlight>

We can now build with:
<syntaxhighlight lang=console>
$ nix-build
[...]
$ ./result/bin/web_interface.py 
 * Serving Flask app ".web_interface" (lazy loading)
 [...]
</syntaxhighlight>
or just enter a nix-shell, and directly execute your program or python if it's easier to develop:
<syntaxhighlight lang=console>
$ nix-shell
[...]
[nix-shell]$ chmod +x web_interface.py
[nix-shell]$ ./web_interface.py 
 * Serving Flask app "web_interface" (lazy loading)
[...]

[nix-shell]$ python
Python 3.8.7 (default, Dec 21 2020, 17:18:55) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>
</syntaxhighlight>

== Nixpkgs Python contribution guidelines ==

=== Libraries ===

According to the [https://nixos.org/nixpkgs/manual/#contributing-guidelines official guidelines] for Python, new package expressions for libraries should be placed in <syntaxhighlight lang="bash" inline>pkgs/development/python-modules/<name>/default.nix</syntaxhighlight>.

Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> as in

<syntaxhighlight lang="nix">
  aenum = callPackage ../development/python-modules/aenum { };
</syntaxhighlight>

=== Applications ===

Applications meant to be executed should be referenced directly from <code>pkgs/top-level/all-packages.nix</code>.

Other Python packages used in the Python package of the application should be taken from the <code>callPackage</code> argument <code>pythonPackages</code> , which guarantees that they belong to the same "pythonPackage" set. For example:

<syntaxhighlight lang="nix" line="1">
{ lib
, pythonPackages
}:
buildPythonApplication {
  propagatedBuildInputs = [ pythonPackages.numpy ];
  # ...
}
</syntaxhighlight>
== Special Modules ==
=== GNOME ===
<code>gobject-introspection</code> based python modules need some environment variables to work correctly. For standalone
applications, <code>wrapGAppsHook</code> (see the [https://nixos.org/nixpkgs/manual/#sec-language-gnome relevant documentation]) wraps the executable with the necessary variables. But this is not fit for development.
In this case use a <code>nix-shell</code> with <code>gobject-introspection</code> and all the libraries you are using (gtk and so on) as <code>buildInputs</code>.
For example:
{{Commands|<nowiki>$ nix-shell -p gobjectIntrospection gtk3 'python2.withPackages (ps: with ps; [ pygobject3 ])' --run "python -c \"import pygtkcompat; pygtkcompat.enable_gtk(version='3.0')\""</nowiki>}}

Or, if you want to use matplotlib interactively:
<syntaxhighlight lang=console>
$ nix-shell -p gobject-introspection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'
$ ipython
</syntaxhighlight>
<syntaxhighlight lang=python>
In [1]: import matplotlib
In [2]: matplotlib.use('gtk3agg')
In [3]: import matplotlib.pyplot as plt
In [4]: plt.ion()
In [5]: plt.plot([1,3,2,4])
</syntaxhighlight>

You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.

== Performance ==
The derivation of CPython that is available via <code>nixpkgs</code> only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options] 
Additionally, when compiling something within <code>nix-shell</code> or a derivation security hardening flags are passed to the compiler by default which may have a small performance impact.

At the time of writing certain optimizations cause Python wheels to be non-reproducible and increase install times. For a detailed overview of the trials and tribulations of discovering such performance regressions see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?].
=== Regression ===
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example: 
<syntaxhighlight lang=python>## Ubuntu's Python 3.8
username:dir$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[7.831622750498354, 7.82998560462147, 7.830805554986, 7.823807033710182, 7.84282516874373]

## nix-shell's Python 3.8
[nix-shell:~/src]$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[10.431915327906609, 10.435049421153963, 10.449542525224388, 10.440207410603762, 10.431304694153368]
</syntaxhighlight>

However, synthetic benchmarks are not necessarily reflective of real-world performance. In common real-world situations, the performance difference between optimized and non-optimized interpreters is minimal. For example, using <code>pylint</code> with a significant number of custom linters to scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference. Other workflows that were not performance sensitive saw no impact to their run times.
=== Possible Optimizations ===
If you run code that heavily depends on Python performance, and you desire the most performant Python interpreter possible, here are some possible things you can do:

* '''Enable the <code>enableOptimizations</code> flag for your Python derivation'''. See [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example]. Do note that this will cause you to compile Python the first time that you run it which will take a few minutes.
* '''Switch to a newer version of Python'''. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement, but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
* '''Disable hardening'''. Beware this only yields a small performance boost and it has impacts beyond Python code. See [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs].

'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.'''

== Troubleshooting ==
=== My module cannot be imported ===

If you are unable to do `import yourmodule` there are a number of reasons that could explain that.

First, make sure that you installed/added your module to python. Typically you would use something like <code> (python3.withPackages (ps: with ps; [ yourmodule ]))</code> in the list of installed applications.

It is also still possible (e.g. when using nix-shell) that you aren't using the python interpreter you want because another package provides its own <code>python3.withPackages</code> in buildInputs, for example, yosys. In this case, you should either include that package (or all needed packages) in your withPackages list to only have a single Python interpreter. Or you can change the order of your packages, such that the <code>python3.withPackages</code> comes first, and becomes the Python interpreter that you get.

If you packaged yourself your application, make sure to use <code>buildPythonPackage</code> and **not** <code>buildPythonApplication</code> or <code>stdenv.mkDerivation</code>. The reason is that <code>python3.withPackages</code> [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L57 filters] the packages to check that they are built using the appropriate python interpreter: this is done by verifying that the derivation has a <code>pythonModule</code> attribute and only buildPythonPackage [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L43 sets this value] (passthru [https://github.com/NixOS/nixpkgs/blob/91d1eb9f2a9c4e3c9d68a59f6c0cada8c63d5340/pkgs/top-level/python-packages.nix#L75 here]) thanks to, notably <code>passthru = { pythonModule = python; }</code>. If you used <code>stdenv.mkDerivation</code> then you can maybe set this value manually, but it's safer to simply use <code>buildPythonPackage  {format = "other"; … your derivation …}</code> instead of <code>mkDerivation</code>.

== See also ==

* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
[[Category:Languages]]