Python

Python environments with Nix

Python environment allow you to run Python code with the python command. It may also have Python packages installed as well.

Nix allows to create many independent Python environment, much like venv.

There are many ways to create a Python environment with Nix.

Using regular Python virtual environment

It is possible to use Nix to create a Python virtual environment for your project. You can then use the Python virtual environment as usual.

To create a Python virtual environment:

$ nix-shell -p python3 --command "python -m venv .venv --copies"

You can then activate and use the Python virtual environment as usual and install dependencies with pip and similar.

On NixOS

This method may not work on NixOS, as installing packages with pip that need to compile code or use C libraries will fail due to not finding dependencies in the expected places.

There are multiple ways to make it work:

• Use fix-python, this is most suited for beginners.

• Create a FHS user env with buildFHSUserEnv.
• Setup nix-ld in your NixOS configuration.

Using the Nixpkgs Python infrastructure

Nixpkgs has the few last Python versions packaged, as well as a consequent set of Python packages packaged that you can use to quickly create a Python environment.

Create a file shell.nix in the project directory, with the following template:

# shell.nix
let
  pkgs = import <nixpkgs> {};
in pkgs.mkShell {
  packages = [
    (pkgs.python3.withPackages (python-pkgs: [
      # select Python packages here
      python-pkgs.pandas
      python-pkgs.requests
    ]))
  ];
}

This example will make Nix create a Python environment with Python packages "pandas" and "requests". You can remove them, or add more.

You can find Python packages that are available in Nixpkgs using search.nixos.org. For instance, type a Python package name (for example numpy) in the search bar and click on the search button on the right. Then on the left in the "Package sets" section click on "python311Packages" to list only Python packages. for Python (version 3.11).

Note that in the snippet above, on line 8 and 9, each package to install is in the form python-pkgs.<name>. The package name must correspond to the one found in search.nixos.org as explained above. To understand why it is prefixed by python-pkgs, read the Nix language basics.

Then, once you have picked the Python packages you want, run nix-shell (or nix develop -f shell.nix) to make Nix create the Python environment and enter it. If it succeeds, Python should be available in your PATH, try using python --version at this point.

As a word of warning, using pkgs = import <nixpkgs> {}; as in the snippet above (line 3) will result in a non-reproducible environment. Use Nixpkgs pinning or Nix Flakes to make your environment reproducible.

Using a Python packages not in Nixpkgs

Python packages in Nixpkgs are created and updated by Nixpkgs maintainers. Although the community invests a great effort to keep a complete and up-to-date package set, it isn't uncommon for packages you want to be missing, out of date or broken. To use your own packages in a Nix environment, you may package it yourself.

The following is a high-level overview. For a complete explanation, see Developing with Python in the Nixpkgs Manual.

Generally, you may create a file that looks like this:

# toolz.nix
{ lib
, buildPythonPackage
, fetchPypi
, setuptools
, wheel
}:

buildPythonPackage rec {
  pname = "toolz";
  version = "0.10.0";

  src = fetchPypi {
    inherit pname version;
    hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
  };

  # do not run tests
  doCheck = false;

  # specific to buildPythonPackage, see its reference
  pyproject = true;
  build-system = [
    setuptools
    wheel
  ];
}

The tool pip2nix can help you generate such files. Given the file above is named toolz.nix and is the same directory as the previous shell.nix , you can edit shell.nix to use the package toolz above like so:

# shell.nix
let
  pkgs = import <nixpkgs> {};
in pkgs.mkShell {
  packages = [
    (pkgs.python3.withPackages (python-pkgs: [
      # select Python packages here
      python-pkgs.pandas
      python-pkgs.requests
      (pkgs.callPackage ./toolz.nix)
    ]))
  ];
}

Note that the parenthesis (line 10) are required.

Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.

Using micromamba

Install the micromamba package to create environments and install packages as documented by micromamba.

To activate an environment you will need a FHS environment e.g.:

$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ python
>>> import numpy as np

Eventually you'll probably want to put this in a shell.nix so you won't have to type all that stuff every time e.g.:

{ pkgs ? import <nixpkgs> {}}:
let
  fhs = pkgs.buildFHSUserEnv {
    name = "my-fhs-environment";

    targetPkgs = _: [
      pkgs.micromamba
    ];

    profile = ''
      set -e
      eval "$(micromamba shell hook --shell=posix)"
      export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
      micromamba create -q -n my-mamba-environment
      micromamba activate my-mamba-environment
      micromamba install --yes -f conda-requirements.txt -c conda-forge
      set +e
    '';
  };
in fhs.env

Using conda

Install the package conda and run

$ conda-shell
$ conda-install
$ conda env update --file environment.yml

Imperative use

It is also possible to use conda-install directly. On first use, run:

$ conda-shell
$ conda-install

to set up conda in ~/.conda

Package a Python application

It is possible to use buildPythonApplication to package python applications. As explained in the nixpkgs manual, it uses the widely used `setup.py` file in order to package properly the application. We now show how to package a simple python application: a basic flask web server.

First, we write the python code, say in a file web_interface.py. Here we create a basic flask web server;

#!/usr/bin/env python

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)

Then, we create the setup.py file, which basically explains which are the executables:

#!/usr/bin/env python

from setuptools import setup, find_packages

setup(name='demo-flask-vuejs-rest',
      version='1.0',
      # Modules to import from other scripts:
      packages=find_packages(),
      # Executables
      scripts=["web_interface.py"],
     )

Finally, our nix derivation is now trivial: the file derivation.nix just needs to provide the python packages (here flask):

{ lib, python3Packages }:
with python3Packages;
buildPythonApplication {
  pname = "demo-flask-vuejs-rest";
  version = "1.0";

  propagatedBuildInputs = [ flask ];

  src = ./.;
}

and we can now load this derivation from our file default.nix:

{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}

We can now build with:

$ nix-build
[...]
$ ./result/bin/web_interface.py 
 * Serving Flask app ".web_interface" (lazy loading)
 [...]

or just enter a nix-shell, and directly execute your program or python if it's easier to develop:

$ nix-shell
[...]
[nix-shell]$ chmod +x web_interface.py
[nix-shell]$ ./web_interface.py 
 * Serving Flask app "web_interface" (lazy loading)
[...]

[nix-shell]$ python
Python 3.8.7 (default, Dec 21 2020, 17:18:55) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>

Nixpkgs Python contribution guidelines

Libraries

According to the official guidelines for Python, new package expressions for libraries should be placed in pkgs/development/python-modules/<name>/default.nix.

Those expressions are then referenced from pkgs/top-level/python-packages.nix like in this example:

  aenum = callPackage ../development/python-modules/aenum { };

Applications

Applications meant to be executed should be referenced directly from pkgs/top-level/all-packages.nix.

Other Python packages used in the Python package of the application should be taken from the callPackage argument pythonPackages , which guarantees that they belong to the same "pythonPackage" set. For example:

{ lib
, pythonPackages
}:

buildPythonApplication rec {
  # ...
  propagatedBuildInputs = [
    pythonPackages.numpy
  ];
  # ...
}

Special Modules

GNOME

gobject-introspection based python modules need some environment variables to work correctly. For standalone applications, wrapGAppsHook (see the relevant documentation) wraps the executable with the necessary variables. But this is not fit for development. In this case use a nix-shell with gobject-introspection and all the libraries you are using (gtk and so on) as buildInputs. For example:

$ nix-shell -p gobjectIntrospection gtk3 'python2.withPackages (ps: with ps; [ pygobject3 ])' --run "python -c \"import pygtkcompat; pygtkcompat.enable_gtk(version='3.0')\""

Or, if you want to use matplotlib interactively:

$ nix-shell -p gobject-introspection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'
$ ipython

In [1]: import matplotlib
In [2]: matplotlib.use('gtk3agg')
In [3]: import matplotlib.pyplot as plt
In [4]: plt.ion()
In [5]: plt.plot([1,3,2,4])

You can also set backend : GTK3Agg in your ~/.config/matplotlib/matplotlibrc file to avoid having to call matplotlib.use('gtk3agg').

Performance

The derivation of CPython that is available via nixpkgs only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See Configuring Python 3.1.3. Performance options Additionally, when you compile something within nix-shell or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.

As of the time of this writing; these optimizations cause Python wheels to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see Why is the nix-compiled Python slower? thread on the nix forums.

Regression

With the nixpkgs version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:

## Ubuntu's Python 3.8
username:dir$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[7.831622750498354, 7.82998560462147, 7.830805554986, 7.823807033710182, 7.84282516874373]

## nix-shell's Python 3.8
[nix-shell:~/src]$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[10.431915327906609, 10.435049421153963, 10.449542525224388, 10.440207410603762, 10.431304694153368]

However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using pylint with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.

Possible Optimizations

If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:

• Enable the enableOptimizations flag for your Python derivation. Example Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.
• Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
• Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. Hardening in Nixpkgs

Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.

Troubleshooting

My module cannot be imported

If you are unable to do `import yourmodule` there are a number of reasons that could explain that.

First, make sure that you installed/added your module to python. Typically you would use something like (python3.withPackages (ps: with ps; [ yourmodule ])) in the list of installed applications.

It is also still possible (e.g. when using nix-shell) that you aren't using the python interpreter you want because another package provides its own python3.withPackages in buildInputs, for example, yosys. In this case, you should either include that package (or all needed packages) in your withPackages list to only have a single Python interpreter. Or you can change the order of your packages, such that the python3.withPackages comes first, and becomes the Python interpreter that you get.

If you packaged yourself your application, make sure to use buildPythonPackage and **not** buildPythonApplication or stdenv.mkDerivation. The reason is that python3.withPackages filters the packages to check that they are built using the appropriate python interpreter: this is done by verifying that the derivation has a pythonModule attribute and only buildPythonPackage sets this value (passthru here) thanks to, notably passthru = { pythonModule = python; }. If you used stdenv.mkDerivation then you can maybe set this value manually, but it's safer to simply use buildPythonPackage {format = "other"; … your derivation …} instead of mkDerivation.

See also

• "Python" in Nixpkgs Manual