NixOS Wiki

Python: Difference between revisions

← Older editNewer edit →
VisualWikitext
Gdforj (talk | contribs)
3 edits
Start improving the Nix+Python wiki guide.
Gdforj (talk | contribs)
3 edits
Overhaul the Python page to make it more up to date and more accessible to Python users who have never used Nix.
Line 1: Line 1:


== Use a Python environment with Nix ==
== Python environments with Nix ==
Python environment allow you to run Python code with the <code>python</code> command. It may also have Python packages installed as well.
Python environment allow you to run Python code with the <code>python</code> command. It may also have Python packages installed as well.


Line 6: Line 6:


There are many ways to create a Python environment with Nix.
There are many ways to create a Python environment with Nix.
=== Using regular Python virtual environment ===
It is possible to use Nix to create a Python virtual environment for your project. You can then use the Python virtual environment as usual.
To create a Python virtual environment:<syntaxhighlight lang="console">
$ nix-shell -p python3 --command "python -m venv .venv --copies"
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
==== On NixOS ====
This method may not work on NixOS, as installing packages with <code>pip</code> that need to compile code or use C libraries will fail due to not finding dependencies in the expected places.
There are multiple ways to make it work:
* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
* Create a FHS user env with <code>buildFHSUserEnv</code>.
* Setup <code>nix-ld</code> in your NixOS configuration.


=== Using the Nixpkgs Python infrastructure ===
=== Using the Nixpkgs Python infrastructure ===
Line 70: Line 88:
   ];
   ];
}
}
</syntaxhighlight>Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
</syntaxhighlight>The tool [https://github.com/nix-community/pip2nix pip2nix] can help you generate such files.
 
Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
# shell.nix
# shell.nix
let
let
Line 87: Line 107:


Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
=== Using micromamba ===
Install the <code>micromamba</code> package to create environments and install packages as [https://github.com/mamba-org/mamba#micromamba documented by micromamba].
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ python
>>> import numpy as np
</syntaxhighlight>
Eventually you'll probably want to put this in a shell.nix so you won't have to type all that stuff every time e.g.:
<syntaxhighlight lang="shell">
{ pkgs ? import <nixpkgs> {}}:
let
  fhs = pkgs.buildFHSUserEnv {
    name = "my-fhs-environment";
    targetPkgs = _: [
      pkgs.micromamba
    ];
    profile = ''
      set -e
      eval "$(micromamba shell hook --shell=posix)"
      export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
      micromamba create -q -n my-mamba-environment
      micromamba activate my-mamba-environment
      micromamba install --yes -f conda-requirements.txt -c conda-forge
      set +e
    '';
  };
in fhs.env
</syntaxhighlight>
=== Using conda ===
Install the package <code>conda</code> and run
<syntaxhighlight lang="console">
$ conda-shell
$ conda-install
$ conda env update --file environment.yml
</syntaxhighlight>
==== Imperative use ====
It is also possible to use <code>conda-install</code> directly. On first use, run:
<syntaxhighlight lang="console">
$ conda-shell
$ conda-install
</syntaxhighlight>
to set up conda in <code>~/.conda</code>


== Package a Python application ==
== Package a Python application ==
Line 166: Line 240:
</syntaxhighlight>
</syntaxhighlight>


=== Python virtual environment ===
== Nixpkgs Python contribution guidelines ==


Starting from Python 3 virtual environment is natively supported.
=== Libraries ===
The Python 3 venv approach has the benefit of forcing you to choose a specific version of the Python 3 interpreter that should be used to create the virtual environment. This avoids any confusion as to which Python installation the new environment is based on.
 
Recommended usage:
* Python 3.3-3.4 (old): the recommended way to create a virtual environment was to use the pyvenv command-line tool that also comes included with your Python 3 installation by default.
* Python 3.6+: <code>python3 -m venv</code> is the way to go.
 
Put your packages in a requirements.txt:
<syntaxhighlight lang=text>
pandas
requests
</syntaxhighlight>
 
Then setup the virtualenv:
<syntaxhighlight lang="shell">
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
</syntaxhighlight>
 
Installing packages with <code>pip</code> that need to compile code or use C libraries will sometimes fail due to not finding dependencies in the expected places. In that case you can use <code>buildFHSUserEnv</code> to make yourself a sandbox that appears like a more typical Linux install (or you can also certainly use <code>nix-ld</code> to turn your whole system into a more standard Linux distribution). For example if you were working with machine learning code you could use:
 
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
(pkgs.buildFHSUserEnv {
  name = "pipzone";
  targetPkgs = pkgs: (with pkgs; [
    python39
    python39Packages.pip
    python39Packages.virtualenv
    cudaPackages.cudatoolkit
  ]);
  runScript = "bash";
}).env
</syntaxhighlight>
 
In <code>pip-shell.nix</code>, and enter the environment with:
 
<syntaxhighlight lang="shell">
nix-shell pip-shell.nix
virtualenv venv
source venv/bin/activate
</syntaxhighlight>
 
==== Virtualenv without nix on NixOS ====
Another option is to follow the [https://gist.github.com/GuillaumeDesforges/7d66cf0f63038724acf06f17331c9280 fix-python gist] to setup a virtualenv without explicitly entering a nix shell.
 
=== Emulating virtualenv with nix-shell ===
In some cases virtualenv fails to install a library because it requires patching on NixOS (example 1, example 2, general issue). In this cases it is better to replace those libraries with ones from Nix.
 
Let's say, that nanomsg library fails to install in virtualenv. Then write a <code>shell.nix</code> file:
 
<syntaxhighlight lang="nix">
let
  pkgs = import <nixpkgs> {};
  nanomsg-py = ...build expression for this python library...;
in pkgs.mkShell {
  buildInputs = [
    pkgs.python3
    pkgs.python3.pkgs.requests
    nanomsg-py
  ];
  shellHook = ''
    # Tells pip to put packages into $PIP_PREFIX instead of the usual locations.
    # See https://pip.pypa.io/en/stable/user_guide/#environment-variables.
    export PIP_PREFIX=$(pwd)/_build/pip_packages
    export PYTHONPATH="$PIP_PREFIX/${pkgs.python3.sitePackages}:$PYTHONPATH"
    export PATH="$PIP_PREFIX/bin:$PATH"
    unset SOURCE_DATE_EPOCH
  '';
}
</syntaxhighlight>


After entering the environment with `nix-shell`, you can install new python libraries with dump `pip install`, but nanomsg will be detected as installed.
According to the [https://nixos.org/nixpkgs/manual/#contributing-guidelines official guidelines] for Python, new package expressions for libraries should be placed in <syntaxhighlight lang="bash" inline>pkgs/development/python-modules/<name>/default.nix</syntaxhighlight>.


Discussion and consequences of this approach are in PR https://github.com/NixOS/nixpkgs/pull/55265.
Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> like in this example:
 
=== micromamba ===
 
Install the <code>micromamba</code> package.
You can create environments and install packages as [https://github.com/mamba-org/mamba#micromamba documented by micromamba] e.g.
<syntaxhighlight lang="shell">
micromamba create -n my-environment python=3.9 numpy=1.23.0 -c conda-forge
</syntaxhighlight>
 
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ python
>>> import numpy as np
</syntaxhighlight>
 
Eventually you'll probably want to put this in a shell.nix so you won't have to type all that stuff every time e.g.:
<syntaxhighlight lang="shell">
{ pkgs ? import <nixpkgs> {}}:
let
  fhs = pkgs.buildFHSUserEnv {
    name = "my-fhs-environment";
 
    targetPkgs = _: [
      pkgs.micromamba
    ];
 
    profile = ''
      set -e
      eval "$(micromamba shell hook --shell=posix)"
      export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
      micromamba create -q -n my-mamba-environment
      micromamba activate my-mamba-environment
      micromamba install --yes -f conda-requirements.txt -c conda-forge
      set +e
    '';
  };
in fhs.env
</syntaxhighlight>
 
 
=== conda ===
 
Install the package <code>conda</code> and run
<syntaxhighlight lang="shell">
conda-shell
conda-install
conda env update --file environment.yml
</syntaxhighlight>
 
==== Imperative use ====
 
It is also possible to use <code>conda-install</code> directly.
On first use, run
<syntaxhighlight lang="shell">
conda-shell
conda-install
</syntaxhighlight>
to set up conda in <code>~/.conda</code>
 
=== pip2nix ===
[https://github.com/nix-community/pip2nix pip2nix] generate nix expressions for Python packages.
 
Also see the [https://github.com/nix-community/pypi2nix pypi2nix]-project (abandoned in 2019).
 
== Contribution guidelines ==
 
=== Libraries ===
 
According to the [https://nixos.org/nixpkgs/manual/#contributing-guidelines official guidelines] for python new package expressions for libraries should be placed in <syntaxhighlight lang="bash" inline>pkgs/development/python-modules/<name>/default.nix</syntaxhighlight>. Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> like in this example:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{
   aenum = callPackage ../development/python-modules/aenum { };
   aenum = callPackage ../development/python-modules/aenum { };
}
</syntaxhighlight>
</syntaxhighlight>
The reasoning behind this is the large size of <code>pkgs/top-level/python-packages.nix</code>.


=== Applications ===
=== Applications ===


Python applications instead should be referenced directly from <code>pkgs/top-level/all-packages.nix</code>.
Applications meant to be executed should be referenced directly from <code>pkgs/top-level/all-packages.nix</code>.


The expression should take <code>pythonPackages</code> as one of the arguments, which guarantees that packages belong to the same set. For example:
Other Python packages used in the Python package of the application should be taken from the <code>callPackage</code> argument <code>pythonPackages</code> , which guarantees that they belong to the same "pythonPackage" set. For example:


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix" line="1">
{ lib
{ lib
, pythonPackages
, pythonPackages
}:
}:
with pythonPackages;


buildPythonApplication rec {
buildPythonApplication rec {
# ...
  # ...
  propagatedBuildInputs = [
    pythonPackages.numpy
  ];
  # ...
}
</syntaxhighlight>
</syntaxhighlight>
== Special Modules ==
== Special Modules ==
Line 369: Line 299:


As of the time of this writing; these optimizations cause Python wheels to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.
As of the time of this writing; these optimizations cause Python wheels to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.
=== Regression ===
=== Regression ===
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:  
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:  
Line 383: Line 311:


However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.
However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.
=== Possible Optimizations ===
=== Possible Optimizations ===
If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:
If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:
Line 408: Line 334:


* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
* [[Packaging/Python]]
* [https://github.com/cachix/nixpkgs-python Nixpgs-python: A repo providing every version of python as flake ]
[[Category:Languages]]
[[Category:Languages]]
Retrieved from "https://wiki.nixos.org/wiki/Python"