Python: Difference between revisions

== Python environments with Nix ==

Python environment allow you to run Python code with the <code>python</code> command. It may also have Python packages installed as well.

 

 

=== Using regular Python virtual environment ===

 

It is possible to use Nix to create a Python virtual environment for your project. You can then use the Python virtual environment as usual.

 

To create a Python virtual environment:<syntaxhighlight lang="console">

$ nix-shell -p python3 --command "python -m venv .venv --copies"

 

 

 

* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.

 

* Setup <code>nix-ld</code> in your NixOS configuration.

 

=== Using the Nixpkgs Python infrastructure ===

   pkgs = import <nixpkgs> {};

     (pkgs.python3.withPackages (python-pkgs: [

       python-pkgs.pandas

       python-pkgs.requests

This example will make Nix create a Python environment with Python packages "pandas" and "requests". You can remove them, or add more.

You can find Python packages that are available in Nixpkgs using [https://search.nixos.org/packages search.nixos.org]. For instance, type a Python package name (for example <code>numpy</code>) in the search bar and click on the search button on the right. Then on the left in the "Package sets" section click on "python311Packages" to list only Python packages. for Python (version 3.11).

Note that in the snippet above, on line 8 and 9, each package to install is in the form <code>python-pkgs.<name></code>. The package name must correspond to the one found in [https://search.nixos.org/packages search.nixos.org] as explained above. To understand why it is prefixed by <code>python-pkgs</code>, read the [https://nix.dev/tutorials/nix-language.html Nix language basics].

Then, once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>) to make Nix create the Python environment and enter it. If it succeeds, Python should be available in your PATH, try using <code>python --version</code> at this point.

As a word of warning, using <code>pkgs = import <nixpkgs> {};</code> as in the snippet above (line 3) will result in a non-reproducible environment. Use Nixpkgs pinning or Nix Flakes to make your environment reproducible.

==== Using a Python packages not in Nixpkgs ====

Python packages in Nixpkgs are created and updated by Nixpkgs maintainers. Although the community invests a great effort to keep a complete and up-to-date package set, it isn't uncommon for packages you want to be missing, out of date or broken. To use your own packages in a Nix environment, you may package it yourself.

{ lib

, buildPythonPackage

, fetchPypi

, setuptools

, wheel

</syntaxhighlight>The tool [https://github.com/nix-community/pip2nix pip2nix] can help you generate such files.

 

Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">

     (pkgs.python3.withPackages (python-pkgs: [

       (pkgs.callPackage ./toolz.nix)

</syntaxhighlight>Note that the parenthesis (line 10) are required.

Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.

When working on a collaborative python project you may want to be able to <code>pip install -r requirements.txt</code> if the project isn't packaged for nix specifically.

The problem is that a lot of python packages won't work out of the box when you pip install them.

To fix this issue, you can create a nix shell that will use <code>pip</code> for the packages that are installing properly, but will fix the environment for the packages that are causing issues.

You can accomplish this by adding these two files to the root of your project:

<syntaxhighlight lang="nix" line="1">

{ pkgs ? import <nixpkgs> {} }:

   myPython = pkgs.python311;

   pythonPackages = pkgs.python311Packages;

  pythonWithPkgs = myPython.withPackages (pythonPkgs: with pythonPkgs; [

    # This list contains tools for Python development.

    # they will be reinstalled when running `pip -r requirements.txt` because

  ]);

   extraLibPackages = [

     # If you want to use CUDA, you should uncomment this line.

     # linuxPackages.nvidia_x11

   extraBuildInputs = with pkgs; [

     # this list contains packages that you want to be available at runtime and might not be able to be installed propely via pip

     # pythonPackages.pandas

     # pythonPackages.requests

import ./python-shell.nix {  

    extraLibPackages=extraLibPackages;  

     myPython=myPython;

    pythonWithPkgs=pythonWithPkgs;

<syntaxhighlight lang="nix" line="1">

# python-shell.nix

{ pkgs ? import <nixpkgs> {}, extraBuildInputs ? [], myPython ? pkgs.python3, extraLibPackages ? [], pythonWithPkgs? myPython }:

  lib-path = with pkgs; lib.makeLibraryPath buildInputs;

shell = pkgs.mkShell {

  buildInputs = [

  shellHook = ''

      SOURCE_DATE_EPOCH=$(date +%s)

      # Augment the dynamic linker path

      # Setup the virtual environment if it doesn't already exist.

      VENV=.venv

     (poetry.override { python = python310; })

$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'

   fhs = pkgs.buildFHSUserEnv {

First, we write the python code, say in a file <code>web_interface.py</code>. Here we create a basic flask web server;

Then, we create the <code>setup.py</code> file, which basically explains which are the executables:

setup(name='demo-flask-vuejs-rest',

       scripts=["web_interface.py"],

Finally, our nix derivation is now trivial: the file <code>derivation.nix</code> just needs to provide the python packages (here flask):

{ lib, python3Packages }:

with python3Packages;

buildPythonApplication {

   pname = "demo-flask-vuejs-rest";

   version = "1.0";

   propagatedBuildInputs = [ flask ];

</syntaxhighlight>

and we can now load this derivation from our file <code>default.nix</code>:

<syntaxhighlight lang="nix">

{ pkgs ? import <nixpkgs> {} }:

pkgs.callPackage ./derivation.nix {}

</syntaxhighlight>

We can now build with:

<syntaxhighlight lang=console>

$ nix-build

</syntaxhighlight>

or just enter a nix-shell, and directly execute your program or python if it's easier to develop:

[nix-shell]$ python

Type "help", "copyright", "credits" or "license" for more information.

>>> import flask

Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> like in this example:

{ lib

, pythonPackages

 

buildPythonApplication rec {

   propagatedBuildInputs = [

    pythonPackages.numpy

  ];

Additionally, when you compile something within <code>nix-shell</code> or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.

As of the time of this writing; these optimizations cause Python wheels to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.

However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.

If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:

* Enable the <code>enableOptimizations</code> flag for your Python derivation. [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example] Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.

* Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.

* Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs]