Python: Difference between revisions

     (pkgs.python3.withPackages (python-pkgs: [

       python-pkgs.pandas

       python-pkgs.requests

==== Using a Python packages not in Nixpkgs ====

{ lib

, buildPythonPackage

, fetchPypi

, setuptools

, wheel

</syntaxhighlight>The tool [https://github.com/nix-community/pip2nix pip2nix] can help you generate such files.

 

Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">

     (pkgs.python3.withPackages (python-pkgs: [

       (pkgs.callPackage ./toolz.nix)

</syntaxhighlight>Note that the parenthesis (line 10) are required.

==== Using nix-shell alongside pip  ====

When working on a collaborative python project you may want to be able to <code>pip install -r requirements.txt</code> if the project isn't packaged for nix specifically.

The problem is that a lot of python packages won't work out of the box when you pip install them.

To fix this issue, you can create a nix shell that will use <code>pip</code> for the packages that are installing properly, but will fix the environment for the packages that are causing issues.

<syntaxhighlight lang="nix" line="1">

# shell.nix

  myPython = pkgs.python311;

  pythonPackages = pkgs.python311Packages;

    # You can also add other tools, like black.

    # Note that even if you add Python packages here like PyTorch or Tensorflow,

    # virtualenv is used below in the shellHook.

   extraLibPackages = [

     # If you want to use CUDA, you should uncomment this line.

     # linuxPackages.nvidia_x11

   extraBuildInputs = with pkgs; [

     # this list contains packages that you want to be available at runtime and might not be able to be installed propely via pip

     # pythonPackages.pandas

     # pythonPackages.requests

import ./python-shell.nix {

{ pkgs ? import <nixpkgs> {}, extraBuildInputs ? [], myPython ? pkgs.python3, extraLibPackages ? [], pythonWithPkgs? myPython }:

let

    buildInputs  = with pkgs; [

   lib-path = with pkgs; lib.makeLibraryPath buildInputs;

shell = pkgs.mkShell {

   buildInputs = [

    # my python and packages

      # other packages needed for compiling python libs

      pkgs.readline

      pkgs.libffi

      pkgs.openssl

 

      # unfortunately needed because of messing with LD_LIBRARY_PATH below

      pkgs.git

      pkgs.openssh

      pkgs.rsync

   ] ++ extraBuildInputs;

   shellHook = ''

      # Allow the use of wheels.

      SOURCE_DATE_EPOCH=$(date +%s)

      export "LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${lib-path}"

      # Setup the virtual environment if it doesn't already exist.

      VENV=.venv

      if test ! -d $VENV; then

       fi

      source ./$VENV/bin/activate

      export PYTHONPATH=$PYTHONPATH:`pwd`/$VENV/${myPython.sitePackages}/

  '';

};

 

==== On NixOS ====

This method may not work on NixOS, as installing packages with <code>pip</code> that need to compile code or use C libraries will fail due to not finding dependencies in the expected places.

* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.

* Create a FHS user env with <code>buildFHSUserEnv</code>.

* Setup <code>nix-ld</code> in your NixOS configuration.

     (poetry.override { python = python310; })

$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'

   fhs = pkgs.buildFHSUserEnv {

First, we write the python code, say in a file <code>web_interface.py</code>. Here we create a basic flask web server;

Then, we create the <code>setup.py</code> file, which basically explains which are the executables:

setup(name='demo-flask-vuejs-rest',

       scripts=["web_interface.py"],

Finally, our nix derivation is now trivial: the file <code>derivation.nix</code> just needs to provide the python packages (here flask):

{ lib, python3Packages }:

with python3Packages;

buildPythonApplication {

   pname = "demo-flask-vuejs-rest";

   version = "1.0";

   propagatedBuildInputs = [ flask ];

</syntaxhighlight>

and we can now load this derivation from our file <code>default.nix</code>:

<syntaxhighlight lang="nix">

{ pkgs ? import <nixpkgs> {} }:

pkgs.callPackage ./derivation.nix {}

</syntaxhighlight>

We can now build with:

<syntaxhighlight lang=console>

$ nix-build

</syntaxhighlight>

or just enter a nix-shell, and directly execute your program or python if it's easier to develop:

[nix-shell]$ python

Type "help", "copyright", "credits" or "license" for more information.

>>> import flask

Those expressions are then referenced from <code>pkgs/top-level/python-packages.nix</code> like in this example:

{ lib

, pythonPackages

 

buildPythonApplication rec {

   propagatedBuildInputs = [

    pythonPackages.numpy

  ];

Additionally, when you compile something within <code>nix-shell</code> or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.

As of the time of this writing; these optimizations cause Python wheels to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.

However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.

If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:

* Enable the <code>enableOptimizations</code> flag for your Python derivation. [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example] Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.

* Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.

* Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs]