NixOS Wiki

Python: Difference between revisions

← Older edit
VisualWikitext
Samuela (talk | contribs)
9 edits
Lukeb (talk | contribs)
1 edit
fixed broken link for pyproject use-cases
 
(39 intermediate revisions by 23 users not shown)
Line 16: Line 16:
in pkgs.mkShell {
in pkgs.mkShell {
   packages = [
   packages = [
     (pkgs.python3.withPackages (python-pkgs: [
     (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [
       # select Python packages here
       # select Python packages here
       python-pkgs.pandas
       pandas
       python-pkgs.requests
       requests
     ]))
     ]))
   ];
   ];
Line 30: Line 30:


Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Note that with NixOS, this method can be used to install packages at the system level, e.g. <syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
  # ...
  (python3.withPackages (python-pkgs: with python-pkgs; [
      pandas
      requests
  ]))
];
</syntaxhighlight>
=== Using Nix shell (new command line) ===
nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])'
If you don't use the channels any more, you can replace <code><nixpkgs></code> by an instance of the <code>NixOS/nixpkgs</code> repository using its absolute path.


==== Using a Python package not in Nixpkgs ====
==== Using a Python package not in Nixpkgs ====
Line 38: Line 52:
Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1">
Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1">
# toolz.nix
# toolz.nix
{ lib
{
, buildPythonPackage
  lib,
, fetchPypi
  buildPythonPackage,
, setuptools
  fetchPypi,
, wheel
  setuptools,
  wheel,
}:
}:


Line 64: Line 79:
   ];
   ];
}
}
</syntaxhighlight>The tool [https://github.com/nix-community/pip2nix pip2nix] can help you generate such files.
</syntaxhighlight>Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
 
Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
# shell.nix
# shell.nix
let
let
   pkgs = import <nixpkgs> {};
   pkgs = import <nixpkgs> {};
  python = pkgs.python3.override {
    self = python;
    packageOverrides = pyfinal: pyprev: {
      toolz = pyfinal.callPackage ./toolz.nix { };
    };
  };
in pkgs.mkShell {
in pkgs.mkShell {
   packages = [
   packages = [
     (pkgs.python3.withPackages (python-pkgs: [
     (python.withPackages (python-pkgs: [
       # select Python packages here
       # select Python packages here
       python-pkgs.pandas
       python-pkgs.pandas
       python-pkgs.requests
       python-pkgs.requests
       (pkgs.callPackage ./toolz.nix)
       python-pkgs.toolz
     ]))
     ]))
   ];
   ];
}
}
</syntaxhighlight>Note that the parenthesis (line 10) are required.
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
 
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> ===


Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session">
==== Using nix-shell alongside pip  ====
$ python -c 'import grpc'
When working on a collaborative python project you may want to be able to <code>pip install -r requirements.txt</code> if the project isn't packaged for nix specifically.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/.../grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory


The problem is that a lot of python packages won't work out of the box when you pip install them.
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.
To fix this issue, you can create a nix shell that will use <code>pip</code> for the packages that are installing properly, but will fix the environment for the packages that are causing issues.


You can accomplish this by adding these two files to the root of your project:
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>.
<syntaxhighlight lang="nix" line="1">
 
# shell.nix
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers.
{ pkgs ? import <nixpkgs> {} }:


let
==== Using nix overlay ====
   myPython = pkgs.python311;
<syntaxhighlight lang="nix">
   pythonPackages = pkgs.python311Packages;
{
   nixpkgs.overlays = [(
    self: super: rec {
      # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
      # > We currently take all libraries from systemd and nix as the default.
      pythonldlibpath = lib.makeLibraryPath (with super; [
        zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
      ]);
      # here we are overriding python program to add LD_LIBRARY_PATH to it's env
      python = super.stdenv.mkDerivation {
        name = "python";
        buildInputs = [ super.makeWrapper ];
        src = super.python311;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/python3 --set LD_LIBRARY_PATH ${pythonldlibpath}
          wrapProgram $out/bin/python3.11 --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
      poetry = super.stdenv.mkDerivation {
        name = "poetry";
        buildInputs = [ super.makeWrapper ];
        src = super.poetry;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
    }
  )];
   environment.systemPackages = with pkgs; [
    python  # here python will be taken from the overlay up here
    poetry
  ];
}
</syntaxhighlight>


  pythonWithPkgs = myPython.withPackages (pythonPkgs: with pythonPkgs; [
==== Using [https://github.com/Mic92/nix-ld nix-ld] ====
    # This list contains tools for Python development.
    # You can also add other tools, like black.
    #
    # Note that even if you add Python packages here like PyTorch or Tensorflow,
    # they will be reinstalled when running `pip -r requirements.txt` because
    # virtualenv is used below in the shellHook.
    ipython
    pip
    setuptools
    virtualenvwrapper
    wheel
    black
  ]);


   extraBuildInputs = with pkgs; [
<syntaxhighlight lang="nix">
     # this list contains packages that you want to be available at runtime and might not be able to be installed propely via pip
{
     # pythonPackages.pandas
   programs.nix-ld = {
     # pythonPackages.requests
    enable = true;
    libraries = with pkgs; [
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
     ];
  };
  # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
     (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
     '')
   ];
   ];
in
  # another (dangerous) solution
import ./python-shell.nix {  
  # environment.systemPackages = with pkgs; [ python3 ];
    extraBuildInputs=extraBuildInputs;  
  # programs.bash = {
    extraLibPackages=extraLibPackages;
  #  enable = true;
     myPython=myPython;
  #  initExtra = ''
    pythonWithPkgs=pythonWithPkgs;
  #     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #  '';
  # };
}
}
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="nix" line="1">
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ====
# python-shell.nix
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}}
{ pkgs ? import <nixpkgs> {}, extraBuildInputs ? [], myPython ? pkgs.python3, extraLibPackages ? [], pythonWithPkgs? myPython }:


Install with:


let
<syntaxhighlight lang="shell">
nix profile install github:GuillaumeDesforges/fix-python
</syntaxhighlight>
 
Enter the venv and run:


    buildInputs  = with pkgs; [
<syntaxhighlight lang="shell">
        clang
fix-python --venv .venv
        llvmPackages_16.bintools
</syntaxhighlight>
        rustup
    ] ++ extraBuildInputs;


   lib-path = with pkgs; lib.makeLibraryPath buildInputs;
==== Using <code>buildFHSEnv</code> (Recommended) ====
<syntaxhighlight lang="nix">
#!/usr/bin/env nix-shell
{ pkgs ? import <nixpkgs> { } }:
(
  let base = pkgs.appimageTools.defaultFhsEnvArgs; in
   pkgs.buildFHSEnv (base // {
    name = "FHS";
    targetPkgs = pkgs: (with pkgs; [
      gcc glibc zlib
    ]);
    runScript = "zsh";
    extraOutputsToInstall = [ "dev" ];
  })
).env
</syntaxhighlight>


==== Using a custom nix-shell ====


shell = pkgs.mkShell {
The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1">
   buildInputs = [
let
    # my python and packages
  python = pkgs.python311;
       pythonWithPkgs
  # We currently take all libraries from systemd and nix as the default
        
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
       # other packages needed for compiling python libs
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
       pkgs.readline
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
       pkgs.libffi
  ]);
       pkgs.openssl
  patchedpython = (python.overrideAttrs (
    previousAttrs: {
       # Add the nix-ld libraries to the LD_LIBRARY_PATH.
       # creating a new library path from all desired libraries
       postInstall = previousAttrs.postInstall + ''
        mv  "$out/bin/python3.11" "$out/bin/unpatched_python3.11"
        cat << EOF >> "$out/bin/python3.11"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_python3.11" "\$@"
        EOF
        chmod +x "$out/bin/python3.11"
       '';
    }
  ));
  # if you want poetry
  patchedpoetry =  ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs (
    previousAttrs: {
       # same as above, but for poetry
      # not that if you dont keep the blank line bellow, it crashes :(
       postInstall = previousAttrs.postInstall + ''


      # unfortunately needed because of messing with LD_LIBRARY_PATH below
        mv "$out/bin/poetry" "$out/bin/unpatched_poetry"
      pkgs.git
        cat << EOF >> "$out/bin/poetry"
      pkgs.openssh
        #!/run/current-system/sw/bin/bash
      pkgs.rsync
        export LD_LIBRARY_PATH="${pythonldlibpath}"
  ] ++ extraBuildInputs;
        exec "$out/bin/unpatched_poetry" "\$@"
  shellHook = ''
        EOF
      # Allow the use of wheels.
        chmod +x "$out/bin/poetry"
      SOURCE_DATE_EPOCH=$(date +%s)
      '';
      # Augment the dynamic linker path
    }
      export "LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${lib-path}"
  ));
      # Setup the virtual environment if it doesn't already exist.
in
      VENV=.venv
{
      if test ! -d $VENV; then
  environment.systemPackages = with pkgs; [
        virtualenv $VENV
    patchedpython
      fi
    # if you want poetry
      source ./$VENV/bin/activate
    patchedpoetry
      export PYTHONPATH=$PYTHONPATH:`pwd`/$VENV/${myPython.sitePackages}/
  ];
  '';
}
};
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.


==== Prefix library paths using wrapProgram ====
wrapProgram is a part of the makeWrapper build input<ref>https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/make-wrapper.sh</ref>. By combining it with the symlinkJoin, we can create a wrapper around the Python executable that will always set the required library paths. It’s worth noting that, for this solution to be compatible with Darwin, we need to use a different wrap prefix, as shown in the example below.<syntaxhighlight lang="nixos" line="1">
let
  # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
  ]);
  # Darwin requires a different library path prefix
  wrapPrefix = if (!pkgs.stdenv.isDarwin) then "LD_LIBRARY_PATH" else "DYLD_LIBRARY_PATH";
  patchedpython = (pkgs.symlinkJoin {
    name = "python";
    paths = [ pkgs.python312 ];
    buildInputs = [ pkgs.makeWrapper ];
    postBuild = ''
      wrapProgram "$out/bin/python3.12" --prefix ${wrapPrefix} : "${pythonldlibpath}"
    '';
  });
in
in
shell
{
  environment.systemPackages = with pkgs; [
    patchedpython
  ];
}
</syntaxhighlight>
</syntaxhighlight>


Line 183: Line 306:
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.


==== On NixOS ====
=== Using uv ===
This method may not work on NixOS, as installing packages with <code>pip</code> that need to compile code or use C libraries will fail due to not finding dependencies in the expected places.
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>
 
uv is written in Rust and does ''not'' need Python as a prerequisite. Use the <code>uv</code> command to initialize Python projects, add Python packages, create or update virtual environments (in <code>.venv</code> folders), etc. as [https://docs.astral.sh/uv/concepts/projects/ described in the uv docs]. Use uv's [https://docs.astral.sh/uv/guides/tools/ tool interface] to install and run Python packages that provide a CLI.
 
As a system package
 
<syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
    uv
];
</syntaxhighlight>
 
or as a home-manager package


There are multiple ways to make it work:
<syntaxhighlight lang="nix">
home.packages = with pkgs; [
    uv
];
</syntaxhighlight>


* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
If you use uv it's recommended that you install the Python versions you need using the <code>uv python install</code> command, e.g.


* Create a FHS user env with <code>buildFHSUserEnv</code>.
<syntaxhighlight lang="python">
* Setup <code>nix-ld</code> in your NixOS configuration.
uv python install 3.14 --preview --default
</syntaxhighlight>
 
You may want to set the <code>UV_PYTHON_DOWNLOADS=never</code> environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting <code>environment.localBinInPath = true;</code> is highly recommended, because uv will install binaries in <code>~/.local/bin</code>.


=== Using poetry ===
=== Using poetry ===
Line 202: Line 344:
   packages = with pkgs; [
   packages = with pkgs; [
     python310
     python310
     (poetry.override { python = python310; })
     (poetry.override { python3 = python310; })
   ];
   ];
}
}
</syntaxhighlight>
</syntaxhighlight>


==== poetry2nix ====
[https://github.com/nix-community/poetry2nix poetry2nix] uses the contents of a <code>poetry.lock</code> and <code>pyproject.toml</code> file to create Nix derivations. It has several functions for generating development environments and python projects. Because some older python projects rely on deprecated build systems (see [https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md edgecase.md] for more info), poetry2nix provides overrides so these packages can still be built.
=== Using micromamba ===
=== Using micromamba ===


Line 213: Line 357:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ micromamba activate my-environment
Line 224: Line 368:
{ pkgs ? import <nixpkgs> {}}:
{ pkgs ? import <nixpkgs> {}}:
let
let
   fhs = pkgs.buildFHSUserEnv {
   fhs = pkgs.buildFHSEnv {
     name = "my-fhs-environment";
     name = "my-fhs-environment";


Line 262: Line 406:
</syntaxhighlight>
</syntaxhighlight>
to set up conda in <code>~/.conda</code>
to set up conda in <code>~/.conda</code>
=== Using pixi ===
Install the <code>pixi</code> package to create environments and install packages as [https://pixi.sh/latest/ documented by pixi]:<syntaxhighlight lang="console">
$ pixi init
$ pixi add python
</syntaxhighlight>To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g. a [https://github.com/NixOS/nixpkgs/issues/316443#issuecomment-2151963505 flake.nix] <ref>Suggested flake.nix by [https://github.com/jonas-w @jonas-w] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos">
{
  description = "pixi env";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs =
    { flake-utils, nixpkgs, ... }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs { inherit system; };
        fhs = pkgs.buildFHSEnv {
          name = "pixi-env";
          targetPkgs = _: [ pkgs.pixi ];
        };
      in
      {
        devShell = fhs.env;
      }
    );
}
</syntaxhighlight>Then using  <code>[https://wiki.nixos.org/wiki/Development_environment_with_nix-shell#nix_develop nix develop]</code> the environment can be activated:<syntaxhighlight lang="console">
$ nix develop
$ pixi s
$ python
>>> import numpy as np
</syntaxhighlight>
==== Using Home Manager ====
Alternatively an [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] can wrap pixi using [[Home Manager]] e.g<ref>Suggested home manager config by [https://github.com/jennydaman @jennydaman] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos">
{ lib, pkgs, ... }:
{
  home.packages = [
    (pkgs.buildFHSEnv {
      name = "pixi";
      runScript = "pixi";
      targetPkgs = pkgs: with pkgs; [ pixi ];
    })
  ];
}
</syntaxhighlight>Then the environment can be activated:<syntaxhighlight lang="console">
$ pixi s
$ python
>>> import numpy as np
</syntaxhighlight>


== Package a Python application ==
== Package a Python application ==
It is possible to use <code>buildPythonApplication</code> to package python applications. As explained in the nixpkgs manual, it uses the widely used <code>setup.py</code> file in order to package properly the application. We now show how to package a simple python application: a basic flask web server.


First, we write the python code, say in a file <code>web_interface.py</code>. Here we create a basic flask web server:
=== With <code>setup.py</code> ===
To package a Python application that uses <code>setup.py</code> you can use <code>buildPythonApplication</code>. More details about this and similar functions can be found in [https://nixos.org/manual/nixpkgs/stable/#building-packages-and-applications the nixpkgs manual].
 
For example, we can package this simple flask server <code>main.py:</code>
<syntaxhighlight lang="python">
<syntaxhighlight lang="python">
#!/usr/bin/env python
#!/usr/bin/env python


from flask import Flask
from flask import Flask
app = Flask(__name__)
app = Flask(__name__)


Line 281: Line 480:
</syntaxhighlight>
</syntaxhighlight>


Then, we create the <code>setup.py</code> file, which basically explains which are the executables:
We also need a <code>setup.py</code> file, like this:
<syntaxhighlight lang="python">
<syntaxhighlight lang="python">
#!/usr/bin/env python
from setuptools import setup, find_packages
from setuptools import setup, find_packages


setup(name='demo-flask-vuejs-rest',
setup(name='myFlaskServer',
       version='1.0',
       version='1.0',
       # Modules to import from other scripts:
       # Modules to import from other scripts:
       packages=find_packages(),
       packages=find_packages(),
       # Executables
       # Executables
       scripts=["web_interface.py"],
       scripts=["main.py"],
     )
     )
</syntaxhighlight>
</syntaxhighlight>


Finally, our nix derivation is now trivial: the file <code>derivation.nix</code> just needs to provide the python packages (here flask):
Then, we use the <code>buildPythonApplication</code> in the <code>default.nix</code>:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ lib, python3Packages }:
{ pkgs ? import <nixpkgs> {} }:
with python3Packages;
 
buildPythonApplication {
pkgs.python3Packages.buildPythonApplication {
   pname = "demo-flask-vuejs-rest";
   pname = "myFlaskApp";
   version = "1.0";
   version = "0.1.0";


   propagatedBuildInputs = [ flask ];
   propagatedBuildInputs = with pkgs.python3Packages; [
    flask
  ];


   src = ./.;
   src = ./.;
}
}
</syntaxhighlight>
</syntaxhighlight>Finally, build your project using <code>nix-build</code>. The result will be executable in <code>./result/bin/app.py</code>.
 
=== With <code>pyproject.toml</code> ===
When your project is using <code>pyproject.toml</code>you can use [https://github.com/nix-community/pyproject.nix pyproject.nix] to package your application.
 
First, a simple file structure could look like this:<syntaxhighlight>
├── app/
    └── main.py
├── flake.nix
├── pyproject.toml
└── README.md
 
 
</syntaxhighlight>To reuse the example from above, we use the same flask application:<syntaxhighlight lang="python">
from flask import Flask
 
app = Flask(__name__)
 
@app.route('/')
def hello_world():
    return 'Hello, World!'
 
if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://pyproject-nix.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
[project]
name = "my-app"
version = "0.1.0"
description = "Simple app"
 
# define any Python dependencies
dependencies = [
  "flask>3",
]
 
# define the CLI executable
# Here, we define the entry point to be the 'main()' function in the module 'app/main.py'
[project.scripts]
cli = "app.main:main"
</syntaxhighlight>We package the application by calling the <code>loadPyproject</code> function from <code>pyproject.nix</code>. Again, we only show a minimal example. More information can be found in the [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation]. Note that this example relies on flakes in contrast to some of the others on this page.<syntaxhighlight lang="nix">
{
  description = "A basic flake using pyproject.toml project metadata";
 
  inputs = {
    pyproject-nix = {
      url = "github:nix-community/pyproject.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
 
  outputs = { nixpkgs, pyproject-nix, ... }:
    let
      inherit (nixpkgs) lib;
 
      project = pyproject-nix.lib.project.loadPyproject {
        # Read & unmarshal pyproject.toml relative to this project root.
        # projectRoot is also used to set `src` for renderers such as buildPythonPackage.
        projectRoot = ./.;
      };
 
      # This example is only using x86_64-linux
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
 
      python = pkgs.python3;
 
    in
    {
      # Build our package using `buildPythonPackage
      packages.x86_64-linux.default =
        let
          # Returns an attribute set that can be passed to `buildPythonPackage`.
          attrs = project.renderers.buildPythonPackage { inherit python; };
        in
        # Pass attributes to buildPythonPackage.
        # Here is a good spot to add on any missing or custom attributes.
        python.pkgs.buildPythonPackage (attrs // {
          env.CUSTOM_ENVVAR = "hello";
        });
    };
}


and we can now load this derivation from our file <code>default.nix</code>:
</syntaxhighlight>To run the application, call <code>nix run</code>.
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}
</syntaxhighlight>


We can now build with:
You can also launch an IDE under <code>nix develop</code> and get full dependency resolution. For example, the following command opens VS Code in the constructed environment:
<syntaxhighlight lang=console>
$ nix-build
[...]
$ ./result/bin/web_interface.py
* Serving Flask app ".web_interface" (lazy loading)
[...]
</syntaxhighlight>
or just enter a nix-shell, and directly execute your program or python if it's easier to develop:
<syntaxhighlight lang=console>
$ nix-shell
[...]
[nix-shell]$ chmod +x web_interface.py
[nix-shell]$ ./web_interface.py
* Serving Flask app "web_interface" (lazy loading)
[...]


[nix-shell]$ python
<syntaxhighlight lang="shell-session">
Python 3.8.7 (default, Dec 21 2020, 17:18:55)
$ nix develop --command code
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>
</syntaxhighlight>
</syntaxhighlight>


Line 360: Line 615:


<syntaxhighlight lang="nix" line="1">
<syntaxhighlight lang="nix" line="1">
{ lib
{
, pythonPackages
  lib,
  pythonPackages,
}:
}:
buildPythonApplication {
buildPythonApplication {
Line 390: Line 646:


You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
== Debug Build ==
See [https://docs.python.org/3/using/configure.html#python-debug-build python wiki on debug build].
In order to use a CPython interpreter built using <code>--with-pydebug</code> during configure phase, override any of the python packages passing <code>enableDebug = true</code> argument:
<code>pythonDebug = pkgs.python310.override { enableDebug = true; };</code>
== Installing Multiple Versions ==
{{Nixpkgs Manual|name={{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}}|anchor=#function-library-lib.meta.lowPrio}} can be used to install multiple versions without conflicts ({{issue|369997}}):
{{file|/etc/nixos/configuration.nix|nix|3=
environment.systemPackages = with pkgs; [
    python313
    lib.meta.lowPrio python314
];
}}
In this case you may run Python 3.13 via {{ic|python}} or {{ic|python3.13}} and Python 3.14 via {{ic|python3.14}}.
{{Note|If you wrapped Python as described in [[#Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix]], call {{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}} for the wrapped Python instead of calling them inside the wrapping.}}


== Performance ==
== Performance ==
The derivation of CPython that is available via <code>nixpkgs</code> only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options]  
The derivation of CPython that is available via <code>nixpkgs</code> only contains optimizations that do not harm reproducibility. Link-Time-Optimization (LTO) is only enabled on 64-bit Linux systems, while Profile Guided Optimization (PGO) is currently disabled. See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options]  
Additionally, when you compile something within <code>nix-shell</code> or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.
Additionally, when compiling something within <code>nix-shell</code> or a derivation security hardening flags are passed to the compiler by default which may have a small performance impact.


As of the time of this writing; these optimizations cause Python wheels to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.
At the time of writing certain optimizations cause Python wheels to be non-reproducible and increase install times. For a detailed overview of the trials and tribulations of discovering such performance regressions see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?].
=== Regression ===
=== Regression ===
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:  
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:  
Line 407: Line 685:
</syntaxhighlight>
</syntaxhighlight>


However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.
However, synthetic benchmarks are not necessarily reflective of real-world performance. In common real-world situations, the performance difference between optimized and non-optimized interpreters is minimal. For example, using <code>pylint</code> with a significant number of custom linters to scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference. Other workflows that were not performance sensitive saw no impact to their run times.
=== Possible Optimizations ===
=== Possible Optimizations ===
If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:
If you run code that heavily depends on Python performance, and you desire the most performant Python interpreter possible, here are some possible things you can do:


* Enable the <code>enableOptimizations</code> flag for your Python derivation. [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example] Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.
* '''Enable the <code>enableOptimizations</code> flag for your Python derivation'''. See [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example]. Do note that this will cause you to compile Python the first time that you run it which will take a few minutes.
* Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
* '''Switch to a newer version of Python'''. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement, but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
* Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs]
* '''Disable hardening'''. Beware this only yields a small performance boost and it has impacts beyond Python code. See [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs].


'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.'''
'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.'''
Line 432: Line 710:
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
[[Category:Languages]]
[[Category:Languages]]
[[Category:Python]]
Retrieved from "https://wiki.nixos.org/wiki/Python"