NixOS Wiki

Python: Difference between revisions

← Older edit
VisualWikitext
Klinger (talk | contribs)
trusted
904 edits
Lukeb (talk | contribs)
1 edit
fixed broken link for pyproject use-cases
 
(23 intermediate revisions by 16 users not shown)
Line 16: Line 16:
in pkgs.mkShell {
in pkgs.mkShell {
   packages = [
   packages = [
     (pkgs.python3.withPackages (python-pkgs: [
     (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [
       # select Python packages here
       # select Python packages here
       python-pkgs.pandas
       pandas
       python-pkgs.requests
       requests
     ]))
     ]))
   ];
   ];
Line 30: Line 30:


Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Note that with NixOS, this method can be used to install packages at the system level, e.g. <syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
  # ...
  (python3.withPackages (python-pkgs: with python-pkgs; [
      pandas
      requests
  ]))
];
</syntaxhighlight>
=== Using Nix shell (new command line) ===
nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])'
If you don't use the channels any more, you can replace <code><nixpkgs></code> by an instance of the <code>NixOS/nixpkgs</code> repository using its absolute path.


==== Using a Python package not in Nixpkgs ====
==== Using a Python package not in Nixpkgs ====
Line 38: Line 52:
Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1">
Generally, you may create a file that looks like this:<syntaxhighlight lang="nix" line="1">
# toolz.nix
# toolz.nix
{ lib
{
, buildPythonPackage
  lib,
, fetchPypi
  buildPythonPackage,
, setuptools
  fetchPypi,
, wheel
  setuptools,
  wheel,
}:
}:


Line 87: Line 102:
}
}
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
=== Running compiled libraries ===
 
If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> ===
 
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
$ python -c 'import grpc'
Traceback (most recent call last):
Traceback (most recent call last):
Line 98: Line 115:
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory


</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.
 
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>.
 
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers.
 
==== Using nix overlay ====
<syntaxhighlight lang="nix">
{
  nixpkgs.overlays = [(
    self: super: rec {
      # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
      # > We currently take all libraries from systemd and nix as the default.
      pythonldlibpath = lib.makeLibraryPath (with super; [
        zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
      ]);
      # here we are overriding python program to add LD_LIBRARY_PATH to it's env
      python = super.stdenv.mkDerivation {
        name = "python";
        buildInputs = [ super.makeWrapper ];
        src = super.python311;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/python3 --set LD_LIBRARY_PATH ${pythonldlibpath}
          wrapProgram $out/bin/python3.11 --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
      poetry = super.stdenv.mkDerivation {
        name = "poetry";
        buildInputs = [ super.makeWrapper ];
        src = super.poetry;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
    }
  )];
  environment.systemPackages = with pkgs; [
    python  # here python will be taken from the overlay up here
    poetry
  ];
}
</syntaxhighlight>
 
==== Using [https://github.com/Mic92/nix-ld nix-ld] ====
 
<syntaxhighlight lang="nix">
{
  programs.nix-ld = {
    enable = true;
    libraries = with pkgs; [
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    ];
  };
  # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
    (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
    '')
  ];
  # another (dangerous) solution
  # environment.systemPackages = with pkgs; [ python3 ];
  # programs.bash = {
  #  enable = true;
  #  initExtra = ''
  #    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #  '';
  # };
}
</syntaxhighlight>
 
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ====
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}}
 
Install with:
 
<syntaxhighlight lang="shell">
nix profile install github:GuillaumeDesforges/fix-python
</syntaxhighlight>
 
Enter the venv and run:


On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations.
<syntaxhighlight lang="shell">
There are multiple ways to make it work:
fix-python --venv .venv
* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
</syntaxhighlight>


* Create a FHS user env with <code>buildFHSUserEnv</code>.
==== Using <code>buildFHSEnv</code> (Recommended) ====
* Setup <code>nix-ld</code><ref name=":0">https://github.com/Mic92/nix-ld</ref> in your NixOS configuration.
<syntaxhighlight lang="nix">
#!/usr/bin/env nix-shell
{ pkgs ? import <nixpkgs> { } }:
(
  let base = pkgs.appimageTools.defaultFhsEnvArgs; in
  pkgs.buildFHSEnv (base // {
    name = "FHS";
    targetPkgs = pkgs: (with pkgs; [
      gcc glibc zlib
    ]);
    runScript = "zsh";
    extraOutputsToInstall = [ "dev" ];
  })
).env
</syntaxhighlight>


==== Setup nix-ld ====
==== Using a custom nix-shell ====
nix-ld<ref name=":0" /> allow you to <code>Run unpatched dynamic binaries on NixOS.</code>


The following configuration automatically fix the dependencies :<syntaxhighlight lang="nixos" line="1">
The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1">
let
let
   python = pkgs.python311;
   python = pkgs.python311;
  # We currently take all libraries from systemd and nix as the default
   # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
   # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  # > We currently take all libraries from systemd and nix as the default.
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
     zlib
     zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    zstd
    stdenv.cc.cc
    curl
    openssl
    attr
    libssh
    bzip2
    libxml2
    acl
    libsodium
    util-linux
    xz
    systemd
   ]);
   ]);
   patchedpython = (python.overrideAttrs (
   patchedpython = (python.overrideAttrs (
Line 165: Line 266:
in
in
{
{
  # Some other config...
 
   environment.systemPackages = with pkgs; [
   environment.systemPackages = with pkgs; [
     patchedpython
     patchedpython
     # if you want poetry
     # if you want poetry
     patchedpoetry
     patchedpoetry
Line 176: Line 274:
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.


After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...
==== Prefix library paths using wrapProgram ====
wrapProgram is a part of the makeWrapper build input<ref>https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/make-wrapper.sh</ref>. By combining it with the symlinkJoin, we can create a wrapper around the Python executable that will always set the required library paths. It’s worth noting that, for this solution to be compatible with Darwin, we need to use a different wrap prefix, as shown in the example below.<syntaxhighlight lang="nixos" line="1">
let
  # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
  ]);
  # Darwin requires a different library path prefix
  wrapPrefix = if (!pkgs.stdenv.isDarwin) then "LD_LIBRARY_PATH" else "DYLD_LIBRARY_PATH";
  patchedpython = (pkgs.symlinkJoin {
    name = "python";
    paths = [ pkgs.python312 ];
    buildInputs = [ pkgs.makeWrapper ];
    postBuild = ''
      wrapProgram "$out/bin/python3.12" --prefix ${wrapPrefix} : "${pythonldlibpath}"
    '';
  });
in
{
  environment.systemPackages = with pkgs; [
    patchedpython
  ];
}
</syntaxhighlight>


=== Using <code>venv</code> ===
=== Using <code>venv</code> ===
Line 183: Line 305:
$ nix-shell -p python3 --command "python -m venv .venv --copies"
$ nix-shell -p python3 --command "python -m venv .venv --copies"
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
=== Using uv ===
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>
uv is written in Rust and does ''not'' need Python as a prerequisite. Use the <code>uv</code> command to initialize Python projects, add Python packages, create or update virtual environments (in <code>.venv</code> folders), etc. as [https://docs.astral.sh/uv/concepts/projects/ described in the uv docs]. Use uv's [https://docs.astral.sh/uv/guides/tools/ tool interface] to install and run Python packages that provide a CLI.
As a system package
<syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
    uv
];
</syntaxhighlight>
or as a home-manager package
<syntaxhighlight lang="nix">
home.packages = with pkgs; [
    uv
];
</syntaxhighlight>
If you use uv it's recommended that you install the Python versions you need using the <code>uv python install</code> command, e.g.
<syntaxhighlight lang="python">
uv python install 3.14 --preview --default
</syntaxhighlight>
You may want to set the <code>UV_PYTHON_DOWNLOADS=never</code> environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting <code>environment.localBinInPath = true;</code> is highly recommended, because uv will install binaries in <code>~/.local/bin</code>.


=== Using poetry ===
=== Using poetry ===
Line 198: Line 349:
</syntaxhighlight>
</syntaxhighlight>


==== poetry2nix ====
[https://github.com/nix-community/poetry2nix poetry2nix] uses the contents of a <code>poetry.lock</code> and <code>pyproject.toml</code> file to create Nix derivations. It has several functions for generating development environments and python projects. Because some older python projects rely on deprecated build systems (see [https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md edgecase.md] for more info), poetry2nix provides overrides so these packages can still be built.
=== Using micromamba ===
=== Using micromamba ===


Line 204: Line 357:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ micromamba activate my-environment
Line 215: Line 368:
{ pkgs ? import <nixpkgs> {}}:
{ pkgs ? import <nixpkgs> {}}:
let
let
   fhs = pkgs.buildFHSUserEnv {
   fhs = pkgs.buildFHSEnv {
     name = "my-fhs-environment";
     name = "my-fhs-environment";


Line 253: Line 406:
</syntaxhighlight>
</syntaxhighlight>
to set up conda in <code>~/.conda</code>
to set up conda in <code>~/.conda</code>
=== Using pixi ===
Install the <code>pixi</code> package to create environments and install packages as [https://pixi.sh/latest/ documented by pixi]:<syntaxhighlight lang="console">
$ pixi init
$ pixi add python
</syntaxhighlight>To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g. a [https://github.com/NixOS/nixpkgs/issues/316443#issuecomment-2151963505 flake.nix] <ref>Suggested flake.nix by [https://github.com/jonas-w @jonas-w] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos">
{
  description = "pixi env";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs =
    { flake-utils, nixpkgs, ... }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs { inherit system; };
        fhs = pkgs.buildFHSEnv {
          name = "pixi-env";
          targetPkgs = _: [ pkgs.pixi ];
        };
      in
      {
        devShell = fhs.env;
      }
    );
}
</syntaxhighlight>Then using  <code>[https://wiki.nixos.org/wiki/Development_environment_with_nix-shell#nix_develop nix develop]</code> the environment can be activated:<syntaxhighlight lang="console">
$ nix develop
$ pixi s
$ python
>>> import numpy as np
</syntaxhighlight>
==== Using Home Manager ====
Alternatively an [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] can wrap pixi using [[Home Manager]] e.g<ref>Suggested home manager config by [https://github.com/jennydaman @jennydaman] in response to '''[https://github.com/NixOS/nixpkgs/issues/316443 #316443 <bdi>Unable to add / run packages with pixi</bdi>]'''</ref><syntaxhighlight lang="nixos">
{ lib, pkgs, ... }:
{
  home.packages = [
    (pkgs.buildFHSEnv {
      name = "pixi";
      runScript = "pixi";
      targetPkgs = pkgs: with pkgs; [ pixi ];
    })
  ];
}
</syntaxhighlight>Then the environment can be activated:<syntaxhighlight lang="console">
$ pixi s
$ python
>>> import numpy as np
</syntaxhighlight>


== Package a Python application ==
== Package a Python application ==
Line 326: Line 531:
if __name__ == '__main__':
if __name__ == '__main__':
     app.run(host="0.0.0.0", port=8080)
     app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://pyproject-nix.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
[project]
[project]
name = "my-app"
name = "my-app"
Line 341: Line 546:
[project.scripts]
[project.scripts]
cli = "app.main:main"
cli = "app.main:main"
</syntaxhighlight>We package the application by calling the <code>loadPyproject</code> function from <code>pyproject.nix</code>. Again, we only show a minimal example. More information can be found in the [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation].<syntaxhighlight lang="nix">
</syntaxhighlight>We package the application by calling the <code>loadPyproject</code> function from <code>pyproject.nix</code>. Again, we only show a minimal example. More information can be found in the [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation]. Note that this example relies on flakes in contrast to some of the others on this page.<syntaxhighlight lang="nix">
{
{
   description = "A basic flake using pyproject.toml project metadata";
   description = "A basic flake using pyproject.toml project metadata";
Line 384: Line 589:


</syntaxhighlight>To run the application, call <code>nix run</code>.
</syntaxhighlight>To run the application, call <code>nix run</code>.
You can also launch an IDE under <code>nix develop</code> and get full dependency resolution. For example, the following command opens VS Code in the constructed environment:
<syntaxhighlight lang="shell-session">
$ nix develop --command code
</syntaxhighlight>


== Nixpkgs Python contribution guidelines ==
== Nixpkgs Python contribution guidelines ==
Line 404: Line 615:


<syntaxhighlight lang="nix" line="1">
<syntaxhighlight lang="nix" line="1">
{ lib
{
, pythonPackages
  lib,
  pythonPackages,
}:
}:
buildPythonApplication {
buildPythonApplication {
Line 434: Line 646:


You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
== Debug Build ==
See [https://docs.python.org/3/using/configure.html#python-debug-build python wiki on debug build].
In order to use a CPython interpreter built using <code>--with-pydebug</code> during configure phase, override any of the python packages passing <code>enableDebug = true</code> argument:
<code>pythonDebug = pkgs.python310.override { enableDebug = true; };</code>
== Installing Multiple Versions ==
{{Nixpkgs Manual|name={{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}}|anchor=#function-library-lib.meta.lowPrio}} can be used to install multiple versions without conflicts ({{issue|369997}}):
{{file|/etc/nixos/configuration.nix|nix|3=
environment.systemPackages = with pkgs; [
    python313
    lib.meta.lowPrio python314
];
}}
In this case you may run Python 3.13 via {{ic|python}} or {{ic|python3.13}} and Python 3.14 via {{ic|python3.14}}.
{{Note|If you wrapped Python as described in [[#Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix]], call {{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}} for the wrapped Python instead of calling them inside the wrapping.}}


== Performance ==
== Performance ==
Retrieved from "https://wiki.nixos.org/wiki/Python"