NixOS Wiki

Python: Difference between revisions

← Older edit
VisualWikitext
Johtok (talk | contribs)
1 edit
Described and linked solutions to use pixi environment manager with nixos
Lukeb (talk | contribs)
1 edit
fixed broken link for pyproject use-cases
 
(15 intermediate revisions by 9 users not shown)
Line 30: Line 30:


Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Note that with NixOS, this method can be used to install packages at the system level, e.g. <syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
  # ...
  (python3.withPackages (python-pkgs: with python-pkgs; [
      pandas
      requests
  ]))
];
</syntaxhighlight>
=== Using Nix shell (new command line) ===
nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])'
If you don't use the channels any more, you can replace <code><nixpkgs></code> by an instance of the <code>NixOS/nixpkgs</code> repository using its absolute path.


==== Using a Python package not in Nixpkgs ====
==== Using a Python package not in Nixpkgs ====
Line 88: Line 102:
}
}
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
=== Running compiled libraries ===
 
If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> ===
 
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
$ python -c 'import grpc'
Traceback (most recent call last):
Traceback (most recent call last):
Line 99: Line 115:
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory


</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.
 
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>.
 
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers.
 
==== Using nix overlay ====
<syntaxhighlight lang="nix">
{
  nixpkgs.overlays = [(
    self: super: rec {
      # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
      # > We currently take all libraries from systemd and nix as the default.
      pythonldlibpath = lib.makeLibraryPath (with super; [
        zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
      ]);
      # here we are overriding python program to add LD_LIBRARY_PATH to it's env
      python = super.stdenv.mkDerivation {
        name = "python";
        buildInputs = [ super.makeWrapper ];
        src = super.python311;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/python3 --set LD_LIBRARY_PATH ${pythonldlibpath}
          wrapProgram $out/bin/python3.11 --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
      poetry = super.stdenv.mkDerivation {
        name = "poetry";
        buildInputs = [ super.makeWrapper ];
        src = super.poetry;
        installPhase = ''
          mkdir -p $out/bin
          cp -r $src/* $out/
          wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${pythonldlibpath}
        '';
      };
    }
  )];
  environment.systemPackages = with pkgs; [
    python  # here python will be taken from the overlay up here
    poetry
  ];
}
</syntaxhighlight>
 
==== Using [https://github.com/Mic92/nix-ld nix-ld] ====
 
<syntaxhighlight lang="nix">
{
  programs.nix-ld = {
    enable = true;
    libraries = with pkgs; [
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    ];
  };
  # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
    (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
    '')
  ];
  # another (dangerous) solution
  # environment.systemPackages = with pkgs; [ python3 ];
  # programs.bash = {
  #  enable = true;
  #  initExtra = ''
  #    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #  '';
  # };
}
</syntaxhighlight>
 
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ====
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}}
 
Install with:
 
<syntaxhighlight lang="shell">
nix profile install github:GuillaumeDesforges/fix-python
</syntaxhighlight>
 
Enter the venv and run:


On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations.
<syntaxhighlight lang="shell">
There are multiple ways to make it work:
fix-python --venv .venv
* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
</syntaxhighlight>


* Create a FHS user env with <code>buildFHSUserEnv</code>.
==== Using <code>buildFHSEnv</code> (Recommended) ====
* Setup <code>nix-ld</code><ref name=":0">https://github.com/Mic92/nix-ld</ref> in your NixOS configuration.
<syntaxhighlight lang="nix">
* Prefix library paths using wrapProgram utility.
#!/usr/bin/env nix-shell
{ pkgs ? import <nixpkgs> { } }:
(
  let base = pkgs.appimageTools.defaultFhsEnvArgs; in
  pkgs.buildFHSEnv (base // {
    name = "FHS";
    targetPkgs = pkgs: (with pkgs; [
      gcc glibc zlib
    ]);
    runScript = "zsh";
    extraOutputsToInstall = [ "dev" ];
  })
).env
</syntaxhighlight>


==== Setup nix-ld ====
==== Using a custom nix-shell ====
nix-ld<ref name=":0" /> allow you to run unpatched dynamic binaries on NixOS.


The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1">
The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1">
Line 118: Line 230:
   # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
   # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
     zlib
     zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    zstd
    stdenv.cc.cc
    curl
    openssl
    attr
    libssh
    bzip2
    libxml2
    acl
    libsodium
    util-linux
    xz
    systemd
   ]);
   ]);
   patchedpython = (python.overrideAttrs (
   patchedpython = (python.overrideAttrs (
Line 167: Line 266:
in
in
{
{
  # Some other config...
 
   environment.systemPackages = with pkgs; [
   environment.systemPackages = with pkgs; [
     patchedpython
     patchedpython
     # if you want poetry
     # if you want poetry
     patchedpoetry
     patchedpoetry
Line 177: Line 273:
}
}
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...


==== Prefix library paths using wrapProgram ====
==== Prefix library paths using wrapProgram ====
Line 186: Line 280:
   # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
   # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
     zlib
     zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    zstd
    stdenv.cc.cc
    curl
    openssl
    attr
    libssh
    bzip2
    libxml2
    acl
    libsodium
    util-linux
    xz
    systemd
   ]);
   ]);
   # Darwin requires a different library path prefix
   # Darwin requires a different library path prefix
Line 218: Line 299:
}
}
</syntaxhighlight>
</syntaxhighlight>
=== Using <code>venv</code> ===
=== Using <code>venv</code> ===


Line 225: Line 307:


=== Using uv ===
=== Using uv ===
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>uv is very simple to use. Simply <code>uv init</code> to get started. No need for shells, as it creates virtual environments.
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>
 
uv is written in Rust and does ''not'' need Python as a prerequisite. Use the <code>uv</code> command to initialize Python projects, add Python packages, create or update virtual environments (in <code>.venv</code> folders), etc. as [https://docs.astral.sh/uv/concepts/projects/ described in the uv docs]. Use uv's [https://docs.astral.sh/uv/guides/tools/ tool interface] to install and run Python packages that provide a CLI.
 
As a system package


As a systemPackage<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
environment.systemPackages = with pkgs; [
     uv
     uv
];
];
</syntaxhighlight>or as a home-manager package<syntaxhighlight lang="nix">
</syntaxhighlight>
 
or as a home-manager package
 
<syntaxhighlight lang="nix">
home.packages = with pkgs; [
home.packages = with pkgs; [
     uv
     uv
];
];
</syntaxhighlight>
</syntaxhighlight>
If you use uv it's recommended that you install the Python versions you need using the <code>uv python install</code> command, e.g.
<syntaxhighlight lang="python">
uv python install 3.14 --preview --default
</syntaxhighlight>
You may want to set the <code>UV_PYTHON_DOWNLOADS=never</code> environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting <code>environment.localBinInPath = true;</code> is highly recommended, because uv will install binaries in <code>~/.local/bin</code>.


=== Using poetry ===
=== Using poetry ===
Line 259: Line 357:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ micromamba activate my-environment
Line 270: Line 368:
{ pkgs ? import <nixpkgs> {}}:
{ pkgs ? import <nixpkgs> {}}:
let
let
   fhs = pkgs.buildFHSUserEnv {
   fhs = pkgs.buildFHSEnv {
     name = "my-fhs-environment";
     name = "my-fhs-environment";


Line 325: Line 423:
       let
       let
         pkgs = import nixpkgs { inherit system; };
         pkgs = import nixpkgs { inherit system; };
         fhs = pkgs.buildFHSUserEnv {
         fhs = pkgs.buildFHSEnv {
           name = "pixi-env";
           name = "pixi-env";


Line 348: Line 446:
{
{
   home.packages = [
   home.packages = [
     (pkgs.buildFHSUserEnv {
     (pkgs.buildFHSEnv {
       name = "pixi";
       name = "pixi";
       runScript = "pixi";
       runScript = "pixi";
Line 433: Line 531:
if __name__ == '__main__':
if __name__ == '__main__':
     app.run(host="0.0.0.0", port=8080)
     app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://pyproject-nix.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
[project]
[project]
name = "my-app"
name = "my-app"
Line 548: Line 646:


You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
== Debug Build ==
See [https://docs.python.org/3/using/configure.html#python-debug-build python wiki on debug build].
In order to use a CPython interpreter built using <code>--with-pydebug</code> during configure phase, override any of the python packages passing <code>enableDebug = true</code> argument:
<code>pythonDebug = pkgs.python310.override { enableDebug = true; };</code>
== Installing Multiple Versions ==
{{Nixpkgs Manual|name={{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}}|anchor=#function-library-lib.meta.lowPrio}} can be used to install multiple versions without conflicts ({{issue|369997}}):
{{file|/etc/nixos/configuration.nix|nix|3=
environment.systemPackages = with pkgs; [
    python313
    lib.meta.lowPrio python314
];
}}
In this case you may run Python 3.13 via {{ic|python}} or {{ic|python3.13}} and Python 3.14 via {{ic|python3.14}}.
{{Note|If you wrapped Python as described in [[#Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix]], call {{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}} for the wrapped Python instead of calling them inside the wrapping.}}


== Performance ==
== Performance ==
Retrieved from "https://wiki.nixos.org/wiki/Python"