NixOS Wiki

Python: Difference between revisions

← Older edit
VisualWikitext
34j (talk | contribs)
71 edits
Shorten patch codes as they are looking too long
Lukeb (talk | contribs)
1 edit
fixed broken link for pyproject use-cases
 
(11 intermediate revisions by 7 users not shown)
Line 30: Line 30:


Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Once you have picked the Python packages you want, run <code>nix-shell</code> (or <code>nix develop -f shell.nix</code>)  to build the Python environment and enter it. Once in the environment Python will be available in your PATH, so you can run eg. <code>python --version</code>.
Note that with NixOS, this method can be used to install packages at the system level, e.g. <syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
  # ...
  (python3.withPackages (python-pkgs: with python-pkgs; [
      pandas
      requests
  ]))
];
</syntaxhighlight>
=== Using Nix shell (new command line) ===
nix shell --impure --expr '(import <nixpkgs> {}).python3.withPackages (ps: with ps; [ swh-core swh-scanner ])'
If you don't use the channels any more, you can replace <code><nixpkgs></code> by an instance of the <code>NixOS/nixpkgs</code> repository using its absolute path.


==== Using a Python package not in Nixpkgs ====
==== Using a Python package not in Nixpkgs ====
Line 89: Line 103:
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.


=== Running libraries precompiled without <code>nix</code> ===
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> ===


If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
$ python -c 'import grpc'
Traceback (most recent call last):
Traceback (most recent call last):
Line 101: Line 115:
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory


</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.


On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to make it work.
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>.
 
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers.


==== Using nix overlay ====
==== Using nix overlay ====
Line 139: Line 155:
     }
     }
   )];
   )];
   environment.systemPackages = with pkgs; [
   environment.systemPackages = with pkgs; [
     python  # here python will be taken from the overlay up here
     python  # here python will be taken from the overlay up here
Line 147: Line 162:
</syntaxhighlight>
</syntaxhighlight>


==== Using `buildFHSEnv` in nix-shell ====
==== Using [https://github.com/Mic92/nix-ld nix-ld] ====
 
<syntaxhighlight lang="nix">
{
  programs.nix-ld = {
    enable = true;
    libraries = with pkgs; [
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    ];
  };
  # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
    (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
    '')
  ];
  # another (dangerous) solution
  # environment.systemPackages = with pkgs; [ python3 ];
  # programs.bash = {
  #  enable = true;
  #  initExtra = ''
  #    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #  '';
  # };
}
</syntaxhighlight>
 
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ====
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}}
 
Install with:
 
<syntaxhighlight lang="shell">
nix profile install github:GuillaumeDesforges/fix-python
</syntaxhighlight>
 
Enter the venv and run:
 
<syntaxhighlight lang="shell">
fix-python --venv .venv
</syntaxhighlight>
 
==== Using <code>buildFHSEnv</code> (Recommended) ====
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
#!/usr/bin/env nix-shell
#!/usr/bin/env nix-shell
Line 163: Line 221:
).env
).env
</syntaxhighlight>
</syntaxhighlight>
==== Using 3rd party tools ====
* [https://github.com/Mic92/nix-ld nix-ld] allows you to run unpatched dynamic binaries on NixOS.
* [https://github.com/GuillaumeDesforges/fix-python/ fix-python]


==== Using a custom nix-shell ====
==== Using a custom nix-shell ====
Line 220: Line 273:
}
}
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...


==== Prefix library paths using wrapProgram ====
==== Prefix library paths using wrapProgram ====
Line 248: Line 299:
}
}
</syntaxhighlight>
</syntaxhighlight>
=== Using <code>venv</code> ===
=== Using <code>venv</code> ===


Line 255: Line 307:


=== Using uv ===
=== Using uv ===
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>uv is very simple to use. Simply <code>uv init</code> to get started. No need for shells, as it creates virtual environments.
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>


As a systemPackage<syntaxhighlight lang="nix">
uv is written in Rust and does ''not'' need Python as a prerequisite. Use the <code>uv</code> command to initialize Python projects, add Python packages, create or update virtual environments (in <code>.venv</code> folders), etc. as [https://docs.astral.sh/uv/concepts/projects/ described in the uv docs]. Use uv's [https://docs.astral.sh/uv/guides/tools/ tool interface] to install and run Python packages that provide a CLI.
 
As a system package
 
<syntaxhighlight lang="nix">
environment.systemPackages = with pkgs; [
environment.systemPackages = with pkgs; [
     uv
     uv
];
];
</syntaxhighlight>or as a home-manager package<syntaxhighlight lang="nix">
</syntaxhighlight>
 
or as a home-manager package
 
<syntaxhighlight lang="nix">
home.packages = with pkgs; [
home.packages = with pkgs; [
     uv
     uv
];
];
</syntaxhighlight>
</syntaxhighlight>
If you use uv it's recommended that you install the Python versions you need using the <code>uv python install</code> command, e.g.
<syntaxhighlight lang="python">
uv python install 3.14 --preview --default
</syntaxhighlight>
You may want to set the <code>UV_PYTHON_DOWNLOADS=never</code> environment variable in your shell to stop uv from downloading Python binaries automatically if needed. Setting <code>environment.localBinInPath = true;</code> is highly recommended, because uv will install binaries in <code>~/.local/bin</code>.


=== Using poetry ===
=== Using poetry ===
Line 289: Line 357:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
To activate an environment you will need a [https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments FHS environment] e.g.:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ micromamba activate my-environment
Line 300: Line 368:
{ pkgs ? import <nixpkgs> {}}:
{ pkgs ? import <nixpkgs> {}}:
let
let
   fhs = pkgs.buildFHSUserEnv {
   fhs = pkgs.buildFHSEnv {
     name = "my-fhs-environment";
     name = "my-fhs-environment";


Line 355: Line 423:
       let
       let
         pkgs = import nixpkgs { inherit system; };
         pkgs = import nixpkgs { inherit system; };
         fhs = pkgs.buildFHSUserEnv {
         fhs = pkgs.buildFHSEnv {
           name = "pixi-env";
           name = "pixi-env";


Line 378: Line 446:
{
{
   home.packages = [
   home.packages = [
     (pkgs.buildFHSUserEnv {
     (pkgs.buildFHSEnv {
       name = "pixi";
       name = "pixi";
       runScript = "pixi";
       runScript = "pixi";
Line 463: Line 531:
if __name__ == '__main__':
if __name__ == '__main__':
     app.run(host="0.0.0.0", port=8080)
     app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://pyproject-nix.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
[project]
[project]
name = "my-app"
name = "my-app"
Line 578: Line 646:


You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
== Debug Build ==
See [https://docs.python.org/3/using/configure.html#python-debug-build python wiki on debug build].
In order to use a CPython interpreter built using <code>--with-pydebug</code> during configure phase, override any of the python packages passing <code>enableDebug = true</code> argument:
<code>pythonDebug = pkgs.python310.override { enableDebug = true; };</code>
== Installing Multiple Versions ==
{{Nixpkgs Manual|name={{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}}|anchor=#function-library-lib.meta.lowPrio}} can be used to install multiple versions without conflicts ({{issue|369997}}):
{{file|/etc/nixos/configuration.nix|nix|3=
environment.systemPackages = with pkgs; [
    python313
    lib.meta.lowPrio python314
];
}}
In this case you may run Python 3.13 via {{ic|python}} or {{ic|python3.13}} and Python 3.14 via {{ic|python3.14}}.
{{Note|If you wrapped Python as described in [[#Running_Python_packages_which_requires_compilation_and/or_contains_libraries_precompiled_without_nix]], call {{ic|lib.meta.lowPrio}} and {{ic|lib.meta.highPrio}} for the wrapped Python instead of calling them inside the wrapping.}}


== Performance ==
== Performance ==
Retrieved from "https://wiki.nixos.org/wiki/Python"