Python: Difference between revisions

← Older edit
VisualWikitext
m Poetry doesn't have an input named `python` https://github.com/NixOS/nixpkgs/blob/7c503a8a9d182df45a4e4fc2f836bcc7316fed2c/pkgs/tools/package-management/poetry/default.nix#L1
Include mention of poetry2nix
 
(16 intermediate revisions by 8 users not shown)
Line 16: Line 16:
in pkgs.mkShell {
in pkgs.mkShell {
   packages = [
   packages = [
     (pkgs.python3.withPackages (python-pkgs: [
     (pkgs.python3.withPackages (python-pkgs: with python-pkgs; [
       # select Python packages here
       # select Python packages here
       python-pkgs.pandas
       pandas
       python-pkgs.requests
       requests
     ]))
     ]))
   ];
   ];
Line 64: Line 64:
   ];
   ];
}
}
</syntaxhighlight>The tool [https://github.com/nix-community/pip2nix pip2nix] can help you generate such files.
</syntaxhighlight>Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
 
Given the file above is named <code>toolz.nix</code> and is the same directory as the previous <code>shell.nix</code> , you can edit <code>shell.nix</code> to use the package <code>toolz</code> above like so: <syntaxhighlight lang="nix" line="1">
# shell.nix
# shell.nix
let
let
   pkgs = import <nixpkgs> {};
   pkgs = import <nixpkgs> {};
  python = pkgs.python3.override {
    self = python;
    packageOverrides = pyfinal: pyprev: {
      toolz = pyfinal.callPackage ./toolz.nix { };
    };
  };
in pkgs.mkShell {
in pkgs.mkShell {
   packages = [
   packages = [
     (pkgs.python3.withPackages (python-pkgs: [
     (python.withPackages (python-pkgs: [
       # select Python packages here
       # select Python packages here
       python-pkgs.pandas
       python-pkgs.pandas
       python-pkgs.requests
       python-pkgs.requests
       (pkgs.callPackage ./toolz.nix)
       python-pkgs.toolz
     ]))
     ]))
   ];
   ];
}
}
</syntaxhighlight>Note that the parenthesis (line 10) are required.
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
=== Running compiled libraries ===
If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/.../grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory


Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.
==== Using nix-shell alongside pip  ====
When working on a collaborative python project you may want to be able to <code>pip install -r requirements.txt</code> if the project isn't packaged for nix specifically.


The problem is that a lot of python packages won't work out of the box when you pip install them.
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations.
To fix this issue, you can create a nix shell that will use <code>pip</code> for the packages that are installing properly, but will fix the environment for the packages that are causing issues.
There are multiple ways to make it work:
* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
 
* Create a FHS user env with <code>buildFHSUserEnv</code>.
* Setup <code>nix-ld</code><ref name=":0">https://github.com/Mic92/nix-ld</ref> in your NixOS configuration.
* Prefix library paths using wrapProgram utility.


You can accomplish this by adding these two files to the root of your project:
==== Setup nix-ld ====
<syntaxhighlight lang="nix" line="1">
nix-ld<ref name=":0" /> allow you to run unpatched dynamic binaries on NixOS.
# shell.nix
{ pkgs ? import <nixpkgs> {} }:


The following configuration automatically fix the dependencies:<syntaxhighlight lang="nixos" line="1">
let
let
   myPython = pkgs.python311;
   python = pkgs.python311;
   pythonPackages = pkgs.python311Packages;
   # We currently take all libraries from systemd and nix as the default
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
    zlib
    zstd
    stdenv.cc.cc
    curl
    openssl
    attr
    libssh
    bzip2
    libxml2
    acl
    libsodium
    util-linux
    xz
    systemd
  ]);
  patchedpython = (python.overrideAttrs (
    previousAttrs: {
      # Add the nix-ld libraries to the LD_LIBRARY_PATH.
      # creating a new library path from all desired libraries
      postInstall = previousAttrs.postInstall + ''
        mv  "$out/bin/python3.11" "$out/bin/unpatched_python3.11"
        cat << EOF >> "$out/bin/python3.11"
        #!/run/current-system/sw/bin/bash
        export LD_LIBRARY_PATH="${pythonldlibpath}"
        exec "$out/bin/unpatched_python3.11" "\$@"
        EOF
        chmod +x "$out/bin/python3.11"
      '';
    }
  ));
  # if you want poetry
  patchedpoetry = ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs (
    previousAttrs: {
      # same as above, but for poetry
      # not that if you dont keep the blank line bellow, it crashes :(
      postInstall = previousAttrs.postInstall + ''


  pythonWithPkgs = myPython.withPackages (pythonPkgs: with pythonPkgs; [
        mv "$out/bin/poetry" "$out/bin/unpatched_poetry"
    # This list contains tools for Python development.
        cat << EOF >> "$out/bin/poetry"
    # You can also add other tools, like black.
        #!/run/current-system/sw/bin/bash
    #
        export LD_LIBRARY_PATH="${pythonldlibpath}"
    # Note that even if you add Python packages here like PyTorch or Tensorflow,
        exec "$out/bin/unpatched_poetry" "\$@"
    # they will be reinstalled when running `pip -r requirements.txt` because
        EOF
    # virtualenv is used below in the shellHook.
        chmod +x "$out/bin/poetry"
     ipython
      '';
    pip
     }
    setuptools
  ));
    virtualenvwrapper
in
    wheel
{
    black
  # Some other config...
   ]);
 
   environment.systemPackages = with pkgs; [
    patchedpython


  extraBuildInputs = with pkgs; [
     # if you want poetry
     # this list contains packages that you want to be available at runtime and might not be able to be installed properly via pip
     patchedpoetry
     # pythonPackages.pandas
    # pythonPackages.requests
   ];
   ];
in
import ./python-shell.nix {
    extraBuildInputs=extraBuildInputs;
    extraLibPackages=extraLibPackages;
    myPython=myPython;
    pythonWithPkgs=pythonWithPkgs;
}
}
</syntaxhighlight>
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
 
<syntaxhighlight lang="nix" line="1">
# python-shell.nix
{ pkgs ? import <nixpkgs> {}, extraBuildInputs ? [], myPython ? pkgs.python3, extraLibPackages ? [], pythonWithPkgs? myPython }:


After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...


==== Prefix library paths using wrapProgram ====
wrapProgram is a part of the makeWrapper build input<ref>https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/make-wrapper.sh</ref>. By combining it with the symlinkJoin, we can create a wrapper around the Python executable that will always set the required library paths. It’s worth noting that, for this solution to be compatible with Darwin, we need to use a different wrap prefix, as shown in the example below.<syntaxhighlight lang="nixos" line="1">
let
let
 
  # We currently take all libraries from systemd and nix as the default
    buildInputs  = with pkgs; [
  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
        clang
   pythonldlibpath = lib.makeLibraryPath (with pkgs; [
        llvmPackages_16.bintools
    zlib
        rustup
    zstd
    ] ++ extraBuildInputs;
    stdenv.cc.cc
 
    curl
   lib-path = with pkgs; lib.makeLibraryPath buildInputs;
    openssl
 
    attr
 
    libssh
shell = pkgs.mkShell {
    bzip2
  buildInputs = [
    libxml2
    # my python and packages
    acl
      pythonWithPkgs
    libsodium
     
    util-linux
      # other packages needed for compiling python libs
    xz
      pkgs.readline
    systemd
      pkgs.libffi
   ]);
      pkgs.openssl
   # Darwin requires a different library path prefix
 
  wrapPrefix = if (!pkgs.stdenv.isDarwin) then "LD_LIBRARY_PATH" else "DYLD_LIBRARY_PATH";
      # unfortunately needed because of messing with LD_LIBRARY_PATH below
  patchedpython = (pkgs.symlinkJoin {
      pkgs.git
    name = "python";
      pkgs.openssh
    paths = [ pkgs.python312 ];
      pkgs.rsync
    buildInputs = [ pkgs.makeWrapper ];
   ] ++ extraBuildInputs;
    postBuild = ''
   shellHook = ''
       wrapProgram "$out/bin/python3.12" --prefix ${wrapPrefix} : "${pythonldlibpath}"
      # Allow the use of wheels.
    '';
      SOURCE_DATE_EPOCH=$(date +%s)
  });
      # Augment the dynamic linker path
      export "LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${lib-path}"
      # Setup the virtual environment if it doesn't already exist.
      VENV=.venv
      if test ! -d $VENV; then
        virtualenv $VENV
       fi
      source ./$VENV/bin/activate
      export PYTHONPATH=$PYTHONPATH:`pwd`/$VENV/${myPython.sitePackages}/
  '';
};
 
in
in
shell
{
  environment.systemPackages = with pkgs; [
    patchedpython
  ];
}
</syntaxhighlight>
</syntaxhighlight>
=== Using <code>venv</code> ===
=== Using <code>venv</code> ===


Line 183: Line 223:
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.


==== On NixOS ====
=== Using uv ===
This method may not work on NixOS, as installing packages with <code>pip</code> that need to compile code or use C libraries will fail due to not finding dependencies in the expected places.
<blockquote>A single tool to replace <code>pip</code>, <code>pip-tools</code>, <code>pipx</code>, <code>poetry</code>, <code>pyenv</code>, <code>virtualenv</code>, and more.</blockquote>uv is very simple to use. Simply <code>uv init</code> to get started. No need for shells, as it creates virtual environments.


There are multiple ways to make it work:
As a systemPackage<syntaxhighlight lang="nix">
 
environment.systemPackages = with pkgs; [
* Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
    uv
 
];
* Create a FHS user env with <code>buildFHSUserEnv</code>.
</syntaxhighlight>or as a home-manager package<syntaxhighlight lang="nix">
* Setup <code>nix-ld</code> in your NixOS configuration.
home.packages = with pkgs; [
    uv
];
</syntaxhighlight>


=== Using poetry ===
=== Using poetry ===
Line 207: Line 250:
</syntaxhighlight>
</syntaxhighlight>


==== poetry2nix ====
[https://github.com/nix-community/poetry2nix poetry2nix] uses the contents of a <code>poetry.lock</code> and <code>pyproject.toml</code> file to create Nix derivations. It has several functions for generating development environments and python projects. Because some older python projects rely on deprecated build systems (see [https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md edgecase.md] for more info), poetry2nix provides overrides so these packages can still be built.
=== Using micromamba ===
=== Using micromamba ===


Line 264: Line 309:


== Package a Python application ==
== Package a Python application ==
It is possible to use <code>buildPythonApplication</code> to package python applications. As explained in the nixpkgs manual, it uses the widely used <code>setup.py</code> file in order to package properly the application. We now show how to package a simple python application: a basic flask web server.


First, we write the python code, say in a file <code>web_interface.py</code>. Here we create a basic flask web server:
=== With <code>setup.py</code> ===
To package a Python application that uses <code>setup.py</code> you can use <code>buildPythonApplication</code>. More details about this and similar functions can be found in [https://nixos.org/manual/nixpkgs/stable/#building-packages-and-applications the nixpkgs manual].
 
For example, we can package this simple flask server <code>main.py:</code>
<syntaxhighlight lang="python">
<syntaxhighlight lang="python">
#!/usr/bin/env python
#!/usr/bin/env python


from flask import Flask
from flask import Flask
app = Flask(__name__)
app = Flask(__name__)


Line 281: Line 329:
</syntaxhighlight>
</syntaxhighlight>


Then, we create the <code>setup.py</code> file, which basically explains which are the executables:
We also need a <code>setup.py</code> file, like this:
<syntaxhighlight lang="python">
<syntaxhighlight lang="python">
#!/usr/bin/env python
from setuptools import setup, find_packages
from setuptools import setup, find_packages


setup(name='demo-flask-vuejs-rest',
setup(name='myFlaskServer',
       version='1.0',
       version='1.0',
       # Modules to import from other scripts:
       # Modules to import from other scripts:
       packages=find_packages(),
       packages=find_packages(),
       # Executables
       # Executables
       scripts=["web_interface.py"],
       scripts=["main.py"],
     )
     )
</syntaxhighlight>
</syntaxhighlight>


Finally, our nix derivation is now trivial: the file <code>derivation.nix</code> just needs to provide the python packages (here flask):
Then, we use the <code>buildPythonApplication</code> in the <code>default.nix</code>:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ lib, python3Packages }:
{ pkgs ? import <nixpkgs> {} }:
with python3Packages;
buildPythonApplication {
  pname = "demo-flask-vuejs-rest";
  version = "1.0";


   propagatedBuildInputs = [ flask ];
pkgs.python3Packages.buildPythonApplication {
  pname = "myFlaskApp";
  version = "0.1.0";
 
   propagatedBuildInputs = with pkgs.python3Packages; [
    flask
  ];


   src = ./.;
   src = ./.;
}
}
</syntaxhighlight>
</syntaxhighlight>Finally, build your project using <code>nix-build</code>. The result will be executable in <code>./result/bin/app.py</code>.
 
=== With <code>pyproject.toml</code> ===
When your project is using <code>pyproject.toml</code>you can use [https://github.com/nix-community/pyproject.nix pyproject.nix] to package your application.
 
First, a simple file structure could look like this:<syntaxhighlight>
├── app/
    └── main.py
├── flake.nix
├── pyproject.toml
└── README.md
 
 
</syntaxhighlight>To reuse the example from above, we use the same flask application:<syntaxhighlight lang="python">
from flask import Flask
 
app = Flask(__name__)
 
@app.route('/')
def hello_world():
    return 'Hello, World!'
 
if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)
</syntaxhighlight>Also, you need to define the <code>pyproject.toml</code>. Here, we only show some of the important parts. Please refer to <code>pyproject.nix</code> [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation] for a full example.<syntaxhighlight lang="toml">
[project]
name = "my-app"
version = "0.1.0"
description = "Simple app"
 
# define any Python dependencies
dependencies = [
  "flask>3",
]
 
# define the CLI executable
# Here, we define the entry point to be the 'main()' function in the module 'app/main.py'
[project.scripts]
cli = "app.main:main"
</syntaxhighlight>We package the application by calling the <code>loadPyproject</code> function from <code>pyproject.nix</code>. Again, we only show a minimal example. More information can be found in the [https://nix-community.github.io/pyproject.nix/use-cases/pyproject.html documentation].<syntaxhighlight lang="nix">
{
  description = "A basic flake using pyproject.toml project metadata";
 
  inputs = {
    pyproject-nix = {
      url = "github:nix-community/pyproject.nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
 
  outputs = { nixpkgs, pyproject-nix, ... }:
    let
      inherit (nixpkgs) lib;
 
      project = pyproject-nix.lib.project.loadPyproject {
        # Read & unmarshal pyproject.toml relative to this project root.
        # projectRoot is also used to set `src` for renderers such as buildPythonPackage.
        projectRoot = ./.;
      };
 
      # This example is only using x86_64-linux
      pkgs = nixpkgs.legacyPackages.x86_64-linux;


and we can now load this derivation from our file <code>default.nix</code>:
      python = pkgs.python3;
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}
</syntaxhighlight>


We can now build with:
    in
<syntaxhighlight lang=console>
    {
$ nix-build
      # Build our package using `buildPythonPackage
[...]
      packages.x86_64-linux.default =
$ ./result/bin/web_interface.py
        let
* Serving Flask app ".web_interface" (lazy loading)
          # Returns an attribute set that can be passed to `buildPythonPackage`.
[...]
          attrs = project.renderers.buildPythonPackage { inherit python; };
</syntaxhighlight>
        in
or just enter a nix-shell, and directly execute your program or python if it's easier to develop:
        # Pass attributes to buildPythonPackage.
<syntaxhighlight lang=console>
        # Here is a good spot to add on any missing or custom attributes.
$ nix-shell
        python.pkgs.buildPythonPackage (attrs // {
[...]
          env.CUSTOM_ENVVAR = "hello";
[nix-shell]$ chmod +x web_interface.py
        });
[nix-shell]$ ./web_interface.py
    };
* Serving Flask app "web_interface" (lazy loading)
}
[...]


[nix-shell]$ python
</syntaxhighlight>To run the application, call <code>nix run</code>.
Python 3.8.7 (default, Dec 21 2020, 17:18:55)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>
</syntaxhighlight>


== Nixpkgs Python contribution guidelines ==
== Nixpkgs Python contribution guidelines ==
Line 432: Line 530:
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
* [https://nixos.org/manual/nixpkgs/unstable/#python "Python" in Nixpkgs Manual]
[[Category:Languages]]
[[Category:Languages]]
[[Category:Python]]
Retrieved from "https://wiki.nixos.org/wiki/Python"