NixOS Wiki

Python: Difference between revisions

← Older editNewer edit →
VisualWikitext
34j (talk | contribs)
71 edits
Shorten patch codes as they are looking too long
34j (talk | contribs)
71 edits
Add nix-ld and fix-python
Line 89: Line 89:
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.
</syntaxhighlight>Next time you enter the shell specified by this file, Nix will build and include the Python package you have written.


=== Running libraries precompiled without <code>nix</code> ===
=== Running Python packages which requires compilation and/or contains libraries precompiled without <code>nix</code> ===


If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
If you want to use some Python packages containing libraries precompiled without <code>nix</code> as for example <code>[https://pypi.org/project/grpcio/ grpcio]</code> or <code>[https://pypi.org/project/numpy Numpy]</code>, you may encounter the following error :<syntaxhighlight lang="shell-session">
$ python -c 'import grpc'
$ python -c 'import grpc'
Traceback (most recent call last):
Traceback (most recent call last):
Line 101: Line 101:
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory


</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.
</syntaxhighlight>This means that the Python package depends on compiled dynamically linked binaries that your NixOs environment fail to resolve.


On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to make it work.
On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations. There are multiple ways to solve this, and most of them are just different ways for adding <code>/nix/store/...</code> to <code>$LD_LIBRARY_PATH</code>.
 
After doing one of the following, you should be able to install compiled libraries using <code>venv</code>, <code>poetry</code>, <code>uv</code>, <code>conda</code> or other packages managers.


==== Using nix overlay ====
==== Using nix overlay ====
Line 139: Line 141:
     }
     }
   )];
   )];
   environment.systemPackages = with pkgs; [
   environment.systemPackages = with pkgs; [
     python  # here python will be taken from the overlay up here
     python  # here python will be taken from the overlay up here
Line 147: Line 148:
</syntaxhighlight>
</syntaxhighlight>


==== Using `buildFHSEnv` in nix-shell ====
==== Using [https://github.com/Mic92/nix-ld nix-ld] ====
 
<syntaxhighlight lang="nix">
{
  programs.nix-ld = {
    enable = true;
    libraries = with pkgs; [
      zlib zstd stdenv.cc.cc curl openssl attr libssh bzip2 libxml2 acl libsodium util-linux xz systemd
    ];
  };
  # https://github.com/nix-community/nix-ld?tab=readme-ov-file#my-pythonnodejsrubyinterpreter-libraries-do-not-find-the-libraries-configured-by-nix-ld
  environment.systemPackages = [
    (pkgs.writeShellScriptBin "python" ''
      export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
      exec ${pkgs.python3}/bin/python "$@"
    '')
  ];
  # another (dangerous) solution
  # environment.systemPackages = with pkgs; [ python3 ];
  # programs.bash = {
  #  enable = true;
  #  initExtra = ''
  #    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH''${LD_LIBRARY_PATH:+:}$NIX_LD_LIBRARY_PATH
  #  '';
  # };
}
</syntaxhighlight>
 
==== Using [https://github.com/GuillaumeDesforges/fix-python/ fix-python] ====
{{Note|fix-python will <strong>patch</strong> binary files in the virtual environment without following symlinks.}}
 
Install with:
 
<syntaxhighlight lang="shell">
nix profile install github:GuillaumeDesforges/fix-python
</syntaxhighlight>
 
Enter the venv and run:
 
<syntaxhighlight lang="shell">
fix-python --venv .venv
</syntaxhighlight>
 
==== Using <code>buildFHSEnv</code> ====
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
#!/usr/bin/env nix-shell
#!/usr/bin/env nix-shell
Line 163: Line 207:
).env
).env
</syntaxhighlight>
</syntaxhighlight>
==== Using 3rd party tools ====
* [https://github.com/Mic92/nix-ld nix-ld] allows you to run unpatched dynamic binaries on NixOS.
* [https://github.com/GuillaumeDesforges/fix-python/ fix-python]


==== Using a custom nix-shell ====
==== Using a custom nix-shell ====
Line 220: Line 259:
}
}
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...


==== Prefix library paths using wrapProgram ====
==== Prefix library paths using wrapProgram ====
Line 248: Line 285:
}
}
</syntaxhighlight>
</syntaxhighlight>
=== Using <code>venv</code> ===
=== Using <code>venv</code> ===


Retrieved from "https://wiki.nixos.org/wiki/Python"