Python: Difference between revisions

Python (view source)

Revision as of 13:53, 15 June 2024

3,295 bytes added , 15 June

Poetry fix was not working... it should work now :)

VisualWikitext

N42

3

edits

@@ Line 177: / Line 177: @@
 </syntaxhighlight>
-=== Using <code>venv</code> ===
+=== Running compiled libraries ===
+If you want to run some compiled libraries as for example <code>grpcio</code><ref>https://pypi.org/project/grpcio/</ref>, you may encounter the following error :<syntaxhighlight lang="shell-session">
+$ python -c 'import grpc'
+Traceback (most recent call last):
+  File "<string>", line 1, in <module>
+  File "/.../grpc/__init__.py", line 22, in <module>
+    from grpc import _compression
+  File "/.../grpc/_compression.py", line 20, in <module>
+    from grpc._cython import cygrpc
+ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
-To create a Python virtual environment with <code>venv</code>:<syntaxhighlight lang="console">
+</syntaxhighlight>This means that the library use compiled dynamically linked binaries that your NixOs environment fail to resolve.
-$ nix-shell -p python3 --command "python -m venv .venv --copies"
-</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
-==== On NixOS ====
-This method may not work on NixOS, as installing packages with <code>pip</code> that need to compile code or use C libraries will fail due to not finding dependencies in the expected places.
+On NixOS, installing packages that need to compile code or use C libraries from outside of the <code>nix</code> package manager may fail if dependencies are not found in the expected locations.
 There are multiple ways to make it work:
 * Use [https://github.com/GuillaumeDesforges/fix-python/ fix-python], this is most suited for beginners.
 * Create a FHS user env with <code>buildFHSUserEnv</code>.
-* Setup <code>nix-ld</code> in your NixOS configuration.
+* Setup <code>nix-ld</code><ref name=":0">https://github.com/Mic92/nix-ld</ref> in your NixOS configuration.
+==== Setup nix-ld ====
+nix-ld<ref name=":0" /> allow you to <code>Run unpatched dynamic binaries on NixOS.</code>
+The following configuration automatically fix the dependencies :<syntaxhighlight lang="nixos" line="1">
+let
+  python = pkgs.python311;
+  # https://github.com/NixOS/nixpkgs/blob/c339c066b893e5683830ba870b1ccd3bbea88ece/nixos/modules/programs/nix-ld.nix#L44
+  # > We currently take all libraries from systemd and nix as the default.
+  pythonldlibpath = lib.makeLibraryPath (with pkgs; [
+    zlib
+    zstd
+    stdenv.cc.cc
+    curl
+    openssl
+    attr
+    libssh
+    bzip2
+    libxml2
+    acl
+    libsodium
+    util-linux
+    xz
+    systemd
+  ]);
+  patchedpython = (python.overrideAttrs (
+    previousAttrs: {
+      # Add the nix-ld libraries to the LD_LIBRARY_PATH.
+      # creating a new library path from all desired libraries
+      postInstall = previousAttrs.postInstall + ''
+        mv  "$out/bin/python3.11" "$out/bin/unpatched_python3.11"
+        cat << EOF >> "$out/bin/python3.11"
+        #!/run/current-system/sw/bin/bash
+        export LD_LIBRARY_PATH="${pythonldlibpath}"
+        exec "$out/bin/unpatched_python3.11" "\$@"
+        EOF
+        chmod +x "$out/bin/python3.11"
+      '';
+    }
+  ));
+  # if you want poetry
+  patchedpoetry =  ((pkgs.poetry.override { python3 = patchedpython; }).overrideAttrs (
+    previousAttrs: {
+      # same as above, but for poetry
+      # not that if you dont keep the blank line bellow, it crashes :(
+      postInstall = previousAttrs.postInstall + ''
+        mv "$out/bin/poetry" "$out/bin/unpatched_poetry"
+        cat << EOF >> "$out/bin/poetry"
+        #!/run/current-system/sw/bin/bash
+        export LD_LIBRARY_PATH="${pythonldlibpath}"
+        exec "$out/bin/unpatched_poetry" "\$@"
+        EOF
+        chmod +x "$out/bin/poetry"
+      '';
+    }
+  ));
+in
+{
+  # Some other config...
+  environment.systemPackages = with pkgs; [
+    patchedpython
+    # if you want poetry
+    patchedpoetry
+  ];
+}
+</syntaxhighlight>This configuration set the <code>LD_LIBRARY_PATH</code> environment variable before running python using the <code>overrideAttrs</code><ref>https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs</ref> function to override the <code>postInstall</code> script of cpython <code>mkDerivation</code><ref>https://github.com/NixOS/nixpkgs/blob/24.05/pkgs/development/interpreters/python/cpython/default.nix</ref>.
+After this step, you should be able to install compiled libraries using venv, poetry, conda or other packages managers...
+=== Using <code>venv</code> ===
+To create a Python virtual environment with <code>venv</code>:<syntaxhighlight lang="console">
+$ nix-shell -p python3 --command "python -m venv .venv --copies"
+</syntaxhighlight>You can then activate and use the Python virtual environment as usual and install dependencies with <code>pip</code> and similar.
 === Using poetry ===