Ollama: Difference between revisions
Malteneuss (talk | contribs) m Add link to ollama library |
m →Configuration of GPU acceleration: Document vulkan option |
||
| (5 intermediate revisions by 4 users not shown) | |||
| Line 23: | Line 23: | ||
* "rocm": supported by most modern AMD GPUs | * "rocm": supported by most modern AMD GPUs | ||
* "cuda": supported by most modern NVIDIA GPUs | * "cuda": supported by most modern NVIDIA GPUs | ||
* "vulkan": supported by most modern GPUs on Linux | |||
| Line 94: | Line 95: | ||
== Troubleshooting == | == Troubleshooting == | ||
=== AMD GPU with open source driver === | === AMD GPU with open source driver === | ||
Use the ollama-rocm nix package: | |||
<syntaxhighlight lang="nix"> | |||
environment.systemPackages = [ pkgs.ollama-rocm ]; | |||
</syntaxhighlight> | |||
And make sure the kernel loads the amdgpu driver: | |||
<syntaxhighlight lang="nix"> | |||
boot.initrd.kernelModules = [ "amdgpu" ]; | |||
</syntaxhighlight> | |||
In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible. | In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible. | ||
However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu. | However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.mdx#overrides-on-linux</ref> | ||
You can get the version for your GPU from the logs or like so: | You can get the version for your GPU from the logs or like so: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
# classical | |||
$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" | grep "gfx" | $ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" | grep "gfx" | ||
Name: gfx1031 | |||
# flakes | |||
$ nix run nixpkgs#"rocmPackages.rocminfo" -- --run "rocminfo" | grep "gfx" | |||
Name: gfx1031 | Name: gfx1031 | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 114: | Line 132: | ||
HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore | HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore | ||
}; | }; | ||
# results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3. | # results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3.0" | ||
rocmOverrideGfx = "10.3. | rocmOverrideGfx = "10.3.0"; | ||
}; | }; | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 121: | Line 139: | ||
or via an environment variable in front of the standalone app | or via an environment variable in front of the standalone app | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
HSA_OVERRIDE_GFX_VERSION=10.3. | HSA_OVERRIDE_GFX_VERSION=10.3.0 ollama serve | ||
</syntaxhighlight> | </syntaxhighlight> | ||