Ollama: Difference between revisions
mNo edit summary |
Mathtician (talk | contribs) Updated syntax for systemd service config (acceleration -> package); have not tested standalone package config |
||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 19: | Line 19: | ||
== Configuration of GPU acceleration == | == Configuration of GPU acceleration == | ||
Acceleration is configured by selecting a package: | |||
* | * ollama-cpu: disable GPU, only use CPU | ||
* | * ollama-rocm: supported by most modern AMD GPUs | ||
* | * ollama-cuda: supported by most modern NVIDIA GPUs | ||
* ollama-vulkan: supported by most modern GPUs on Linux | |||
| Line 40: | Line 41: | ||
services.ollama = { | services.ollama = { | ||
enable = true; | enable = true; | ||
package = pkgs.ollama-cuda; | |||
}; | }; | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 109: | Line 110: | ||
In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible. | In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible. | ||
However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu. | However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.mdx#overrides-on-linux</ref> | ||
You can get the version for your GPU from the logs or like so: | You can get the version for your GPU from the logs or like so: | ||
| Line 127: | Line 128: | ||
services.ollama = { | services.ollama = { | ||
enable = true; | enable = true; | ||
package = pkgs.ollama-rocm; | |||
environmentVariables = { | environmentVariables = { | ||
HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore | HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore | ||