Ollama: Difference between revisions

From NixOS Wiki
Klinger (talk | contribs)
Add troubleshooting instructions for AMD GPUs
Line 18: Line 18:
ollama run mistral
ollama run mistral
</syntaxhighlight>For ruther models see [https://ollama.ai/library Ollama library].
</syntaxhighlight>For ruther models see [https://ollama.ai/library Ollama library].
== Troubleshooting ==
=== AMD GPU with open source driver ===
In certain cases ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.
However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides</ref>
You can get the version for your GPU from the logs or like so:
<syntaxhighlight lang="bash">
$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" | grep "gfx"
Name:                    gfx1031
</syntaxhighlight>
In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for ollama:
<syntaxhighlight lang="nix">
services.ollama = {
  enable = true;
  acceleration = "rocm";
};
systemd.services.ollama.environment = {
  HSA_OVERRIDE_GFX_VERSION = "10.3.1";
};
</syntaxhighlight>
If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here].


[[Category:Server]]
[[Category:Server]]

Revision as of 18:18, 25 May 2024

Ollama is an open-source framework designed to facilitate the deployment of large language models on local environments. It aims to simplify the complexities involved in running and managing these models, providing a seamless experience for users across different operating systems.

Setup

Add following line to your system configuration

services.ollama.enable = true;

Configuration

Enable GPU acceleration for Nvidia graphic cards

services.ollama = {
  enable = true;
  acceleration = "cuda";
};

Usage

Download and run Mistral LLM model as an interactive prompt

ollama run mistral

For ruther models see Ollama library.

Troubleshooting

AMD GPU with open source driver

In certain cases ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.

However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. [1]

You can get the version for your GPU from the logs or like so:

$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" | grep "gfx"
Name:                    gfx1031

In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for ollama:

services.ollama = {
  enable = true;
  acceleration = "rocm";
};
systemd.services.ollama.environment = {
  HSA_OVERRIDE_GFX_VERSION = "10.3.1";
};

If there are still errors, you can attempt to set a similar value that is listed here.