Ollama: Difference between revisions

Line 87:

== Usage via web API ==

Other software can use the web API (default at: http://localhost:11434 ) to query ~~ollama~~. This works well e.g. in Intellij-IDEs with the "ProxyAI" and the "Ollama Commit Summarizer" plugins.

Other software can use the web API (default at: http://localhost:11434 ) to query Ollama. This works well e.g. in Intellij-IDEs with the "ProxyAI" and the "Ollama Commit Summarizer" plugins.

Alternatively, on enabling "open-webui", a web portal is available at: http://localhost:8080/:

Line 95:

=== AMD GPU with open source driver ===

In certain cases ~~ollama~~ might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.

In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.

However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides</ref>

Line 106:

</syntaxhighlight>

In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for ~~ollama~~:

In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for Ollama for the systemd service:

services.ollama = {

Line 114:

HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore

};

# results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3.1"

rocmOverrideGfx = "10.3.1";

};

</syntaxhighlight>

or via an environment variable in front of the standalone app

HSA_OVERRIDE_GFX_VERSION=10.3.1 ollama serve

</syntaxhighlight>

If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here].

@@ Line 87: / Line 87: @@
 == Usage via web API ==
-Other software can use the web API (default at: http://localhost:11434 ) to query ollama. This works well e.g. in Intellij-IDEs with the "ProxyAI" and the "Ollama Commit Summarizer" plugins.
+Other software can use the web API (default at: http://localhost:11434 ) to query Ollama. This works well e.g. in Intellij-IDEs with the "ProxyAI" and the "Ollama Commit Summarizer" plugins.
 Alternatively, on enabling "open-webui", a web portal is available at: http://localhost:8080/:
@@ Line 95: / Line 95: @@
 === AMD GPU with open source driver ===
-In certain cases ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.
+In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.
 However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides</ref>
@@ Line 106: / Line 106: @@
 </syntaxhighlight>
-In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for ollama:
+In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for Ollama for the systemd service:
 <syntaxhighlight lang="nix">
 services.ollama = {
@@ Line 114: / Line 114: @@
      HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore
    };
+  # results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3.1"
    rocmOverrideGfx = "10.3.1";
 };
 </syntaxhighlight>
+or via an environment variable in front of the standalone app
+<syntaxhighlight lang="bash">
+HSA_OVERRIDE_GFX_VERSION=10.3.1 ollama serve
+</syntaxhighlight>
 If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here].