Ollama: Difference between revisions
mNo edit summary |
Improved the configuration and usage for ollama |
||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 6: | Line 6: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Configuration == | == Configuration of GPU acceleration == | ||
Enable GPU acceleration for Nvidia graphic cards<syntaxhighlight lang="nix"> | Its possible to use following values for acceleration: | ||
* false: disable GPU, only use CPU | |||
* "rocm": supported by most modern AMD GPUs | |||
* "cuda": supported by most modern NVIDIA GPUs | |||
Example: Enable GPU acceleration for Nvidia graphic cards<syntaxhighlight lang="nix"> | |||
services.ollama = { | services.ollama = { | ||
enable = true; | enable = true; | ||
| Line 14: | Line 20: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Usage == | == Usage via CLI == | ||
Download and run Mistral LLM model as an interactive prompt<syntaxhighlight lang="bash"> | === Download a model and run interactive prompt === | ||
Example: Download and run Mistral LLM model as an interactive prompt<syntaxhighlight lang="bash"> | |||
ollama run mistral | ollama run mistral | ||
</syntaxhighlight>For | </syntaxhighlight>For other models see [https://ollama.ai/library Ollama library]. | ||
=== Send a prompt to ollama === | |||
Example: To download and run codellama with 13 billion parameters in the "instruct" variant and send a prompt: | |||
<syntaxhighlight lang="bash"> | |||
ollama run codellama:13b-instruct "Write an extended Python program with a typical structure. It should print the numbers 1 to 10 to standard output." | |||
</syntaxhighlight> | |||
== Usage via web API == | |||
Other software can use the web API (default at: http://localhost:11434 ) to query ollama. This works well e.g. in Intellij-IDEs with the CodeGPT and the "Ollama Commit Summarizer" plugins. | |||
== Troubleshooting == | == Troubleshooting == | ||
| Line 39: | Line 55: | ||
acceleration = "rocm"; | acceleration = "rocm"; | ||
environmentVariables = { | environmentVariables = { | ||
HCC_AMDGPU_TARGET = "gfx1031"; | HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore | ||
}; | }; | ||
rocmOverrideGfx = "10.3.1"; | rocmOverrideGfx = "10.3.1"; | ||
| Line 46: | Line 62: | ||
If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here]. | If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here]. | ||
[[Category:Server]] | [[Category:Server]] [[Category:Applications]] | ||