Ollama: Difference between revisions
m The name of the plugin was changed after a corporation stepped on them with copyright. The current plugin by that name is unassociated with the original. |
Malteneuss (talk | contribs) m Add standalone systempackage example |
||
| Line 2: | Line 2: | ||
== Setup == | == Setup == | ||
You can add Ollama in two ways to your system configuration. | |||
As a standalone package: | |||
<syntaxhighlight lang="nix"> | |||
environment.systemPackages = [ pkgs.ollama ]; | |||
</syntaxhighlight> | |||
As a systemd service: | |||
<syntaxhighlight lang="nix"> | |||
services.ollama = { | services.ollama = { | ||
enable = true; | enable = true; | ||
| Line 17: | Line 25: | ||
Example: Enable GPU acceleration for Nvidia graphic cards<syntaxhighlight lang="nix"> | Example: Enable GPU acceleration for Nvidia graphic cards | ||
As a standalone package: | |||
<syntaxhighlight lang="nix"> | |||
environment.systemPackages = [ | |||
(pkgs.ollama.override { | |||
acceleration = "cuda"; | |||
}) | |||
]; | |||
</syntaxhighlight> | |||
As a systemd service: | |||
<syntaxhighlight lang="nix"> | |||
services.ollama = { | services.ollama = { | ||
enable = true; | enable = true; | ||
| Line 23: | Line 43: | ||
}; | }; | ||
</syntaxhighlight> | </syntaxhighlight> | ||
To find out whether a model is running on CPU or GPU, you can either | |||
look at the logs of | |||
ollama serve | |||
or while a model is answering run | |||
<syntaxhighlight lang="bash"> | |||
$ ollama ps | |||
NAME ID SIZE PROCESSOR UNTIL | |||
gemma3:4b c0494fe00251 4.7 GB 100% GPU 4 minutes from now | |||
</syntaxhighlight> | |||
In this example we see "100% GPU". | |||
== Usage via CLI == | == Usage via CLI == | ||
=== Download a model and run interactive prompt === | === Download a model and run interactive prompt === | ||
Example: Download and run Mistral LLM model as an interactive prompt<syntaxhighlight lang="bash"> | Example: Download and run Mistral LLM model as an interactive prompt<syntaxhighlight lang="bash"> | ||
ollama run mistral | $ ollama run mistral | ||
</syntaxhighlight>For other models see [https://ollama.ai/library Ollama library]. | </syntaxhighlight>For other models see [https://ollama.ai/library Ollama library]. | ||
| Line 33: | Line 65: | ||
Example: To download and run codellama with 13 billion parameters in the "instruct" variant and send a prompt: | Example: To download and run codellama with 13 billion parameters in the "instruct" variant and send a prompt: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
ollama run codellama:13b-instruct "Write an extended Python program with a typical structure. It should print the numbers 1 to 10 to standard output." | $ ollama run codellama:13b-instruct "Write an extended Python program with a typical structure. It should print the numbers 1 to 10 to standard output." | ||
</syntaxhighlight> | </syntaxhighlight> | ||