Llama-cpp: Difference between revisions

Line 146:

Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.

In your ~~shell~~:<syntaxhighlight lang="bash">llama-cli \

In your terminal try one of these (if they don't work, check you are running the latest llama-cpp version):<syntaxhighlight lang="bash"># LFM2.5-8B-A1B - Requires 8GB VRAM

-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \

llama-cli \

--temp 1.0 --top-p 0.95 --top-k 40 \

-hf unsloth/LFM2.5-8B-A1B-GGUF:UD-Q4_K_XL \

--temp 0.2 --top-p 0.95 --top-k 80 \

--repeat-penalty 1.05 \

-p "briefly explain journalctl in one paragraph"

# Qwen3-Coder-Next - Requires 56GB VRAM

llama-cli \

-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \

--temp 1.0 --top-p 0.95 --top-k 40 \

-p "briefly explain journalctl in one paragraph"</syntaxhighlight>

Line 155:

Line 163:

<code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].

You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, <syntaxhighlight lang="bash">

You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>.

Try any of these models <syntaxhighlight lang="bash"># LFM2.5-8B-A1B - Requires 8GB VRAM

llama-server \

-hf unsloth/LFM2.5-8B-A1B-GGUF:UD-Q4_K_XL \

--temp 0.2 --top-p 0.95 --top-k 80 \

--repeat-penalty 1.05

# Qwen3-Coder-Next - Requires 56GB VRAM

llama-server \

-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \

--temp 1.0 --top-p 0.95 --top-k 40

--temp 1.0 --top-p 0.95 --top-k 40</syntaxhighlight>

</syntaxhighlight>

Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">{

Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">

{

services.llama-cpp = {

enable = true;

Line 170:

Line 184:

# Takes care of downloading if model not present

modelsPreset = {

# Requires 8GB VRAM

"LFM2.5-8B-A1B" = {

hf-repo = "unsloth/LFM2.5-8B-A1B-GGUF";

hf-file = "LFM2.5-8B-A1B-UD-Q4_K_XL.gguf";

alias = "unsloth/LFM2.5-8B-A1B-GGUF";

temp = "0.2";

repeat-penalty = "1.05";

top-k = "80";

};

# Requires 56GB VRAM

"Qwen3-Coder-Next" = {

hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";

Line 180:

Line 204:

};

}

}</syntaxhighlight>And do a switch to the new configuration

</syntaxhighlight>And do a switch to the new configuration

<pre>

@@ Line 146: / Line 146: @@
 Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.
-In your shell:<syntaxhighlight lang="bash">llama-cli \
+In your terminal try one of these (if they don't work, check you are running the latest llama-cpp version):<syntaxhighlight lang="bash"># LFM2.5-8B-A1B - Requires 8GB VRAM
-   -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
+llama-cli \
-   --temp 1.0 --top-p 0.95 --top-k 40 \
+  -hf unsloth/LFM2.5-8B-A1B-GGUF:UD-Q4_K_XL \
+  --temp 0.2 --top-p 0.95 --top-k 80 \
+  --repeat-penalty 1.05 \
+  -p "briefly explain journalctl in one paragraph"
+# Qwen3-Coder-Next - Requires 56GB VRAM
+llama-cli \
+   -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
+   --temp 1.0 --top-p 0.95 --top-k 40 \
    -p "briefly explain journalctl in one paragraph"</syntaxhighlight>
@@ Line 155: / Line 163: @@
 <code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].
-You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, <syntaxhighlight lang="bash">
+You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>.
+Try any of these models <syntaxhighlight lang="bash"># LFM2.5-8B-A1B - Requires 8GB VRAM
+llama-server \
+  -hf unsloth/LFM2.5-8B-A1B-GGUF:UD-Q4_K_XL \
+  --temp 0.2 --top-p 0.95 --top-k 80 \
+  --repeat-penalty 1.05
+# Qwen3-Coder-Next - Requires 56GB VRAM
 llama-server \
      -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
-     --temp 1.0 --top-p 0.95 --top-k 40
+     --temp 1.0 --top-p 0.95 --top-k 40</syntaxhighlight>
-</syntaxhighlight>
+Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">{
-Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">
-{
    services.llama-cpp = {
      enable = true;
@@ Line 170: / Line 184: @@
      # Takes care of downloading if model not present
      modelsPreset = {
+      # Requires 8GB VRAM
+      "LFM2.5-8B-A1B" = {
+        hf-repo = "unsloth/LFM2.5-8B-A1B-GGUF";
+        hf-file = "LFM2.5-8B-A1B-UD-Q4_K_XL.gguf";
+        alias = "unsloth/LFM2.5-8B-A1B-GGUF";
+        temp = "0.2";
+        repeat-penalty = "1.05";
+        top-k = "80";
+      };
+      # Requires 56GB VRAM
        "Qwen3-Coder-Next" = {
          hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
@@ Line 180: / Line 204: @@
      };
    };
-}
+}</syntaxhighlight>And do a switch to the new configuration
-</syntaxhighlight>And do a switch to the new configuration
 <pre>