Llama-cpp: Difference between revisions

Newer edit →

VisualWikitext

@@ Line 76: / Line 76: @@
 Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.
+{| class="wikitable"
-* Nvidia RTX 5090 | 1792 GB/s
+|+
-* Nvidia RTX 4090 | 1008 GB/s
+!System
-* Apple M3 Ultra | 800 GB/s
+!Memory bandwith
-* Radeon RX 9070 XT | 640 GB/s
+!Est. Tokens/Sec (8B, Q4)
-* Strix Halo (AI Max+ 395+) | 256 GB/s
+!Notes
-* Strix Point (HX 370) | 89 – 136 GB/s
+|-
+|Nvidia RTX 5090
+|1792 GB/s
+|~310 – 330 t/s
+|
+|-
+|Nvidia RTX 4090
+|1008 GB/s
+|~180 – 200 t/s
+|
+|-
+|Apple M3 Ultra
+|800 GB/s
+|~145 – 155 t/s
+|
+|-
+|Radeon RX 9070 XT
+|640 GB/s
+|~110 – 125 t/s
+|
+|-
+|Strix Halo (AI Max+ 395+)
+|256 GB/s
+|~45 – 50 t/s
+|
+|-
+|Strix Point (HX 370)
+|89 – 136 GB/s
+|~12 – 25 t/s
+|Depends on the type of RAM used
+|}
 == llama-cli ==
-Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, a straightforward to use tool.
+Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.
 In your shell:
@@ Line 102: / Line 132: @@
 You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.
+ services.llama-cpp = {
-<pre>
+   enable = true;
-services.llama-cpp = {
+   package = pkgs.llama-cpp-vulkan;
-  enable = true;
-  package = pkgs.llama-cpp-vulkan;
+   # Takes care of downloading if model not present
+   modelsPreset = {
-  # Takes care of downloading if model not present
+     "Qwen3-Coder-Next" = {
-  modelsPreset = {
+       hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
-    "Qwen3-Coder-Next" = {
+       hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
-      hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
+       alias = "unsloth/Qwen3-Coder-Next";
-      hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
+       temp = "1.0";
-      alias = "unsloth/Qwen3-Coder-Next";
+       top-p = "0.95";
-      temp = "1.0";
+       top-k = "40";
-      top-p = "0.95";
+     };
-      top-k = "40";
+   };
-    };
+ };
-  };
-};
-</pre>
 And do a switch to the new configuration