Llama-cpp: Difference between revisions

Line 59:

|RDNA 3 APU

|11.0.0

|~~e.g:~~ 780M

|780M

|-

|Strix Point

|11.5.0

|~~e.g:~~ 880M

|880M

|-

|Strix Halo

|11.5.1

|~~e.g:~~ Radeon 8060S

|Radeon 8060S

|-

|RDNA 4 "Navi 48"

|12.0.1

|~~e.g:~~ Radeon RX 9070 XT

|Radeon RX 9070 XT

|}

Line 92:

A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.

When your GPU doesn't have enough RAM, with <code>llama-cli</code> or <code>llama-server</code> you can offload some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]

When your '''GPU doesn't have enough RAM''', with <code>llama-cli</code> or <code>llama-server</code> you can '''offload''' some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]

=== What are Mixture of Experts (MoE)? ===

MoE refers to models with a high param count (e.g: 35B), but they only activate a small amount during usage (e.g: 3B).

This ends up improving performance, but the model still needs to fit in RAM.

For example, <code>Qwen3.6-35B-A3B</code> has 35B param count, but only 3B are active.

=== Performance ===

Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.

Performance is '''bottle-necked by memory bandwidth'''. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance from a model.

Don't confuse GPU's memory bandwidth with GPU's memory (RAM).

{| class="wikitable"

|+

Line 178:

Line 187:

==== Failed to create //.cache for shader cache ====

This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="~~nix~~">

This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nixos">

systemd.services.llama-cpp = {

{

environment = {

systemd.services.llama-cpp = {

XDG_CACHE_HOME = "/var/cache/llama-cpp";

environment = {

MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";

XDG_CACHE_HOME = "/var/cache/llama-cpp";

MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";

};

}

</syntaxhighlight>

@@ Line 59: / Line 59: @@
 |RDNA 3 APU
 |11.0.0
-|e.g: 780M
+|780M
 |-
 |Strix Point
 |11.5.0
-|e.g: 880M
+|880M
 |-
 |Strix Halo
 |11.5.1
-|e.g: Radeon 8060S
+|Radeon 8060S
 |-
 |RDNA 4 "Navi 48"
 |12.0.1
-|e.g: Radeon RX 9070 XT
+|Radeon RX 9070 XT
 |}
@@ Line 92: / Line 92: @@
 A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.
-When your GPU doesn't have enough RAM, with <code>llama-cli</code> or <code>llama-server</code> you can offload some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]
+When your '''GPU doesn't have enough RAM''', with <code>llama-cli</code> or <code>llama-server</code> you can '''offload''' some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]
+=== What are Mixture of Experts (MoE)? ===
+MoE refers to models with a high param count (e.g: 35B), but they only activate a small amount during usage (e.g: 3B).
+This ends up improving performance, but the model still needs to fit in RAM.
+For example, <code>Qwen3.6-35B-A3B</code> has 35B param count, but only 3B are active.
 === Performance ===
-Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.
+Performance is '''bottle-necked by memory bandwidth'''. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance from a model.
+Don't confuse GPU's memory bandwidth with GPU's memory (RAM).
 {| class="wikitable"
 |+
@@ Line 178: / Line 187: @@
 ==== Failed to create //.cache for shader cache ====
-This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nix">
+This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nixos">
-systemd.services.llama-cpp = {
+{
-  environment = {
+  systemd.services.llama-cpp = {
-    XDG_CACHE_HOME = "/var/cache/llama-cpp";
+    environment = {
-    MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";
+      XDG_CACHE_HOME = "/var/cache/llama-cpp";
+      MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";
+    };
    };
-};
+}
 </syntaxhighlight>