Llama-cpp - Revision history

Woile: new heading MoE and styling improvements

2026-05-22T08:20:47Z

new heading MoE and styling improvements

← Older revision		Revision as of 08:20, 22 May 2026
Line 59:		Line 59:
	\|RDNA 3 APU		\|RDNA 3 APU
	\|11.0.0		\|11.0.0
	\|~~e.g:~~ 780M		\|780M
	\|-		\|-
	\|Strix Point		\|Strix Point
	\|11.5.0		\|11.5.0
	\|~~e.g:~~ 880M		\|880M
	\|-		\|-
	\|Strix Halo		\|Strix Halo
	\|11.5.1		\|11.5.1
	\|~~e.g:~~ Radeon 8060S		\|Radeon 8060S
	\|-		\|-
	\|RDNA 4 "Navi 48"		\|RDNA 4 "Navi 48"
	\|12.0.1		\|12.0.1
	\|~~e.g:~~ Radeon RX 9070 XT		\|Radeon RX 9070 XT
	\|}		\|}

Line 92:		Line 92:
	A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.		A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.

	When your GPU doesn't have enough RAM, with <code>llama-cli</code> or <code>llama-server</code> you can offload some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]		When your '''GPU doesn't have enough RAM''', with <code>llama-cli</code> or <code>llama-server</code> you can '''offload''' some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]

			=== What are Mixture of Experts (MoE)? ===
			MoE refers to models with a high param count (e.g: 35B), but they only activate a small amount during usage (e.g: 3B).

			This ends up improving performance, but the model still needs to fit in RAM.

			For example, <code>Qwen3.6-35B-A3B</code> has 35B param count, but only 3B are active.

	=== Performance ===		=== Performance ===

	Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.		Performance is '''bottle-necked by memory bandwidth'''. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance from a model.

			Don't confuse GPU's memory bandwidth with GPU's memory (RAM).
	{\| class="wikitable"		{\| class="wikitable"
	\|+		\|+
Line 178:		Line 187:

	==== Failed to create //.cache for shader cache ====		==== Failed to create //.cache for shader cache ====
	This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="~~nix~~">		This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nixos">
	systemd.services.llama-cpp = {		{
	environment = {		systemd.services.llama-cpp = {
	XDG_CACHE_HOME = "/var/cache/llama-cpp";		environment = {
	MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";		XDG_CACHE_HOME = "/var/cache/llama-cpp";
			MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";
			};
	};		};
	};		}
	</syntaxhighlight>		</syntaxhighlight>

Woile: missing title

2026-05-22T08:07:01Z

missing title

← Older revision		Revision as of 08:07, 22 May 2026
Line 73:		Line 73:
	\|e.g: Radeon RX 9070 XT		\|e.g: Radeon RX 9070 XT
	\|}		\|}
	~~Models~~

			== Models ==
	When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.		When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.

Woile: corrrections and style

2026-05-22T08:03:46Z

corrrections and style

← Older revision		Revision as of 08:03, 22 May 2026
Line 46:		Line 46:

	=== AMD ROCm ===		=== AMD ROCm ===
	Sometimes, ROCm might not be using the correct GPU architecture, or you simply want to try a different one, because it might work better. To tell ROCm which GPU architecture to use, you can use the HSA_OVERRIDE_GFX_VERSION environmental variable.		Sometimes, ROCm might not be using the correct GPU architecture, or you simply want to try a different one, because it might work better. To tell ROCm which GPU architecture to use, you can use the <code>HSA_OVERRIDE_GFX_VERSION</code> environmental variable.

	E.g:<syntaxhighlight lang="bash">		E.g:<syntaxhighlight lang="bash">
Line 86:		Line 86:

	And more general purpose models, may include multiple options sets for different purposes, e.g: chat, coding, agent, etc.		And more general purpose models, may include multiple options sets for different purposes, e.g: chat, coding, agent, etc.
	=== ~~Does it run on your machine~~? ===		=== How much RAM do I need? ===

	To calculate how much RAM you are going to need, the first signal is the number of parameters. You can make a loose estimate that 80B params will require 80GB RAM.		To calculate how much RAM you are going to need, the first signal is the number of parameters. You can make a loose estimate that 80B params will require 80GB RAM.

Woile: improve styles and added troubleshooting

2026-05-22T06:50:36Z

improve styles and added troubleshooting

← Older revision		Revision as of 06:50, 22 May 2026
Line 19:		Line 19:
	==== in NixOS ====		==== in NixOS ====

	After enable Unfree software in NixOS add CUDA to your packages		After enable Unfree software in NixOS add CUDA to your packages<syntaxhighlight lang="nixos">
			{
	<~~pre~~>		environment.systemPackages = [
	environment.systemPackages = [		(pkgs.llama-cpp.override { cudaSupport = true; })
	(pkgs.llama-cpp.override { cudaSupport = true; })		];
	];		}
	</~~pre~~>		</syntaxhighlight>And do a switch to the new configuration

	And do a switch to the new configuration
	sudo nixos-rebuild switch		sudo nixos-rebuild switch

	==== in a shell ====		==== in a shell ====

	If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:		If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:<syntaxhighlight lang="bash">

	<~~pre~~>
	export NIXPKGS_ALLOW_UNFREE=1		export NIXPKGS_ALLOW_UNFREE=1
	nix shell --impure --expr '(import (builtins.getFlake "nixpkgs") {}).llama-cpp.override { cudaSupport = true; }'		nix shell --impure --expr '(import (builtins.getFlake "nixpkgs") {}).llama-cpp.override { cudaSupport = true; }'
	</~~pre~~>		</syntaxhighlight>

	=== BLAS Support ===		=== BLAS Support ===

	BLAS support is automatically enabled if none of the GPU accelerators are enabled. You can still manually enable it in your nix configuration by doing:		BLAS support is automatically enabled if none of the GPU accelerators are enabled. You can still manually enable it in your nix configuration by doing:<syntaxhighlight lang="nixos">
			{
	<~~pre~~>		environment.systemPackages = [
	environment.systemPackages = [		(pkgs.llama-cpp.override { blasSupport = true; })
	(pkgs.llama-cpp.override { blasSupport = true; })		];
	];		}
	</~~pre~~>		</syntaxhighlight>

	=== AMD ROCm ===		=== AMD ROCm ===
Line 143:		Line 139:
	Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.		Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.

	In your shell:		In your shell:<syntaxhighlight lang="bash">
			llama-cli \
	<~~pre~~>		-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \
	llama-cli \		--temp 1.0 --top-p 0.95 --top-k 40 \
	-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \		-p "briefly explain journalctl in one paragraph"
	--temp 1.0 --top-p 0.95 --top-k 40 \		</syntaxhighlight>
	-p "briefly explain journalctl in one paragraph"
	</~~pre~~>

	== llama-server ==		== llama-server ==
Line 156:		Line 150:
	<code>llama-server</code> runs a server, and it can run models on demand. It's quite similar to [[Ollama]].		<code>llama-server</code> runs a server, and it can run models on demand. It's quite similar to [[Ollama]].

	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.		You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.<syntaxhighlight lang="nixos">
	services.llama-cpp = {		{
	enable = true;		services.llama-cpp = {
	package = pkgs.llama-cpp-vulkan;		enable = true;
			package = pkgs.llama-cpp-vulkan;
	# Takes care of downloading if model not present		# Takes care of downloading if model not present
	modelsPreset = {		modelsPreset = {
	"Qwen3-Coder-Next" = {		"Qwen3-Coder-Next" = {
	hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";		hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
	hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";		hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
	alias = "unsloth/Qwen3-Coder-Next";		alias = "unsloth/Qwen3-Coder-Next";
	temp = "1.0";		temp = "1.0";
	top-p = "0.95";		top-p = "0.95";
	top-k = "40";		top-k = "40";
	};		};
	};		};
	};		};
	And do a switch to the new configuration		}

			</syntaxhighlight>And do a switch to the new configuration

	<pre>		<pre>
	sudo nixos-rebuild switch		sudo nixos-rebuild switch
	</pre>		</pre>

			=== Troubleshooting ===

			==== Failed to create //.cache for shader cache ====
			This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nix">
			systemd.services.llama-cpp = {
			environment = {
			XDG_CACHE_HOME = "/var/cache/llama-cpp";
			MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";
			};
			};
			</syntaxhighlight>

Woile: Notes about AMD ROCm

2026-05-19T12:31:41Z

Notes about AMD ROCm

@@ Line 28: / Line 28: @@
 And do a switch to the new configuration
+ sudo nixos-rebuild switch
 ==== in a shell ====
@@ Line 52: / Line 49: @@
 </pre>
-== Models ==
+=== AMD ROCm ===
 When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.

Woile: titles improved

2026-05-19T12:19:26Z

titles improved

← Older revision		Revision as of 12:19, 19 May 2026
Line 11:		Line 11:
	You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system		You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system

	== ~~customization~~ ==		== Customization ==

	=== Nvidia CUDA ===		=== Nvidia CUDA ===
Line 17:		Line 17:
	Nvidia CUDA contains [[Unfree software]], so you have to enable it first, either in your NixOS configuration or via environmental variables.		Nvidia CUDA contains [[Unfree software]], so you have to enable it first, either in your NixOS configuration or via environmental variables.

	==== NixOS ====		==== in NixOS ====

	After enable Unfree software in NixOS add CUDA to your packages		After enable Unfree software in NixOS add CUDA to your packages
Line 33:		Line 33:
	</pre>		</pre>

	==== ~~Creating~~ a shell ====		==== in a shell ====

	If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:		If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:
Line 52:		Line 52:
	</pre>		</pre>

	== ~~models~~ ==		== Models ==

	When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.		When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.

Woile at 12:16, 19 May 2026

2026-05-19T12:16:51Z

← Older revision		Revision as of 12:16, 19 May 2026
Line 6:		Line 6:

	* <code>llama-cpp</code>: the umbrella package, it uses the CPU if it doesn't find any GPU. On Mac Sillicon, it automatically detects that it should use the Metal backend. And for NVIDIA CUDA, you need to enable cudaSupport and unfree packages.		* <code>llama-cpp</code>: the umbrella package, it uses the CPU if it doesn't find any GPU. On Mac Sillicon, it automatically detects that it should use the Metal backend. And for NVIDIA CUDA, you need to enable cudaSupport and unfree packages.
	* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with ~~<code>~~rocmSupport ~~= true</code>~~.		* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with rocmSupport enabled.
	* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with ~~<code>~~vulkanSupport ~~= true</code>~~. In some situations, it may perform even better than ROCm.		* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with vulkanSupport enabled. In some situations, it may perform even better than ROCm.

	You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system		You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system

Woile at 08:27, 19 May 2026

2026-05-19T08:27:19Z

← Older revision		Revision as of 08:27, 19 May 2026
Line 76:		Line 76:

	Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.		Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.
			{\| class="wikitable"
	* Nvidia RTX 5090 \| 1792 GB/s		\|+
	* Nvidia RTX 4090 \| 1008 GB/s		!System
	* Apple M3 Ultra \| 800 GB/s		!Memory bandwith
	* Radeon RX 9070 XT \| 640 GB/s		!Est. Tokens/Sec (8B, Q4)
	* Strix Halo (AI Max+ 395+) \| 256 GB/s		!Notes
	* Strix Point (HX 370) \| 89 – 136 GB/s		\|-
			\|Nvidia RTX 5090
			\|1792 GB/s
			\|~310 – 330 t/s
			\|
			\|-
			\|Nvidia RTX 4090
			\|1008 GB/s
			\|~180 – 200 t/s
			\|
			\|-
			\|Apple M3 Ultra
			\|800 GB/s
			\|~145 – 155 t/s
			\|
			\|-
			\|Radeon RX 9070 XT
			\|640 GB/s
			\|~110 – 125 t/s
			\|
			\|-
			\|Strix Halo (AI Max+ 395+)
			\|256 GB/s
			\|~45 – 50 t/s
			\|
			\|-
			\|Strix Point (HX 370)
			\|89 – 136 GB/s
			\|~12 – 25 t/s
			\|Depends on the type of RAM used
			\|}

	== llama-cli ==		== llama-cli ==

	Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, a straightforward to use tool.		Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.

	In your shell:		In your shell:
Line 102:		Line 132:

	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.		You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.
			services.llama-cpp = {
	~~<pre>~~		enable = true;
	services.llama-cpp = {		package = pkgs.llama-cpp-vulkan;
	enable = true;
	package = pkgs.llama-cpp-vulkan;		# Takes care of downloading if model not present
			modelsPreset = {
	# Takes care of downloading if model not present		"Qwen3-Coder-Next" = {
	modelsPreset = {		hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
	"Qwen3-Coder-Next" = {		hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
	hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";		alias = "unsloth/Qwen3-Coder-Next";
	hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";		temp = "1.0";
	alias = "unsloth/Qwen3-Coder-Next";		top-p = "0.95";
	temp = "1.0";		top-k = "40";
	top-p = "0.95";		};
	top-k = "40";		};
	};		};
	};
	};
	~~</pre>~~

	And do a switch to the new configuration		And do a switch to the new configuration

Woile: Add llama-cpp with examples and cli usage

2026-05-19T08:19:13Z

Add llama-cpp with examples and cli usage

New page

The <code>llama-cpp</code> package in [https://search.nixos.org/packages?channel=25.11&query=llama-cpp#show=llama-cpp nixpkgs] contains several tools provided by the [https://github.com/ggml-org/llama.cpp llama.cpp] repository.
A non-exhaustive example includes: <code>llama-cli</code>, <code>llama-server</code>, and <code>llama-bench</code>

The package comes in 3 flavors:

* <code>llama-cpp</code>: the umbrella package, it uses the CPU if it doesn't find any GPU. On Mac Sillicon, it automatically detects that it should use the Metal backend. And for NVIDIA CUDA, you need to enable cudaSupport and unfree packages.
* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with <code>rocmSupport = true</code>.
* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with <code>vulkanSupport = true</code>. In some situations, it may perform even better than ROCm.

You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system

== customization ==

=== Nvidia CUDA ===

Nvidia CUDA contains [[Unfree software]], so you have to enable it first, either in your NixOS configuration or via environmental variables.

==== NixOS ====

After enable Unfree software in NixOS add CUDA to your packages

<pre>
environment.systemPackages = [
(pkgs.llama-cpp.override { cudaSupport = true; })
];
</pre>

And do a switch to the new configuration

<pre>
sudo nixos-rebuild switch
</pre>

==== Creating a shell ====

If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:

<pre>
export NIXPKGS_ALLOW_UNFREE=1
nix shell --impure --expr '(import (builtins.getFlake "nixpkgs") {}).llama-cpp.override { cudaSupport = true; }'
</pre>

=== BLAS Support ===

BLAS support is automatically enabled if none of the GPU accelerators are enabled. You can still manually enable it in your nix configuration by doing:

<pre>
environment.systemPackages = [
(pkgs.llama-cpp.override { blasSupport = true; })
];
</pre>

== models ==

When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.

Open models, usually include a card in their model page explaining how to optimize the parameters for different tasks.

For example, [https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF#best-practices Qwen3-Coder-Next-GGUF] reads:

<blockquote>
To achieve optimal performance, we recommend the following sampling parameters: temperature=1.0, top_p=0.95, top_k=40.
</blockquote>

And more general purpose models, may include multiple options sets for different purposes, e.g: chat, coding, agent, etc.
=== Does it run on your machine? ===

To calculate how much RAM you are going to need, the first signal is the number of parameters. You can make a loose estimate that 80B params will require 80GB RAM.

A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.

When your GPU doesn't have enough RAM, with <code>llama-cli</code> or <code>llama-server</code> you can offload some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]

=== Performance ===

Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.

* Nvidia RTX 5090 | 1792 GB/s
* Nvidia RTX 4090 | 1008 GB/s
* Apple M3 Ultra | 800 GB/s
* Radeon RX 9070 XT | 640 GB/s
* Strix Halo (AI Max+ 395+) | 256 GB/s
* Strix Point (HX 370) | 89 – 136 GB/s

== llama-cli ==

Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, a straightforward to use tool.

In your shell:

<pre>
llama-cli \
-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \
--temp 1.0 --top-p 0.95 --top-k 40 \
-p "briefly explain journalctl in one paragraph"
</pre>

== llama-server ==

<code>llama-server</code> runs a server, and it can run models on demand. It's quite similar to [[Ollama]].

You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.

<pre>
services.llama-cpp = {
enable = true;
package = pkgs.llama-cpp-vulkan;

# Takes care of downloading if model not present
modelsPreset = {
"Qwen3-Coder-Next" = {
hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
alias = "unsloth/Qwen3-Coder-Next";
temp = "1.0";
top-p = "0.95";
top-k = "40";
};
};
};
</pre>

And do a switch to the new configuration

<pre>
sudo nixos-rebuild switch
</pre>