Llama-cpp - Revision history

Woile: Add a model with less RAM requirements

2026-07-20T07:15:07Z

Add a model with less RAM requirements

← Older revision		Revision as of 07:15, 20 July 2026
Line 146:		Line 146:
	Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.		Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.

	In your ~~shell~~:<syntaxhighlight lang="bash">llama-cli \		In your terminal try one of these (if they don't work, check you are running the latest llama-cpp version):<syntaxhighlight lang="bash"># LFM2.5-8B-A1B - Requires 8GB VRAM
	-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \		llama-cli \
	--temp 1.0 --top-p 0.95 --top-k 40 \		-hf unsloth/LFM2.5-8B-A1B-GGUF:UD-Q4_K_XL \
			--temp 0.2 --top-p 0.95 --top-k 80 \
			--repeat-penalty 1.05 \
			-p "briefly explain journalctl in one paragraph"

			# Qwen3-Coder-Next - Requires 56GB VRAM
			llama-cli \
			-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
			--temp 1.0 --top-p 0.95 --top-k 40 \
	-p "briefly explain journalctl in one paragraph"</syntaxhighlight>		-p "briefly explain journalctl in one paragraph"</syntaxhighlight>

Line 155:		Line 163:
	<code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].		<code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].

	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, <syntaxhighlight lang="bash">		You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>.

			Try any of these models <syntaxhighlight lang="bash"># LFM2.5-8B-A1B - Requires 8GB VRAM
			llama-server \
			-hf unsloth/LFM2.5-8B-A1B-GGUF:UD-Q4_K_XL \
			--temp 0.2 --top-p 0.95 --top-k 80 \
			--repeat-penalty 1.05

			# Qwen3-Coder-Next - Requires 56GB VRAM
	llama-server \		llama-server \
	-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \		-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
	--temp 1.0 --top-p 0.95 --top-k 40		--temp 1.0 --top-p 0.95 --top-k 40</syntaxhighlight>
	</syntaxhighlight>		Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning\|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">{
	Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning\|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">
	{
	services.llama-cpp = {		services.llama-cpp = {
	enable = true;		enable = true;
Line 170:		Line 184:
	# Takes care of downloading if model not present		# Takes care of downloading if model not present
	modelsPreset = {		modelsPreset = {
			# Requires 8GB VRAM
			"LFM2.5-8B-A1B" = {
			hf-repo = "unsloth/LFM2.5-8B-A1B-GGUF";
			hf-file = "LFM2.5-8B-A1B-UD-Q4_K_XL.gguf";
			alias = "unsloth/LFM2.5-8B-A1B-GGUF";
			temp = "0.2";
			repeat-penalty = "1.05";
			top-k = "80";
			};
			# Requires 56GB VRAM
	"Qwen3-Coder-Next" = {		"Qwen3-Coder-Next" = {
	hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";		hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
Line 180:		Line 204:
	};		};
	};		};
	}		}</syntaxhighlight>And do a switch to the new configuration

	</syntaxhighlight>And do a switch to the new configuration

	<pre>		<pre>

Woile: use unsloth instead of bartowski models

2026-07-15T14:43:05Z

use unsloth instead of bartowski models

← Older revision		Revision as of 14:43, 15 July 2026
Line 147:		Line 147:

	In your shell:<syntaxhighlight lang="bash">llama-cli \		In your shell:<syntaxhighlight lang="bash">llama-cli \
	-hf ~~bartowski~~/~~Qwen_Qwen3~~-Coder-Next-GGUF:Q4_K_M \		-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
	--temp 1.0 --top-p 0.95 --top-k 40 \		--temp 1.0 --top-p 0.95 --top-k 40 \
	-p "briefly explain journalctl in one paragraph"</syntaxhighlight>		-p "briefly explain journalctl in one paragraph"</syntaxhighlight>
Line 157:		Line 157:
	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, <syntaxhighlight lang="bash">		You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, <syntaxhighlight lang="bash">
	llama-server \		llama-server \
	-hf ~~bartowski~~/~~Qwen_Qwen3~~-Coder-Next-GGUF:Q4_K_M \		-hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M \
	--temp 1.0 --top-p 0.95 --top-k 40		--temp 1.0 --top-p 0.95 --top-k 40
	</syntaxhighlight>		</syntaxhighlight>

Woile: remove comments to avoid distraction

2026-06-30T08:45:34Z

remove comments to avoid distraction

← Older revision		Revision as of 08:45, 30 June 2026
Line 210:		Line 210:
	enable = true;		enable = true;
	package = pkgs.llama-cpp-vulkan;		package = pkgs.llama-cpp-vulkan;
	~~# package = (pkgs.llama-cpp.override { cudaSupport = true; })~~
	~~# package = pkgs.llama-cpp-rocm;~~
	~~# Takes care of downloading if model not present~~
	- port = 8083;		- port = 8083;
	- modelsPreset = {		- modelsPreset = {

Woile: add migration to RFC42

2026-06-27T21:25:35Z

add migration to RFC42

← Older revision		Revision as of 21:25, 27 June 2026
Line 204:		Line 204:
	}		}
	</syntaxhighlight>		</syntaxhighlight>

			==== Migration to nixos-unstable (RFC42) ====
			Since [https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md RFC42] was approved, services are being migrated to use <code>.settings</code>, including <code>llama-cpp</code>. This is already the case for <code>nixos-unstable</code>. If you are using unstable, this is how you can migrate your service:<syntaxhighlight lang="diff">{
			services.llama-cpp = {
			enable = true;
			package = pkgs.llama-cpp-vulkan;
			# package = (pkgs.llama-cpp.override { cudaSupport = true; })
			# package = pkgs.llama-cpp-rocm;
			# Takes care of downloading if model not present
			- port = 8083;
			- modelsPreset = {
			+ settings.port = 8083;
			+ settings.models-preset = (pkgs.formats.ini { }).generate "models-preset.ini" {
			"Qwen3-Coder-Next" = {
			hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
			hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
			alias = "unsloth/Qwen3-Coder-Next";
			temp = "1.0";
			top-p = "0.95";
			top-k = "40";
			};
			};
			};
			}</syntaxhighlight>{{Note\|TODO: When current unstable becomes stable, remove this troubleshooting and update `llama-server` section}}

Woile: added example for llama-server on the terminal

2026-05-27T07:23:54Z

added example for llama-server on the terminal

← Older revision		Revision as of 07:23, 27 May 2026
Line 146:		Line 146:
	Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.		Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.

	In your shell:<syntaxhighlight lang="bash">		In your shell:<syntaxhighlight lang="bash">llama-cli \
	llama-cli \
	-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \		-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \
	--temp 1.0 --top-p 0.95 --top-k 40 \		--temp 1.0 --top-p 0.95 --top-k 40 \
	-p "briefly explain journalctl in one paragraph"		-p "briefly explain journalctl in one paragraph"</syntaxhighlight>
	</syntaxhighlight>

	== llama-server ==		== llama-server ==
Line 157:		Line 155:
	<code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].		<code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].

			You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, <syntaxhighlight lang="bash">
			llama-server \
	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, ~~but we are going to see~~ the ~~integration with~~ NixOS ~~as a~~ service.		-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \
	{{Warning\|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">		--temp 1.0 --top-p 0.95 --top-k 40
			</syntaxhighlight>
			Or alternatively, you can '''enable the NixOS service''' for llama-cpp, which runs the server.{{Warning\|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">
	{		{
	services.llama-cpp = {		services.llama-cpp = {

Woile: added webui segment, and a warning about naming

2026-05-27T07:19:04Z

added webui segment, and a warning about naming

← Older revision		Revision as of 07:19, 27 May 2026
Line 19:		Line 19:
	==== in NixOS ====		==== in NixOS ====

	After enable Unfree software in NixOS add CUDA to your packages<syntaxhighlight lang="nixos">		After enable Unfree software in NixOS add CUDA to your packages<syntaxhighlight lang="nixos">{
	{
	environment.systemPackages = [		environment.systemPackages = [
	(pkgs.llama-cpp.override { cudaSupport = true; })		(pkgs.llama-cpp.override { cudaSupport = true; })
	];		];
	}		}</syntaxhighlight>And do a switch to the new configuration
	</syntaxhighlight>And do a switch to the new configuration
	sudo nixos-rebuild switch		sudo nixos-rebuild switch

Line 157:		Line 155:
	== llama-server ==		== llama-server ==

	<code>llama-server</code> runs a server, and it can run models on demand. It's quite similar to [[Ollama]].		<code>llama-server</code> runs a server, and it can run models on demand. It supports OpenAI API standard. It's quite similar to [[Ollama]].

	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.<syntaxhighlight lang="nixos">

			You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.
			{{Warning\|Pay attention, that the service is actually called llama-cpp not llama-server}}<syntaxhighlight lang="nixos">
	{		{
	services.llama-cpp = {		services.llama-cpp = {
	enable = true;		enable = true;
	package = pkgs.llama-cpp-vulkan;		package = pkgs.llama-cpp-vulkan;
			# package = (pkgs.llama-cpp.override { cudaSupport = true; })
			# package = pkgs.llama-cpp-rocm;

	# Takes care of downloading if model not present		# Takes care of downloading if model not present
	modelsPreset = {		modelsPreset = {
Line 183:		Line 187:
	sudo nixos-rebuild switch		sudo nixos-rebuild switch
	</pre>		</pre>

			=== Web UI ===
			The llama-cpp service includes a web interface, where you can chat. To access you must navigate to http://localhost:8080 . Or the <code>services.llama-cpp.port</code> configured.

	=== Troubleshooting ===		=== Troubleshooting ===

Woile: new heading MoE and styling improvements

2026-05-22T08:20:47Z

new heading MoE and styling improvements

← Older revision		Revision as of 08:20, 22 May 2026
Line 59:		Line 59:
	\|RDNA 3 APU		\|RDNA 3 APU
	\|11.0.0		\|11.0.0
	\|~~e.g:~~ 780M		\|780M
	\|-		\|-
	\|Strix Point		\|Strix Point
	\|11.5.0		\|11.5.0
	\|~~e.g:~~ 880M		\|880M
	\|-		\|-
	\|Strix Halo		\|Strix Halo
	\|11.5.1		\|11.5.1
	\|~~e.g:~~ Radeon 8060S		\|Radeon 8060S
	\|-		\|-
	\|RDNA 4 "Navi 48"		\|RDNA 4 "Navi 48"
	\|12.0.1		\|12.0.1
	\|~~e.g:~~ Radeon RX 9070 XT		\|Radeon RX 9070 XT
	\|}		\|}

Line 92:		Line 92:
	A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.		A better signal, is the size of the model. An 80B params model may be quantized, and the final size might be 60GB. By adding a 15% for context buffer, you would need an aprox. of 69 GB RAM.

	When your GPU doesn't have enough RAM, with <code>llama-cli</code> or <code>llama-server</code> you can offload some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]		When your '''GPU doesn't have enough RAM''', with <code>llama-cli</code> or <code>llama-server</code> you can '''offload''' some of it to your system's RAM, by using the flag <code>-ngl</code>. Read the [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#command-line-arguments-reference cli reference]

			=== What are Mixture of Experts (MoE)? ===
			MoE refers to models with a high param count (e.g: 35B), but they only activate a small amount during usage (e.g: 3B).

			This ends up improving performance, but the model still needs to fit in RAM.

			For example, <code>Qwen3.6-35B-A3B</code> has 35B param count, but only 3B are active.

	=== Performance ===		=== Performance ===

	Performance is bottle-necked by memory bandwidth. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance.		Performance is '''bottle-necked by memory bandwidth'''. You can keep an eye for new developments and breakthroughs. But otherwise, it's hard to squeeze more performance from a model.

			Don't confuse GPU's memory bandwidth with GPU's memory (RAM).
	{\| class="wikitable"		{\| class="wikitable"
	\|+		\|+
Line 178:		Line 187:

	==== Failed to create //.cache for shader cache ====		==== Failed to create //.cache for shader cache ====
	This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="~~nix~~">		This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nixos">
	systemd.services.llama-cpp = {		{
	environment = {		systemd.services.llama-cpp = {
	XDG_CACHE_HOME = "/var/cache/llama-cpp";		environment = {
	MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";		XDG_CACHE_HOME = "/var/cache/llama-cpp";
			MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";
			};
	};		};
	};		}
	</syntaxhighlight>		</syntaxhighlight>

Woile: missing title

2026-05-22T08:07:01Z

missing title

← Older revision		Revision as of 08:07, 22 May 2026
Line 73:		Line 73:
	\|e.g: Radeon RX 9070 XT		\|e.g: Radeon RX 9070 XT
	\|}		\|}
	~~Models~~

			== Models ==
	When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.		When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model.

Woile: corrrections and style

2026-05-22T08:03:46Z

corrrections and style

← Older revision		Revision as of 08:03, 22 May 2026
Line 46:		Line 46:

	=== AMD ROCm ===		=== AMD ROCm ===
	Sometimes, ROCm might not be using the correct GPU architecture, or you simply want to try a different one, because it might work better. To tell ROCm which GPU architecture to use, you can use the HSA_OVERRIDE_GFX_VERSION environmental variable.		Sometimes, ROCm might not be using the correct GPU architecture, or you simply want to try a different one, because it might work better. To tell ROCm which GPU architecture to use, you can use the <code>HSA_OVERRIDE_GFX_VERSION</code> environmental variable.

	E.g:<syntaxhighlight lang="bash">		E.g:<syntaxhighlight lang="bash">
Line 86:		Line 86:

	And more general purpose models, may include multiple options sets for different purposes, e.g: chat, coding, agent, etc.		And more general purpose models, may include multiple options sets for different purposes, e.g: chat, coding, agent, etc.
	=== ~~Does it run on your machine~~? ===		=== How much RAM do I need? ===

	To calculate how much RAM you are going to need, the first signal is the number of parameters. You can make a loose estimate that 80B params will require 80GB RAM.		To calculate how much RAM you are going to need, the first signal is the number of parameters. You can make a loose estimate that 80B params will require 80GB RAM.

Woile: improve styles and added troubleshooting

2026-05-22T06:50:36Z

improve styles and added troubleshooting

← Older revision		Revision as of 06:50, 22 May 2026
Line 19:		Line 19:
	==== in NixOS ====		==== in NixOS ====

	After enable Unfree software in NixOS add CUDA to your packages		After enable Unfree software in NixOS add CUDA to your packages<syntaxhighlight lang="nixos">
			{
	<~~pre~~>		environment.systemPackages = [
	environment.systemPackages = [		(pkgs.llama-cpp.override { cudaSupport = true; })
	(pkgs.llama-cpp.override { cudaSupport = true; })		];
	];		}
	</~~pre~~>		</syntaxhighlight>And do a switch to the new configuration

	And do a switch to the new configuration
	sudo nixos-rebuild switch		sudo nixos-rebuild switch

	==== in a shell ====		==== in a shell ====

	If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:		If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell:<syntaxhighlight lang="bash">

	<~~pre~~>
	export NIXPKGS_ALLOW_UNFREE=1		export NIXPKGS_ALLOW_UNFREE=1
	nix shell --impure --expr '(import (builtins.getFlake "nixpkgs") {}).llama-cpp.override { cudaSupport = true; }'		nix shell --impure --expr '(import (builtins.getFlake "nixpkgs") {}).llama-cpp.override { cudaSupport = true; }'
	</~~pre~~>		</syntaxhighlight>

	=== BLAS Support ===		=== BLAS Support ===

	BLAS support is automatically enabled if none of the GPU accelerators are enabled. You can still manually enable it in your nix configuration by doing:		BLAS support is automatically enabled if none of the GPU accelerators are enabled. You can still manually enable it in your nix configuration by doing:<syntaxhighlight lang="nixos">
			{
	<~~pre~~>		environment.systemPackages = [
	environment.systemPackages = [		(pkgs.llama-cpp.override { blasSupport = true; })
	(pkgs.llama-cpp.override { blasSupport = true; })		];
	];		}
	</~~pre~~>		</syntaxhighlight>

	=== AMD ROCm ===		=== AMD ROCm ===
Line 143:		Line 139:
	Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.		Once you've made <code>llama-cpp</code> available in your system. You can use <code>llama-cli</code>, which is a straightforward to use tool.

	In your shell:		In your shell:<syntaxhighlight lang="bash">
			llama-cli \
	<~~pre~~>		-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \
	llama-cli \		--temp 1.0 --top-p 0.95 --top-k 40 \
	-hf bartowski/Qwen_Qwen3-Coder-Next-GGUF:Q4_K_M \		-p "briefly explain journalctl in one paragraph"
	--temp 1.0 --top-p 0.95 --top-k 40 \		</syntaxhighlight>
	-p "briefly explain journalctl in one paragraph"
	</~~pre~~>

	== llama-server ==		== llama-server ==
Line 156:		Line 150:
	<code>llama-server</code> runs a server, and it can run models on demand. It's quite similar to [[Ollama]].		<code>llama-server</code> runs a server, and it can run models on demand. It's quite similar to [[Ollama]].

	You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.		You can manually start the server from your terminal, it's usage, is not that different from <code>llama-cli</code>, but we are going to see the integration with NixOS as a service.<syntaxhighlight lang="nixos">
	services.llama-cpp = {		{
	enable = true;		services.llama-cpp = {
	package = pkgs.llama-cpp-vulkan;		enable = true;
			package = pkgs.llama-cpp-vulkan;
	# Takes care of downloading if model not present		# Takes care of downloading if model not present
	modelsPreset = {		modelsPreset = {
	"Qwen3-Coder-Next" = {		"Qwen3-Coder-Next" = {
	hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";		hf-repo = "unsloth/Qwen3-Coder-Next-GGUF";
	hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";		hf-file = "Qwen3-Coder-Next-UD-Q4_K_XL.gguf";
	alias = "unsloth/Qwen3-Coder-Next";		alias = "unsloth/Qwen3-Coder-Next";
	temp = "1.0";		temp = "1.0";
	top-p = "0.95";		top-p = "0.95";
	top-k = "40";		top-k = "40";
	};		};
	};		};
	};		};
	And do a switch to the new configuration		}

			</syntaxhighlight>And do a switch to the new configuration

	<pre>		<pre>
	sudo nixos-rebuild switch		sudo nixos-rebuild switch
	</pre>		</pre>

			=== Troubleshooting ===

			==== Failed to create //.cache for shader cache ====
			This is a known issue ([https://github.com/NixOS/nixpkgs/issues/441531 441531]), until it gets fixed, you can add to your conf:<syntaxhighlight lang="nix">
			systemd.services.llama-cpp = {
			environment = {
			XDG_CACHE_HOME = "/var/cache/llama-cpp";
			MESA_SHADER_CACHE_DIR = "/var/cache/llama-cpp";
			};
			};
			</syntaxhighlight>