Talk:Llama-cpp - Revision history

Hans4687: /* Command llama-cli gives error */ Reply

2026-07-20T07:52:01Z

Command llama-cli gives error: Reply

← Older revision		Revision as of 07:52, 20 July 2026
Line 66:		Line 66:
	:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>		:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>
	:::::I think yours is quite old, it's not even in stable right now. [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 07:51, 16 July 2026 (UTC)		:::::I think yours is quite old, it's not even in stable right now. [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 07:51, 16 July 2026 (UTC)
			::::::It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only. I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 07:52, 20 July 2026 (UTC)

Woile: /* Command llama-cli gives error */ Reply

2026-07-16T07:51:03Z

Command llama-cli gives error: Reply

← Older revision		Revision as of 07:51, 16 July 2026
Line 60:		Line 60:
	:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)		:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)
	::::I have 64GB ram. [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:37, 15 July 2026 (UTC)		::::I have 64GB ram. [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:37, 15 July 2026 (UTC)
			:::::And you are using unified memory, right? It's strange, because this just worked for me:
			:::::<code>nix-shell -p llama-cpp-vulkan</code>
			:::::<code>llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"</code>
			:::::I think I'll display a smaller model in the page anyways.
			:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>
			:::::I think yours is quite old, it's not even in stable right now. [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 07:51, 16 July 2026 (UTC)

Hans4687: /* Command llama-cli gives error */ Reply

2026-07-15T16:37:08Z

Command llama-cli gives error: Reply

← Older revision		Revision as of 16:37, 15 July 2026
Line 59:		Line 59:
	::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)		::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)
	:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)		:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)
			::::I have 64GB ram. [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:37, 15 July 2026 (UTC)

Woile: /* Command llama-cli gives error */ Reply

2026-07-15T16:26:11Z

Command llama-cli gives error: Reply

← Older revision		Revision as of 16:26, 15 July 2026
Line 58:		Line 58:
	::...		::...
	::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)		::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)
			:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)

Hans4687: /* Command llama-cli gives error */ Reply

2026-07-15T16:12:32Z

Command llama-cli gives error: Reply

← Older revision		Revision as of 16:12, 15 July 2026
Line 49:		Line 49:

	:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 14:45, 15 July 2026 (UTC)		:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 14:45, 15 July 2026 (UTC)
			::This is not working because the model architecture is the same as before (qwen3next). The error remains.
			::<syntaxhighlight lang="mediawiki">
			::...
			::llama_model_loader: loaded meta data with 52 key-value pairs and 843 tensors from /home/hans/.cache/llama.cpp/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-UD-Q4_K_M.gguf (version GGUF V3 (latest))
			::llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
			::llama_model_loader: - kv 0: general.architecture str = qwen3next
			::llama_model_loader: - kv 1: general.type str = model
			::...
			::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)

Woile: /* Command llama-cli gives error */ Reply

2026-07-15T14:45:02Z

Command llama-cli gives error: Reply

← Older revision		Revision as of 14:45, 15 July 2026
Line 47:		Line 47:
	ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.		ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.
	</syntaxhighlight>[[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 14:36, 15 July 2026 (UTC)		</syntaxhighlight>[[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 14:36, 15 July 2026 (UTC)

			:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 14:45, 15 July 2026 (UTC)

Hans4687: /* Command llama-cli gives error */ new section

2026-07-15T14:36:57Z

Command llama-cli gives error: new section

← Older revision		Revision as of 14:36, 15 July 2026
Line 6:		Line 6:

	Thanks [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 12:36, 19 May 2026 (UTC)		Thanks [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 12:36, 19 May 2026 (UTC)

			== Command llama-cli gives error ==

			I tried this command on nix-shell with llama-cpp version: 6981 (647b960).

			Error:

			<syntaxhighlight lang="mediawiki">
			llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen3next' llama_model_load_from_file_impl: failed to load model
			</syntaxhighlight>

			The model is not supported (not yet) bij llama-cpp

			If I try another model e.g.
			<syntaxhighlight lang="mediawiki">
			llama-cli -hf bartowski/Qwen2.5-Coder-3B-Instruct-GGUF:Q6_K_L --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"

			...

			== Running in interactive mode. ==
			- Press Ctrl+C to interject at any time.

			- Press Return to return control to the AI.

			- To return control without starting a new line, end your input with '/'.

			- If you want to submit another line, end your input with '\'.

			- Not using system message. To change it, set a different value via -sys PROMPT
			user

			briefly explain journalctl in one paragraph

			assistant

			Journalctl is a powerful command-line utility provided by the systemd system and service manager that allows you to view, search, and monitor the system's logs in real-time. It provides detailed information about vario

			us aspects of the system, such as system boot, services, and applications, and can be used to troubleshoot issues and understand system behavior. Journalctl supports various log levels, filtering options, and can be us

			ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.
			</syntaxhighlight>[[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 14:36, 15 July 2026 (UTC)

Woile: /* Review the AMD ROCm section */ new section

2026-05-19T12:36:04Z

Review the AMD ROCm section: new section

New page

== Review the AMD ROCm section ==

I wrote from memory, and I'm not entirely sure my knowledge is accurate. When changing the env variable, I've noticed performance changes in llama-cpp. But again, I'm not sure. Could someone verify what I wrote?

I also wonder if one should compile llama-cpp on their own machine with rocmGpuTargets ??

Thanks [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 12:36, 19 May 2026 (UTC)

← Older revision		Revision as of 16:37, 15 July 2026
Line 59:		Line 59:
	::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)		::</syntaxhighlight> [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:12, 15 July 2026 (UTC)
	:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)		:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile\|Woile]] ([[User talk:Woile\|talk]]) 16:26, 15 July 2026 (UTC)
			::::I have 64GB ram. [[User:Hans4687\|Hans4687]] ([[User talk:Hans4687\|talk]]) 16:37, 15 July 2026 (UTC)