Talk:Llama-cpp

Review the AMD ROCm section

I wrote from memory, and I'm not entirely sure my knowledge is accurate. When changing the env variable, I've noticed performance changes in llama-cpp. But again, I'm not sure. Could someone verify what I wrote?

I also wonder if one should compile llama-cpp on their own machine with rocmGpuTargets ??

Thanks Woile (talk) 12:36, 19 May 2026 (UTC)

Command llama-cli gives error

I tried this command on nix-shell with llama-cpp version: 6981 (647b960).

Error:

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen3next' llama_model_load_from_file_impl: failed to load model

The model is not supported (not yet) bij llama-cpp

If I try another model e.g.

llama-cli -hf bartowski/Qwen2.5-Coder-3B-Instruct-GGUF:Q6_K_L --temp 1.0 --top-p 0.95  --top-k 40 -p "briefly explain journalctl in one paragraph"

...

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time. 

 - Press Return to return control to the AI. 

 - To return control without starting a new line, end your input with '/'. 

 - If you want to submit another line, end your input with '\'. 

 - Not using system message. To change it, set a different value via -sys PROMPT 
user

briefly explain journalctl in one paragraph

assistant

Journalctl is a powerful command-line utility provided by the systemd system and service manager that allows you to view, search, and monitor the system's logs in real-time. It provides detailed information about vario

us aspects of the system, such as system boot, services, and applications, and can be used to troubleshoot issues and understand system behavior. Journalctl supports various log levels, filtering options, and can be us

ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.

Hans4687 (talk) 14:36, 15 July 2026 (UTC)

Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. Woile (talk) 14:45, 15 July 2026 (UTC)

This is not working because the model architecture is the same as before (qwen3next). The error remains.

::...
::llama_model_loader: loaded meta data with 52 key-value pairs and 843 tensors from /home/hans/.cache/llama.cpp/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-UD-Q4_K_M.gguf (version GGUF V3 (latest))
::llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
::llama_model_loader: - kv   0:                       general.architecture str              = qwen3next
::llama_model_loader: - kv   1:                               general.type str              = model
::...
::

Hans4687 (talk) 16:12, 15 July 2026 (UTC)

How much RAM do you have? (Or VRAM if GPU) Woile (talk) 16:26, 15 July 2026 (UTC)

I have 64GB ram. Hans4687 (talk) 16:37, 15 July 2026 (UTC)

And you are using unified memory, right? It's strange, because this just worked for me:

nix-shell -p llama-cpp-vulkan

llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"

I think I'll display a smaller model in the page anyways.

This is the version I'm running: version: 9925 (ed8c261)

I think yours is quite old, it's not even in stable right now. Woile (talk) 07:51, 16 July 2026 (UTC)

It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only. I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. Hans4687 (talk) 07:52, 20 July 2026 (UTC)