Talk:Llama-cpp: Difference between revisions

(6 intermediate revisions by 2 users not shown)

Line 6:

Thanks [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 12:36, 19 May 2026 (UTC)

== Command llama-cli gives error ==

I tried this command on nix-shell with llama-cpp version: 6981 (647b960).

Error:

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen3next' llama_model_load_from_file_impl: failed to load model

</syntaxhighlight>

The model is not supported (not yet) bij llama-cpp

If I try another model e.g.

llama-cli -hf bartowski/Qwen2.5-Coder-3B-Instruct-GGUF:Q6_K_L --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"

...

== Running in interactive mode. ==

- Press Ctrl+C to interject at any time.

- Press Return to return control to the AI.

- To return control without starting a new line, end your input with '/'.

- If you want to submit another line, end your input with '\'.

- Not using system message. To change it, set a different value via -sys PROMPT

user

briefly explain journalctl in one paragraph

assistant

Journalctl is a powerful command-line utility provided by the systemd system and service manager that allows you to view, search, and monitor the system's logs in real-time. It provides detailed information about vario

us aspects of the system, such as system boot, services, and applications, and can be used to troubleshoot issues and understand system behavior. Journalctl supports various log levels, filtering options, and can be us

ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.

</syntaxhighlight>[[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 14:36, 15 July 2026 (UTC)

:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 14:45, 15 July 2026 (UTC)

::This is not working because the model architecture is the same as before (qwen3next). The error remains.

::<syntaxhighlight lang="mediawiki">

::...

::llama_model_loader: loaded meta data with 52 key-value pairs and 843 tensors from /home/hans/.cache/llama.cpp/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-UD-Q4_K_M.gguf (version GGUF V3 (latest))

::llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.

::llama_model_loader: - kv 0: general.architecture str = qwen3next

::llama_model_loader: - kv 1: general.type str = model

::...

::</syntaxhighlight> [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:12, 15 July 2026 (UTC)

:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 16:26, 15 July 2026 (UTC)

::::I have 64GB ram. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:37, 15 July 2026 (UTC)

:::::And you are using unified memory, right? It's strange, because this just worked for me:

:::::<code>nix-shell -p llama-cpp-vulkan</code>

:::::<code>llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"</code>

:::::I think I'll display a smaller model in the page anyways.

:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>

:::::I think yours is quite old, it's not even in stable right now. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 07:51, 16 July 2026 (UTC)

::::::It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only. I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 07:52, 20 July 2026 (UTC)

@@ Line 6: / Line 6: @@
 Thanks [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 12:36, 19 May 2026 (UTC)
+== Command llama-cli gives error ==
+I tried this command on nix-shell with llama-cpp  version: 6981 (647b960).
+Error:
+<syntaxhighlight lang="mediawiki">
+llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen3next' llama_model_load_from_file_impl: failed to load model
+</syntaxhighlight>
+The model is not supported (not yet)  bij llama-cpp
+If I try another model e.g.
+<syntaxhighlight lang="mediawiki">
+llama-cli -hf bartowski/Qwen2.5-Coder-3B-Instruct-GGUF:Q6_K_L --temp 1.0 --top-p 0.95  --top-k 40 -p "briefly explain journalctl in one paragraph"
+...
+== Running in interactive mode. ==
+ - Press Ctrl+C to interject at any time.
+ - Press Return to return control to the AI.
+ - To return control without starting a new line, end your input with '/'.
+ - If you want to submit another line, end your input with '\'.
+ - Not using system message. To change it, set a different value via -sys PROMPT
+user
+briefly explain journalctl in one paragraph
+assistant
+Journalctl is a powerful command-line utility provided by the systemd system and service manager that allows you to view, search, and monitor the system's logs in real-time. It provides detailed information about vario
+us aspects of the system, such as system boot, services, and applications, and can be used to troubleshoot issues and understand system behavior. Journalctl supports various log levels, filtering options, and can be us
+ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.
+</syntaxhighlight>[[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 14:36, 15 July 2026 (UTC)
+:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 14:45, 15 July 2026 (UTC)
+::This is not working because the model architecture is the same as before (qwen3next). The error remains.
+::<syntaxhighlight lang="mediawiki">
+::...
+::llama_model_loader: loaded meta data with 52 key-value pairs and 843 tensors from /home/hans/.cache/llama.cpp/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-UD-Q4_K_M.gguf (version GGUF V3 (latest))
+::llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
+::llama_model_loader: - kv   0:                       general.architecture str              = qwen3next
+::llama_model_loader: - kv   1:                               general.type str              = model
+::...
+::</syntaxhighlight> [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:12, 15 July 2026 (UTC)
+:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 16:26, 15 July 2026 (UTC)
+::::I have 64GB ram. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:37, 15 July 2026 (UTC)
+:::::And you are using unified memory, right? It's strange, because this just worked for me:
+:::::<code>nix-shell -p llama-cpp-vulkan</code>
+:::::<code>llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"</code>
+:::::I think I'll display a smaller model in the page anyways.
+:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>
+:::::I think yours is quite old, it's not even in stable right now. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 07:51, 16 July 2026 (UTC)
+::::::It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only.  I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 07:52, 20 July 2026 (UTC)