Talk:Llama-cpp: Difference between revisions

(5 intermediate revisions by 2 users not shown)

Line 47:

ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.

</syntaxhighlight>[[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 14:36, 15 July 2026 (UTC)

:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 14:45, 15 July 2026 (UTC)

::This is not working because the model architecture is the same as before (qwen3next). The error remains.

::<syntaxhighlight lang="mediawiki">

::...

::llama_model_loader: loaded meta data with 52 key-value pairs and 843 tensors from /home/hans/.cache/llama.cpp/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-UD-Q4_K_M.gguf (version GGUF V3 (latest))

::llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.

::llama_model_loader: - kv 0: general.architecture str = qwen3next

::llama_model_loader: - kv 1: general.type str = model

::...

::</syntaxhighlight> [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:12, 15 July 2026 (UTC)

:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 16:26, 15 July 2026 (UTC)

::::I have 64GB ram. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:37, 15 July 2026 (UTC)

:::::And you are using unified memory, right? It's strange, because this just worked for me:

:::::<code>nix-shell -p llama-cpp-vulkan</code>

:::::<code>llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"</code>

:::::I think I'll display a smaller model in the page anyways.

:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>

:::::I think yours is quite old, it's not even in stable right now. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 07:51, 16 July 2026 (UTC)

::::::It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only. I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 07:52, 20 July 2026 (UTC)

@@ Line 47: / Line 47: @@
 ed to monitor specific services, users, or applications by using various filters and selectors. It is a versatile tool that can be used to gain insights into the system's activities and operations.
 </syntaxhighlight>[[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 14:36, 15 July 2026 (UTC)
+:Thanks for reporting, I've updated the models to use `unsloth` which is more reliable and it's working. Can you give it a try? The model is aprox 50GB so it will require at least 50GB of RAM. If it fails, it might be because of that. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 14:45, 15 July 2026 (UTC)
+::This is not working because the model architecture is the same as before (qwen3next). The error remains.
+::<syntaxhighlight lang="mediawiki">
+::...
+::llama_model_loader: loaded meta data with 52 key-value pairs and 843 tensors from /home/hans/.cache/llama.cpp/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-UD-Q4_K_M.gguf (version GGUF V3 (latest))
+::llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
+::llama_model_loader: - kv   0:                       general.architecture str              = qwen3next
+::llama_model_loader: - kv   1:                               general.type str              = model
+::...
+::</syntaxhighlight> [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:12, 15 July 2026 (UTC)
+:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 16:26, 15 July 2026 (UTC)
+::::I have 64GB ram. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:37, 15 July 2026 (UTC)
+:::::And you are using unified memory, right? It's strange, because this just worked for me:
+:::::<code>nix-shell -p llama-cpp-vulkan</code>
+:::::<code>llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"</code>
+:::::I think I'll display a smaller model in the page anyways.
+:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code>
+:::::I think yours is quite old, it's not even in stable right now. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 07:51, 16 July 2026 (UTC)
+::::::It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only.  I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 07:52, 20 July 2026 (UTC)