Talk:Llama-cpp: Difference between revisions
| (3 intermediate revisions by 2 users not shown) | |||
| Line 58: | Line 58: | ||
::... | ::... | ||
::</syntaxhighlight> [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:12, 15 July 2026 (UTC) | ::</syntaxhighlight> [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:12, 15 July 2026 (UTC) | ||
:::How much RAM do you have? (Or VRAM if GPU) [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 16:26, 15 July 2026 (UTC) | |||
::::I have 64GB ram. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 16:37, 15 July 2026 (UTC) | |||
:::::And you are using unified memory, right? It's strange, because this just worked for me: | |||
:::::<code>nix-shell -p llama-cpp-vulkan</code> | |||
:::::<code>llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M --temp 1.0 --top-p 0.95 --top-k 40 -p "briefly explain journalctl in one paragraph"</code> | |||
:::::I think I'll display a smaller model in the page anyways. | |||
:::::This is the version I'm running: <code>version: 9925 (ed8c261)</code> | |||
:::::I think yours is quite old, it's not even in stable right now. [[User:Woile|Woile]] ([[User talk:Woile|talk]]) 07:51, 16 July 2026 (UTC) | |||
::::::It is good to have a smaller model because its easier for trying it with lesser hardware. Currently I use 64GB RAM CPU based only. I got the model:Qwen3-Coder-Next:latest (51GB) working with ollama using the ollama service. [[User:Hans4687|Hans4687]] ([[User talk:Hans4687|talk]]) 07:52, 20 July 2026 (UTC) | |||