Llama-cpp: Difference between revisions
No edit summary |
titles improved |
||
| Line 11: | Line 11: | ||
You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system | You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system | ||
== | == Customization == | ||
=== Nvidia CUDA === | === Nvidia CUDA === | ||
| Line 17: | Line 17: | ||
Nvidia CUDA contains [[Unfree software]], so you have to enable it first, either in your NixOS configuration or via environmental variables. | Nvidia CUDA contains [[Unfree software]], so you have to enable it first, either in your NixOS configuration or via environmental variables. | ||
==== NixOS ==== | ==== in NixOS ==== | ||
After enable Unfree software in NixOS add CUDA to your packages | After enable Unfree software in NixOS add CUDA to your packages | ||
| Line 33: | Line 33: | ||
</pre> | </pre> | ||
==== | ==== in a shell ==== | ||
If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell: | If you want take the CUDA package for a spin, before adding it to your system, you can open it in a shell: | ||
| Line 52: | Line 52: | ||
</pre> | </pre> | ||
== | == Models == | ||
When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model. | When usage <code>llama-cli</code> or <code>llama-server</code>, you can tune the parameters of the model. | ||