Llama-cpp: Difference between revisions

Line 6:

* <code>llama-cpp</code>: the umbrella package, it uses the CPU if it doesn't find any GPU. On Mac Sillicon, it automatically detects that it should use the Metal backend. And for NVIDIA CUDA, you need to enable cudaSupport and unfree packages.

* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with ~~<code>~~rocmSupport ~~= true</code>~~.

* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with rocmSupport enabled.

* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with ~~<code>~~vulkanSupport ~~= true</code>~~. In some situations, it may perform even better than ROCm.

* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with vulkanSupport enabled. In some situations, it may perform even better than ROCm.

You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system

@@ Line 6: / Line 6: @@
 * <code>llama-cpp</code>: the umbrella package, it uses the CPU if it doesn't find any GPU. On Mac Sillicon, it automatically detects that it should use the Metal backend. And for NVIDIA CUDA, you need to enable cudaSupport and unfree packages.
-* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with <code>rocmSupport = true</code>.
+* <code>llama-cpp-rocm</code>: for [https://en.wikipedia.org/wiki/ROCm AMD ROCm] software stack. Under the shell, it's just <code>llama-cpp</code> with rocmSupport enabled.
-* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with <code>vulkanSupport = true</code>. In some situations, it may perform even better than ROCm.
+* <code>llama-cpp-vulkan</code>: for [https://en.wikipedia.org/wiki/Vulkan Vulkan], which works with multiple CPU's and GPU's. Under the shell, it's just <code>llama-cpp</code> with vulkanSupport enabled. In some situations, it may perform even better than ROCm.
 You can install any of the 3 in your system depending on your configuration. If your system is not covered by one of those packages, you can probably still install <code>llama-cpp</code> and with some customization make it fit your system