Ollama - Revision history

Mathtician: Updated syntax for systemd service config (acceleration -> package); have not tested standalone package config

2026-01-26T19:49:04Z

Updated syntax for systemd service config (acceleration -> package); have not tested standalone package config

← Older revision		Revision as of 19:49, 26 January 2026
Line 19:		Line 19:

	== Configuration of GPU acceleration ==		== Configuration of GPU acceleration ==
	~~Its possible to use following values for acceleration~~:		Acceleration is configured by selecting a package:
	* ~~false~~: disable GPU, only use CPU		* ollama-cpu: disable GPU, only use CPU
	* "rocm": supported by most modern AMD GPUs		* ollama-rocm: supported by most modern AMD GPUs
	* "cuda": supported by most modern NVIDIA GPUs		* ollama-cuda: supported by most modern NVIDIA GPUs
	* "vulkan": supported by most modern GPUs on Linux		* ollama-vulkan: supported by most modern GPUs on Linux


Line 41:		Line 41:
	services.ollama = {		services.ollama = {
	enable = true;		enable = true;
	~~acceleration~~ = "cuda";		package = pkgs.ollama-cuda;
	};		};
	</syntaxhighlight>		</syntaxhighlight>
Line 128:		Line 128:
	services.ollama = {		services.ollama = {
	enable = true;		enable = true;
	~~acceleration~~ = "rocm";		package = pkgs.ollama-rocm;
	environmentVariables = {		environmentVariables = {
	HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore		HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore

Rebuilt: /* Configuration of GPU acceleration */ Document vulkan option

2025-12-01T04:34:35Z

Configuration of GPU acceleration: Document vulkan option

← Older revision		Revision as of 04:34, 1 December 2025
Line 23:		Line 23:
	* "rocm": supported by most modern AMD GPUs		* "rocm": supported by most modern AMD GPUs
	* "cuda": supported by most modern NVIDIA GPUs		* "cuda": supported by most modern NVIDIA GPUs
			* "vulkan": supported by most modern GPUs on Linux

Kini: fix ref link

2025-11-20T16:25:23Z

fix ref link

← Older revision		Revision as of 16:25, 20 November 2025
Line 109:		Line 109:
	In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.		In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.

	However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides</ref>		However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.mdx#overrides-on-linux</ref>

	You can get the version for your GPU from the logs or like so:		You can get the version for your GPU from the logs or like so:

Chafey at 18:41, 6 November 2025

2025-11-06T18:41:43Z

@@ Line 94: / Line 94: @@
 == Troubleshooting ==
 === AMD GPU with open source driver ===
 In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.

Yuannan: This has been changed so that it actually overrides the proper env vars. The last format shouldn't work as the convention is to use 10.3.0 for other rocm devices.

2025-04-20T19:05:29Z

This has been changed so that it actually overrides the proper env vars. The last format shouldn't work as the convention is to use 10.3.0 for other rocm devices.

← Older revision		Revision as of 19:05, 20 April 2025
Line 119:		Line 119:
	HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore		HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore
	};		};
	# results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3.1"		# results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3.0"
	rocmOverrideGfx = "10.3.1";		rocmOverrideGfx = "10.3.0";
	};		};
	</syntaxhighlight>		</syntaxhighlight>
Line 126:		Line 126:
	or via an environment variable in front of the standalone app		or via an environment variable in front of the standalone app
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	HSA_OVERRIDE_GFX_VERSION=10.3.1 ollama serve		HSA_OVERRIDE_GFX_VERSION=10.3.0 ollama serve
	</syntaxhighlight>		</syntaxhighlight>

Malteneuss: Fix example

2025-04-04T10:37:33Z

Fix example

← Older revision		Revision as of 10:37, 4 April 2025
Line 104:		Line 104:
	# classical		# classical
	$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" \| grep "gfx"		$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" \| grep "gfx"
			Name: gfx1031

	# flakes		# flakes
	$ nix run nixpkgs#"rocmPackages.rocminfo" -- --run "rocminfo" \| grep "gfx"		$ nix run nixpkgs#"rocmPackages.rocminfo" -- --run "rocminfo" \| grep "gfx"

Malteneuss: Add rocm nix flake example

2025-04-04T10:36:47Z

Add rocm nix flake example

← Older revision		Revision as of 10:36, 4 April 2025
Line 102:		Line 102:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
			# classical
	$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" \| grep "gfx"		$ nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" \| grep "gfx"
			# flakes
			$ nix run nixpkgs#"rocmPackages.rocminfo" -- --run "rocminfo" \| grep "gfx"
	Name: gfx1031		Name: gfx1031
	</syntaxhighlight>		</syntaxhighlight>

Malteneuss: Add link to ollama library

2025-04-04T10:29:52Z

Add link to ollama library

← Older revision		Revision as of 10:29, 4 April 2025
Line 13:		Line 13:
	services.ollama = {		services.ollama = {
	enable = true;		enable = true;
	# Optional: ~~load~~ models ~~on startup~~		# Optional: preload models, see https://ollama.com/library
	loadModels = [ ... ];		loadModels = [ "llama3.2:3b" "deepseek-r1:1.5b"];
	};		};
	</syntaxhighlight>		</syntaxhighlight>

Malteneuss: Add standalone amd override hint

2025-03-19T22:03:06Z

Add standalone amd override hint

← Older revision		Revision as of 22:03, 19 March 2025
Line 87:		Line 87:

	== Usage via web API ==		== Usage via web API ==
	Other software can use the web API (default at: http://localhost:11434 ) to query ~~ollama~~. This works well e.g. in Intellij-IDEs with the "ProxyAI" and the "Ollama Commit Summarizer" plugins.		Other software can use the web API (default at: http://localhost:11434 ) to query Ollama. This works well e.g. in Intellij-IDEs with the "ProxyAI" and the "Ollama Commit Summarizer" plugins.

	Alternatively, on enabling "open-webui", a web portal is available at: http://localhost:8080/:		Alternatively, on enabling "open-webui", a web portal is available at: http://localhost:8080/:
Line 95:		Line 95:
	=== AMD GPU with open source driver ===		=== AMD GPU with open source driver ===

	In certain cases ~~ollama~~ might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.		In certain cases Ollama might not allow your system to use GPU acceleration if it cannot be sure your GPU/driver is compatible.

	However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides</ref>		However you can attempt to force-enable the usage of your GPU by overriding the LLVM target. <ref>https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides</ref>
Line 106:		Line 106:
	</syntaxhighlight>		</syntaxhighlight>

	In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for ~~ollama~~:		In this example the LLVM target is "gfx1031", that is, version "10.3.1", you can then override that value for Ollama for the systemd service:
	<syntaxhighlight lang="nix">		<syntaxhighlight lang="nix">
	services.ollama = {		services.ollama = {
Line 114:		Line 114:
	HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore		HCC_AMDGPU_TARGET = "gfx1031"; # used to be necessary, but doesn't seem to anymore
	};		};
			# results in environment variable "HSA_OVERRIDE_GFX_VERSION=10.3.1"
	rocmOverrideGfx = "10.3.1";		rocmOverrideGfx = "10.3.1";
	};		};
	</syntaxhighlight>		</syntaxhighlight>

			or via an environment variable in front of the standalone app
			<syntaxhighlight lang="bash">
			HSA_OVERRIDE_GFX_VERSION=10.3.1 ollama serve
			</syntaxhighlight>

	If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here].		If there are still errors, you can attempt to set a similar value that is listed [https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides here].

Malteneuss: Add verbose flag hint

2025-03-19T21:58:35Z

Add verbose flag hint

← Older revision		Revision as of 21:58, 19 March 2025
Line 46:		Line 46:
	To find out whether a model is running on CPU or GPU, you can either		To find out whether a model is running on CPU or GPU, you can either
	look at the logs of		look at the logs of
	ollama serve		<syntaxhighlight lang="bash">
			$ ollama serve
			</syntaxhighlight>
			and search for "looking for compatible GPUs" and "new model will fit in available VRAM in single GPU, loading"

	or while a model is answering run		or while a model is answering run in another terminal
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	$ ollama ps		$ ollama ps
Line 66:		Line 69:
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	$ ollama run codellama:13b-instruct "Write an extended Python program with a typical structure. It should print the numbers 1 to 10 to standard output."		$ ollama run codellama:13b-instruct "Write an extended Python program with a typical structure. It should print the numbers 1 to 10 to standard output."
			</syntaxhighlight>

			=== See usage and speed statistics ===
			Add "--verbose" to see statistics after each prompt:
			<syntaxhighlight lang="bash">
			$ ollama run codellama:13b-instruct --verbose "Write an extended Python program..."
			...
			total duration: 50.302071991s
			load duration: 50.912267ms
			prompt eval count: 49 token(s)
			prompt eval duration: 4.654s
			prompt eval rate: 10.53 tokens/s <- how fast it processed your input prompt
			eval count: 182 token(s)
			eval duration: 45.595s
			eval rate: 3.99 tokens/s <- how fast it printed a response
	</syntaxhighlight>		</syntaxhighlight>