Intel NPU - Revision history

Ryan4yin: Checking that the NPU is used

2026-11-11T13:06:23Z

Checking that the NPU is used

← Older revision		Revision as of 13:06, 11 November 2026
Line 138:		Line 138:
	[https://github.com/k2-fsa/sherpa-onnx sherpa-onnx] (speech recognition, text-to-speech, speaker identification) uses ONNX Runtime. Set the provider to <code>openvino</code>, with <code>--provider=openvino</code> on the command line or <code>provider="openvino"</code> in the Python API; it then runs on the NPU by default.		[https://github.com/k2-fsa/sherpa-onnx sherpa-onnx] (speech recognition, text-to-speech, speaker identification) uses ONNX Runtime. Set the provider to <code>openvino</code>, with <code>--provider=openvino</code> on the command line or <code>provider="openvino"</code> in the Python API; it then runs on the NPU by default.


			=== Checking that the NPU is used ===

			Setting the device to NPU does not guarantee that a model runs there. The NPU compiles only models it supports, with static input shapes, and applications react differently when it cannot:

			* '''OpenVINO''' raises an error from <code>compile_model</code>; it does not fall back to another device.
			* '''ONNX Runtime''' runs the parts the NPU cannot take on the CPU, without an error. Set the session option <code>session.disable_cpu_ep_fallback</code> to <code>1</code> to get an error instead.
			* '''sherpa-onnx''' logs <code>Fallback to cpu</code> and continues on the CPU if the OpenVINO provider cannot be enabled. It runs only the encoder on the NPU; decoders stay on the CPU.

			To confirm that the NPU is doing the work, check that its busy time grows while the application runs:

			<syntaxhighlight lang="console">
			$ cat /sys/class/accel/accel0/device/npu_busy_time_us
			</syntaxhighlight>

	== How it works ==		== How it works ==

Ryan4yin: add apps that supports openvino/npu

2026-11-11T13:04:07Z

add apps that supports openvino/npu

← Older revision		Revision as of 13:04, 11 November 2026
Line 102:		Line 102:

	For models outside GenAI's pipelines, use the OpenVINO runtime (<code>python3Packages.openvino</code>) and pass <code>"NPU"</code> to <code>compile_model</code>, as in [[#Verification]]. The NPU needs static input shapes, so reshape a model to fixed input sizes before compiling it. See [https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html OpenVINO: NPU Device].		For models outside GenAI's pipelines, use the OpenVINO runtime (<code>python3Packages.openvino</code>) and pass <code>"NPU"</code> to <code>compile_model</code>, as in [[#Verification]]. The NPU needs static input shapes, so reshape a model to fixed input sizes before compiling it. See [https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html OpenVINO: NPU Device].

			== Applications ==

			=== Frigate ===

			[https://frigate.video/ Frigate] runs object detection on the NPU with its OpenVINO detector:

			<syntaxhighlight lang="nix">
			services.frigate.settings.detectors.ov = {
			type = "openvino";
			device = "NPU";
			};
			</syntaxhighlight>

			The NixOS module grants the service access to the device and allows the runtime code generation OpenVINO needs. See [https://docs.frigate.video/configuration/object_detectors/#openvino-detector Frigate: OpenVINO Detector].

			=== ONNX Runtime ===

			<code>onnxruntime</code> and <code>python3Packages.onnxruntime</code> are built with the OpenVINO execution provider on Linux. Select it with <code>device_type</code> set to <code>NPU</code>:

			<syntaxhighlight lang="python">
			import onnxruntime as ort

			session = ort.InferenceSession(
			"model.onnx",
			providers=["OpenVINOExecutionProvider"],
			provider_options=[{"device_type": "NPU"}],
			)
			</syntaxhighlight>

			Applications built on ONNX Runtime can use the NPU this way if they let you choose the execution provider. See [https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html ONNX Runtime: OpenVINO Execution Provider].

			=== sherpa-onnx ===

			[https://github.com/k2-fsa/sherpa-onnx sherpa-onnx] (speech recognition, text-to-speech, speaker identification) uses ONNX Runtime. Set the provider to <code>openvino</code>, with <code>--provider=openvino</code> on the command line or <code>provider="openvino"</code> in the Python API; it then runs on the NPU by default.


	== How it works ==		== How it works ==

Ryan4yin: /* See also */

2026-11-11T12:41:53Z

Ryan4yin at 12:40, 11 November 2026

2026-11-11T12:40:58Z

← Older revision		Revision as of 12:40, 11 November 2026
Line 68:		Line 68:
	The workflow is:		The workflow is:

	# Get a model in OpenVINO format. The [https://huggingface.co/OpenVINO OpenVINO organization on Hugging Face] publishes converted models, including ~~a collection of~~ [https://huggingface.co/collections/OpenVINO/llms-optimized-for-npu-686e7f0bf7bc184bd71f8ba0 LLMs optimized for NPU]~~. Download one with <code>hf download</code> from <code>python3Packages.huggingface-hub</code>~~.		# Get a model in OpenVINO format. The [https://huggingface.co/OpenVINO OpenVINO organization on Hugging Face] publishes converted models, including [https://huggingface.co/collections/OpenVINO/vlms-optimized-for-npu VLMs optimized for NPU] and [https://huggingface.co/collections/OpenVINO/llms-optimized-for-npu-686e7f0bf7bc184bd71f8ba0 LLMs optimized for NPU].
	# Load it with the matching pipeline on <code>"NPU"</code>, and set <code>CACHE_DIR</code>. Compiling a large model for the NPU can take over a minute; with the cache, later runs load the compiled model in seconds.		# Load it with the matching pipeline on <code>"NPU"</code>, and set <code>CACHE_DIR</code>. Compiling a large model for the NPU can take over a minute; with the cache, later runs load the compiled model in seconds.

Ryan4yin: init

2026-11-11T12:26:31Z

init

New page

The '''Intel NPU''' (''Intel AI Boost'') is the neural processing unit in Intel Core Ultra processors. Applications use it through [https://docs.openvino.ai/ OpenVINO].

{{Note|Requires NixOS 26.11 or newer, or a <code>nixos-unstable</code> revision that includes [https://github.com/NixOS/nixpkgs/pull/570230 NixOS/nixpkgs#570230]. Older releases cannot compile models for the NPU.}}

== Setup ==

<syntaxhighlight lang="nix">
hardware.cpu.intel.npu.enable = true;
</syntaxhighlight>

== Verification ==

Check each layer in order; the first one that fails is where the problem is.

The NPU is a PCI device of class <code>1200</code>, bound to the <code>intel_vpu</code> kernel driver:

<syntaxhighlight lang="console">
$ lspci -knnd ::1200
00:0b.0 Processing accelerators [1200]: Intel Corporation Core Ultra 200 Series Processors NPU [8086:ad1d] (rev 01)
...
Kernel driver in use: intel_vpu
Kernel modules: intel_vpu
$ journalctl -kg intel_vpu
...
[drm] Initialized intel_vpu 1.0.0 for 0000:00:0b.0 on minor 0
$ ls -l /dev/accel/
crw-rw-rw- 1 root render 261, 0 ... accel0
</syntaxhighlight>

The userspace driver is installed where Level Zero looks for it:

<syntaxhighlight lang="console">
$ ls /run/opengl-driver/lib | grep ze_intel_npu
libze_intel_npu.so
libze_intel_npu.so.1
libze_intel_npu.so.1.38.0
</syntaxhighlight>

OpenVINO sees the NPU:

<syntaxhighlight lang="console">
$ nix-shell -p 'python3.withPackages (ps: [ ps.openvino ])' --run \
"python -c 'import openvino as ov; c = ov.Core(); print(c.available_devices, c.get_property(\"NPU\", \"FULL_DEVICE_NAME\"))'"
['CPU', 'GPU', 'NPU'] Intel(R) AI Boost
</syntaxhighlight>

A model compiles and runs on the NPU:

<syntaxhighlight lang="python">
import numpy as np
import openvino as ov
import openvino.opset13 as ops

x = ops.parameter([1, 16], np.float32, name="x")
model = ov.Model([ops.relu(ops.add(x, ops.constant(np.ones((1, 16), np.float32))))], [x])

compiled = ov.Core().compile_model(model, "NPU")
print(compiled(np.arange(-8, 8, dtype=np.float32).reshape(1, 16))[0])
# [[0. 0. 0. 0. 0. 0. 0. 0. 1. 2. 3. 4. 5. 6. 7. 8.]]
</syntaxhighlight>

If the driver library is missing, <code>hardware.cpu.intel.npu.enable</code> is not active yet; switch to the new configuration. If OpenVINO lists no <code>NPU</code> although the earlier checks pass, the process may not see <code>/dev/accel/accel0</code>; see [[#Systemd services]].

== Usage ==

The usual way to run models on the NPU is [https://github.com/openvinotoolkit/openvino.genai OpenVINO GenAI] (<code>python3Packages.openvino-genai</code>). It provides ready-made pipelines for common tasks: <code>LLMPipeline</code> for text generation, <code>VLMPipeline</code> for vision-language models, <code>WhisperPipeline</code> for speech recognition, and others. Each pipeline takes a model directory and a device name; pass <code>"NPU"</code>.

The workflow is:

# Get a model in OpenVINO format. The [https://huggingface.co/OpenVINO OpenVINO organization on Hugging Face] publishes converted models, including a collection of [https://huggingface.co/collections/OpenVINO/llms-optimized-for-npu-686e7f0bf7bc184bd71f8ba0 LLMs optimized for NPU]. Download one with <code>hf download</code> from <code>python3Packages.huggingface-hub</code>.
# Load it with the matching pipeline on <code>"NPU"</code>, and set <code>CACHE_DIR</code>. Compiling a large model for the NPU can take over a minute; with the cache, later runs load the compiled model in seconds.

To convert other Hugging Face models, use Optimum Intel's <code>optimum-cli export openvino</code>. It is not packaged in nixpkgs, and LLMs need specific quantization flags for the NPU; see [https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html OpenVINO GenAI on NPU].

=== Example: speech recognition ===

Download a Whisper model:

<syntaxhighlight lang="console">
$ nix-shell -p python3Packages.huggingface-hub --run \
'hf download OpenVINO/whisper-large-v3-turbo-int8-ov --local-dir whisper-large-v3-turbo-int8-ov'
</syntaxhighlight>

Transcribe a WAV file in 16 kHz, mono, 16-bit format:

<syntaxhighlight lang="python">
import wave

import numpy as np
import openvino_genai

with wave.open("speech.wav") as f:
samples = np.frombuffer(f.readframes(f.getnframes()), np.int16).astype(np.float32) / 32768

pipe = openvino_genai.WhisperPipeline("whisper-large-v3-turbo-int8-ov", "NPU", CACHE_DIR="npu-cache")
print(pipe.generate(samples).texts[0])
</syntaxhighlight>

The first run compiles the model for the NPU, which takes about 70 seconds on an Arrow Lake NPU; later runs start in seconds.

=== Using the OpenVINO runtime directly ===

For models outside GenAI's pipelines, use the OpenVINO runtime (<code>python3Packages.openvino</code>) and pass <code>"NPU"</code> to <code>compile_model</code>, as in [[#Verification]]. The NPU needs static input shapes, so reshape a model to fixed input sizes before compiling it. See [https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html OpenVINO: NPU Device].

== How it works ==

The kernel's <code>intel_vpu</code> driver exposes the NPU as <code>/dev/accel/accel0</code>. The option above adds the NPU firmware and puts the userspace driver <code>intel-npu-driver</code> in <code>/run/opengl-driver/lib</code>, where OpenVINO finds it through Level Zero. The driver then loads the NPU compiler from the <code>openvino-intel-npu-compiler</code> package.

== Systemd services ==

A sandboxed service needs access to the accel device, and OpenVINO generates code at runtime:

<syntaxhighlight lang="nix">
systemd.services.<name>.serviceConfig = {
PrivateDevices = false;
DeviceAllow = [ "char-accel rw" ];
MemoryDenyWriteExecute = false;
};
</syntaxhighlight>

== See also ==

* [https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html OpenVINO: NPU Device]
* [https://github.com/intel/linux-npu-driver Intel linux-npu-driver]

[[Category:Hardware]]

← Older revision		Revision as of 12:41, 11 November 2026
Line 123:		Line 123:
	* [https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html OpenVINO: NPU Device]		* [https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html OpenVINO: NPU Device]
	* [https://github.com/intel/linux-npu-driver Intel linux-npu-driver]		* [https://github.com/intel/linux-npu-driver Intel linux-npu-driver]
			* [https://github.com/openvinotoolkit/openvino.genai OpenVINO GenAI]

	[[Category:Hardware]]		[[Category:Hardware]]