Intel NPU

The Intel NPU (Intel AI Boost) is the neural processing unit in Intel Core Ultra processors. Applications use it through OpenVINO.

Note: Requires NixOS 26.11 or newer, or a nixos-unstable revision that includes NixOS/nixpkgs#570230. Older releases cannot compile models for the NPU.

Setup

hardware.cpu.intel.npu.enable = true;

Verification

Check each layer in order; the first one that fails is where the problem is.

The NPU is a PCI device of class 1200, bound to the intel_vpu kernel driver:

$ lspci -knnd ::1200
00:0b.0 Processing accelerators [1200]: Intel Corporation Core Ultra 200 Series Processors NPU [8086:ad1d] (rev 01)
	...
	Kernel driver in use: intel_vpu
	Kernel modules: intel_vpu
$ journalctl -kg intel_vpu
...
[drm] Initialized intel_vpu 1.0.0 for 0000:00:0b.0 on minor 0
$ ls -l /dev/accel/
crw-rw-rw- 1 root render 261, 0 ... accel0

The userspace driver is installed where Level Zero looks for it:

$ ls /run/opengl-driver/lib | grep ze_intel_npu
libze_intel_npu.so
libze_intel_npu.so.1
libze_intel_npu.so.1.38.0

OpenVINO sees the NPU:

$ nix-shell -p 'python3.withPackages (ps: [ ps.openvino ])' --run \
    "python -c 'import openvino as ov; c = ov.Core(); print(c.available_devices, c.get_property(\"NPU\", \"FULL_DEVICE_NAME\"))'"
['CPU', 'GPU', 'NPU'] Intel(R) AI Boost

A model compiles and runs on the NPU:

import numpy as np
import openvino as ov
import openvino.opset13 as ops

x = ops.parameter([1, 16], np.float32, name="x")
model = ov.Model([ops.relu(ops.add(x, ops.constant(np.ones((1, 16), np.float32))))], [x])

compiled = ov.Core().compile_model(model, "NPU")
print(compiled(np.arange(-8, 8, dtype=np.float32).reshape(1, 16))[0])
# [[0. 0. 0. 0. 0. 0. 0. 0. 1. 2. 3. 4. 5. 6. 7. 8.]]

If the driver library is missing, hardware.cpu.intel.npu.enable is not active yet; switch to the new configuration. If OpenVINO lists no NPU although the earlier checks pass, the process may not see /dev/accel/accel0; see #Systemd services.

Usage

The usual way to run models on the NPU is OpenVINO GenAI (python3Packages.openvino-genai). It provides ready-made pipelines for common tasks: LLMPipeline for text generation, VLMPipeline for vision-language models, WhisperPipeline for speech recognition, and others. Each pipeline takes a model directory and a device name; pass "NPU".

The workflow is:

Get a model in OpenVINO format. The OpenVINO organization on Hugging Face publishes converted models, including VLMs optimized for NPU and LLMs optimized for NPU.
Load it with the matching pipeline on "NPU", and set CACHE_DIR. Compiling a large model for the NPU can take over a minute; with the cache, later runs load the compiled model in seconds.

To convert other Hugging Face models, use Optimum Intel's optimum-cli export openvino. It is not packaged in nixpkgs, and LLMs need specific quantization flags for the NPU; see OpenVINO GenAI on NPU.

Example: speech recognition

Download a Whisper model:

$ nix-shell -p python3Packages.huggingface-hub --run \
    'hf download OpenVINO/whisper-large-v3-turbo-int8-ov --local-dir whisper-large-v3-turbo-int8-ov'

Transcribe a WAV file in 16 kHz, mono, 16-bit format:

import wave

import numpy as np
import openvino_genai

with wave.open("speech.wav") as f:
    samples = np.frombuffer(f.readframes(f.getnframes()), np.int16).astype(np.float32) / 32768

pipe = openvino_genai.WhisperPipeline("whisper-large-v3-turbo-int8-ov", "NPU", CACHE_DIR="npu-cache")
print(pipe.generate(samples).texts[0])

The first run compiles the model for the NPU, which takes about 70 seconds on an Arrow Lake NPU; later runs start in seconds.

Using the OpenVINO runtime directly

For models outside GenAI's pipelines, use the OpenVINO runtime (python3Packages.openvino) and pass "NPU" to compile_model, as in #Verification. The NPU needs static input shapes, so reshape a model to fixed input sizes before compiling it. See OpenVINO: NPU Device.

Applications

Frigate

Frigate runs object detection on the NPU with its OpenVINO detector:

services.frigate.settings.detectors.ov = {
  type = "openvino";
  device = "NPU";
};

The NixOS module grants the service access to the device and allows the runtime code generation OpenVINO needs. See Frigate: OpenVINO Detector.

ONNX Runtime

onnxruntime and python3Packages.onnxruntime are built with the OpenVINO execution provider on Linux. Select it with device_type set to NPU:

import onnxruntime as ort

session = ort.InferenceSession(
    "model.onnx",
    providers=["OpenVINOExecutionProvider"],
    provider_options=[{"device_type": "NPU"}],
)

Applications built on ONNX Runtime can use the NPU this way if they let you choose the execution provider. See ONNX Runtime: OpenVINO Execution Provider.

sherpa-onnx

sherpa-onnx (speech recognition, text-to-speech, speaker identification) uses ONNX Runtime. Set the provider to openvino, with --provider=openvino on the command line or provider="openvino" in the Python API; it then runs on the NPU by default.

Checking that the NPU is used

Setting the device to NPU does not guarantee that a model runs there. The NPU compiles only models it supports, with static input shapes, and applications react differently when it cannot:

OpenVINO raises an error from compile_model; it does not fall back to another device.
ONNX Runtime runs the parts the NPU cannot take on the CPU, without an error. Set the session option session.disable_cpu_ep_fallback to 1 to get an error instead.
sherpa-onnx logs Fallback to cpu and continues on the CPU if the OpenVINO provider cannot be enabled. It runs only the encoder on the NPU; decoders stay on the CPU.

To confirm that the NPU is doing the work, check that its busy time grows while the application runs:

$ cat /sys/class/accel/accel0/device/npu_busy_time_us

How it works

The kernel's intel_vpu driver exposes the NPU as /dev/accel/accel0. The option above adds the NPU firmware and puts the userspace driver intel-npu-driver in /run/opengl-driver/lib, where OpenVINO finds it through Level Zero. The driver then loads the NPU compiler from the openvino-intel-npu-compiler package.

Systemd services

A sandboxed service needs access to the accel device, and OpenVINO generates code at runtime:

systemd.services.<name>.serviceConfig = {
  PrivateDevices = false;
  DeviceAllow = [ "char-accel rw" ];
  MemoryDenyWriteExecute = false;
};