CUDA
NixOS supports using NVIDIA GPUs for pure computing purposes, not just for graphics. For example, many users rely on NixOS for machine learning both locally and on cloud instances. These use cases are supported by the @NixOS/cuda-maintainers team on GitHub (project board). If you have an issue using your NVIDIA GPU for computing purposes open an issue on GitHub and tag @NixOS/cuda-maintainers.
Driver Installation
Assuming you've followed the NVIDIA page correctly, and have a CUDA compatible GPU, you shouldn't need to do any further configuration. You can confirm your CUDA version by running the following command in your terminal.
$ nvidia-smi | grep CUDA
cudatoolkit, cudnn, and related packages
The CUDA toolkit is available in a number of different versions. Please use the latest major version. You can see where they're defined in nixpkgs here.
Several "CUDA-X" libraries are packages as well. In particular,
There are some possible ways to setup a development environment using CUDA on NixOS. This can be accomplished in the following ways:
- By making a FHS user env
# flake.nix, run with `nix develop`
# Run with `nix-shell cuda-fhs.nix`
{ pkgs ? import <nixpkgs> {} }:
let
# Change according to the driver used: stable, beta
nvidiaPackage = pkgs.linuxPackages.nvidiaPackages.stable;
in
(pkgs.buildFHSEnv {
name = "cuda-env";
targetPkgs = pkgs: with pkgs; [
git
gitRepo
gnupg
autoconf
curl
procps
gnumake
util-linux
m4
gperf
unzip
cudatoolkit
nvidiaPackage
libGLU libGL
xorg.libXi xorg.libXmu freeglut
xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib
ncurses5
stdenv.cc
binutils
];
multiPkgs = pkgs: with pkgs; [ zlib ];
runScript = "bash";
profile = ''
export CUDA_PATH=${pkgs.cudatoolkit}
# export LD_LIBRARY_PATH=${nvidiaPackage}/lib
export EXTRA_LDFLAGS="-L/lib -L${nvidiaPackage}/lib"
export EXTRA_CCFLAGS="-I/usr/include"
'';
}).env
- By making a nix-shell
# flake.nix, run with `nix develop`
# Run with `nix-shell cuda-shell.nix`
{ pkgs ? import <nixpkgs> {} }:
let
nvidiaPackage = pkgs.linuxPackages.nvidiaPackages.stable;
in
pkgs.mkShell {
name = "cuda-env-shell";
buildInputs = with pkgs; [
git gitRepo gnupg autoconf curl
procps gnumake util-linux m4 gperf unzip
cudatoolkit nvidiaPackage
libGLU libGL
xorg.libXi xorg.libXmu freeglut
xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib
ncurses5 stdenv.cc binutils
];
shellHook = ''
export CUDA_PATH=${pkgs.cudatoolkit}
# export LD_LIBRARY_PATH=${nvidiaPackage}/lib:${pkgs.ncurses}/lib
export EXTRA_LDFLAGS="-L/lib -L${nvidiaPackage}/lib"
export EXTRA_CCFLAGS="-I/usr/include"
'';
}
- By making a flake.nix
# flake.nix, run with `nix develop`
{
description = "CUDA development environment";
outputs = {
self,
nixpkgs,
}: let
system = "x86_64-linux";
pkgs = import nixpkgs {
inherit system;
config.allowUnfree = true;
config.cudaSupport = true;
config.cudaVersion = "12";
};
# Change according to the driver used: stable, beta
nvidiaPackage = pkgs.linuxPackages.nvidiaPackages.stable;
in {
# alejandra is a nix formatter with a beautiful output
formatter."${system}" = nixpkgs.legacyPackages.${system}.alejandra;
devShells.${system}.default = pkgs.mkShell {
buildInputs = with pkgs; [
ffmpeg
fmt.dev
cudaPackages.cuda_cudart
cudatoolkit
nvidiaPackage
cudaPackages.cudnn
libGLU
libGL
xorg.libXi
xorg.libXmu
freeglut
xorg.libXext
xorg.libX11
xorg.libXv
xorg.libXrandr
zlib
ncurses
stdenv.cc
binutils
uv
];
shellHook = ''
export LD_LIBRARY_PATH="${nvidiaPackage}/lib:$LD_LIBRARY_PATH"
export CUDA_PATH=${pkgs.cudatoolkit}
export EXTRA_LDFLAGS="-L/lib -L${nvidiaPackage}/lib"
export EXTRA_CCFLAGS="-I/usr/include"
export CMAKE_PREFIX_PATH="${pkgs.fmt.dev}:$CMAKE_PREFIX_PATH"
export PKG_CONFIG_PATH="${pkgs.fmt.dev}/lib/pkgconfig:$PKG_CONFIG_PATH"
'';
};
};
}
Setting up CUDA Binary Cache
CUDA binary caches provide pre-built CUDA packages. When a matching store path is available, Nix can download it instead of building the package locally, saving time and processing power.
For more information, refer to the Using a binary cache page.
NixOS CUDA cache
NixOS
Add the cache to substituters and trusted-public-keys inside your system configuration:
nix.settings = {
substituters = [
"https://cache.nixos-cuda.org"
];
trusted-public-keys = [
"cache.nixos-cuda.org:74DUi4Ye579gUqzH4ziL9IyiJBlDpMRn9MBN8oNan9M="
];
};
Non-NixOS
You have to add substituters and trusted-public-keys to /etc/nix/nix.conf:
trusted-public-keys = cache.nixos-cuda.org:74DUi4Ye579gUqzH4ziL9IyiJBlDpMRn9MBN8oNan9M=
trusted-substituters = https://cache.nixos-cuda.org
trusted-users = root @wheel
If your user is in trusted-users, you can also add the cache in your home directory:
trusted-public-keys = cache.nixos-cuda.org:74DUi4Ye579gUqzH4ziL9IyiJBlDpMRn9MBN8oNan9M=
trusted-substituters = https://cache.nixos-cuda.org
Flox binary cache
NVIDIA CUDA packages are generally unfree, so cache.nixos.org does not normally provide pre-built store paths for them. The same constraint applies to CUDA-enabled packages whose closures depend on proprietary NVIDIA software. This includes packages such as PyTorch and MAGMA, which, because of their large transitive closures, typically require considerable time and resources to build.
Flox is one of the providers that NVIDIA works with to redistribute CUDA libraries and runtime packages. Flox also distributes PyTorch, MAGMA, and other CUDA-enabled packages whose Nix store closures include proprietary NVIDIA software. This agreement allows Flox to publish pre-built store paths that are not normally available from cache.nixos.org.
Flox builds the release-cuda jobset and pushes the store paths to its public binary cache.
| Cache URL | https://cache.flox.dev
|
|---|---|
| Public key | flox-cache-public-1:7F4OyH7ZCnFhcze3fJdfyXYLQw/aV7GEed86nQ7IsOs=
|
Which store paths are available in the cache depends on which Nixpkgs revisions Flox builds against. Flox currently evaluates and builds CUDA packages against revisions from its own Nixpkgs fork at github.com/flox/nixpkgs. This repo tracks the upstream nixos-unstable branch across four Flox-maintained branches (unstable, staging, stable, and lts). These branch names describe Flox's Nixpkgs update schedules, not the stability of the software they reference. In general, Flox's unstable branch tracks upstream nixos-unstable most closely, sampling and updating on a daily schedule.
Notably, Flox does not currently build against Nixpkgs release channels, so users who pin to revisions from these channels are unlikely to get a cache hit from the Flox binary cache. Because different Nixpkgs revisions can produce different store paths for the same package, using a Nixpkgs revision that Flox builds against significantly increases the likelihood of a cache hit.
NixOS
nix.settings = {
substituters = [
"https://cache.flox.dev"
];
trusted-public-keys = [
"flox-cache-public-1:7F4OyH7ZCnFhcze3fJdfyXYLQw/aV7GEed86nQ7IsOs="
];
};
Non-NixOS
extra-substituters = https://cache.flox.dev
extra-trusted-public-keys = flox-cache-public-1:7F4OyH7ZCnFhcze3fJdfyXYLQw/aV7GEed86nQ7IsOs=
For more on configuring Nix to use the Flox binary cache, maximizing the likelihood of CUDA cache hits, and controlling which builds query the cache, see CUDA packages in the Flox Catalog and binary cache in Flox's documentation.
Enabling CUDA In Packages
By default, software packaged in source code form has CUDA support disabled, because of the unfree license. There are multiple options to solve this.
You can enable builds with CUDA support with a nixpkgs wide configuration.
nixpkgs.config.cudaSupport = true;
Or you can override individual packages.
environment.systemPackages = with pkgs; [
(mlt.override {config.cudaSupport=true;})
];
Or you can use binary-packaged versions of CUDA compatible software, such as blender-bin for Blender.
cudaSupport in packages, it is recommended you utilize a CUDA binary cache.
Without a CUDA cache, any CUDA compatible package installed with cudaSupport will be compiled from source. This is because NixOS Foundation does not build (and therefore cache.nixos.org does not cache) CUDA packages.
For larger programs like Blender, that process can be very resource-intensive. If you are installing large CUDA-enabled package(s) that either are not cached or you are not using a cache, then (especially on older or weaker hardware) it is recommended to reduce the number of cores and/or jobs that the process will take, to prevent a system freeze from resource limits. This can be done with the --max-jobs / -j and --cores flags, for more details see the Tuning Cores & Jobs manual page.
If you don't want to deal with the increased time that compilation will take when --max-jobs / -j and --cores are set below maximum, you can also try simply closing other running processes to see if that frees up enough resources for compilation to be successful.
→ For specifics on setting up Blender with CUDA (and OptiX) see: Blender#CUDA & OptiX.
Some things to keep in mind when setting up CUDA in NixOS
- Some GPUs, like Tesla K80, don't work with the latest drivers, so you must specify them in the option
hardware.nvidia.packagegetting the value from your selected kernel, for example,config.boot.kernelPackages.nvidia_x11_legacy470. You can check which driver version your GPU supports by visiting the nvidia site and checking the driver version. - Even with the drivers correctly installed, some software, like Blender, may not see the CUDA GPU. Make sure your system configuration has the option
hardware.graphics.enableenabled.
CUDA under WSL
This (surprisingly) works just fine using nixpkgs 23.05 provided that you prefix the LD_LIBRARY_PATH in your interactive environment with the WSL library directory. For nix shell this looks like:
shellHook = ''
export CUDA_PATH=${pkgs.cudatoolkit}
export LD_LIBRARY_PATH=/usr/lib/wsl/lib:${pkgs.linuxPackages.nvidia_x11}/lib:${pkgs.ncurses5}/lib
export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
export EXTRA_CCFLAGS="-I/usr/include"
'';