NVIDIA: Difference between revisions

From NixOS Wiki
imported>NewAM
m Minor typo
Pluieuwu (talk | contribs)
m →‎Offload mode: Make the different modes stylistically more similar
 
(75 intermediate revisions by 33 users not shown)
Line 1: Line 1:
= Installing NVIDIA official drivers on NixOS =
This page attempts to cover everything related to the use of NVIDIA GPUs on NixOS.
If you're using NixOS, installing and using the official NVIDIA drivers is as simple as,
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{
  # NVIDIA drivers are unfree.
  nixpkgs.config.allowUnfree = true;


  services.xserver.videoDrivers = [ "nvidia" ];
== Enabling ==
  hardware.opengl.enable = true;


  # Optionally, you may need to select the appropriate driver version for your specific GPU.
=== Kernel modules from NVIDIA ===
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.stable;
Kernel modules from NVIDIA offer better performance than other alternatives, but make the system unfree by requiring proprietary userspace libraries that can interface with the kernel modules. Users that want to have a fully free and open-source system should use [[NVIDIA#Nouveau|Nouveau]] instead.


  # nvidia-drm.modeset=1 is required for some wayland compositors, e.g. sway
To enable them, add <code>"nvidia"</code> to the list of enabled video drivers defined by the <code>services.xserver.videoDrivers</code> option.
  hardware.nvidia.modesetting.enable = true;
  ...
}
</nowiki>}}


As noted in the final comment, you'll need to determine the appropriate driver version for your card. For "legacy" cards, you can consult [https://www.nvidia.com/en-us/drivers/unix/legacy-gpu/ nvidia official legacy driver list]. You can consult the set of possible options in the source [https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/os-specific/linux/nvidia-x11/default.nix here].
{{Note|<code>hardware.graphics.enable</code> was named <code>hardware.opengl.enable</code> '''until NixOS 24.11'''.}}


(The <code>services.xserver.videoDrivers</code> setting is also valid for wayland installations despite it's name.)
{{Note|Since driver version 560, you also will need to decide whether to use the open-source or proprietary modules by setting the <code>hardware.nvidia.open</code> option to either <code>true</code> or <code>false</code> respectively.<br><br>Open-source kernel modules are preferred over and planned to steadily replace proprietary modules<ref>https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/</ref>, although they only support GPUs of the Turing architecture or newer (from GeForce RTX 20 series and GeForce GTX 16 series onwards). Data center GPUs starting from Grace Hopper or Blackwall must use open-source modules — proprietary modules are no longer supported.<br><br>''If you're using proprietary modules'', make sure to allow [[Unfree software|unfree software]]. The unfree NVIDIA packages include <code>nvidia-x11</code>, <code>nvidia-settings</code>, and <code>nvidia-persistenced</code>.
}}{{Warning|If you use a laptop with both dedicated and integrated GPUs, remember to [[#Hybrid_graphics_with_PRIME|configure PRIME]] in order to make your dedicated NVIDIA GPU work properly with your integrated GPU. Your configuration '''might not work''' if you skip this step.}}{{file|configuration.nix|nix|<nowiki>
{
  hardware.graphics.enable = true;


= Using GPUs on non-NixOS =
  services.xserver.videoDrivers = [ "nvidia" ];
If you're using Nix-packaged software on a non-NixOS system, you'll need a workaround to get everything up-and-running. The [https://github.com/guibou/nixGL nixGL project] provides wrapper to use GL drivers on non-NixOS systems. You need to have GPU drivers installed on your distro (for kernel modules). With nixGL installed, you'll run <code>nixGL foobar</code> instead of  <code>foobar</code>.
  hardware.nvidia.open = true; # Set to false for proprietary drivers
}
</nowiki>}}
==== Legacy branches ====
GPUs of the Kepler architecture or older (most GeForce 600/700/800M cards and older) are no longer supported by latest proprietary modules. Instead, users of these GPUs must use legacy branches that may still receive updates, as long as the GPUs themselves remain supported by NVIDIA. You can find which legacy branch you need to use by searching for your GPU model on [https://www.nvidia.com/en-us/drivers/unix/legacy-gpu/ NVIDIA's official legacy driver support list].


Note that nixGL is not specific to NVIDIA GPUs, and should work with just about any GPU.
To use legacy branches, you need to set the <code>hardware.nvidia.package</code> option to a package set named <code>config.boot.kernelPackages.nvidiaPackages.legacy_<branch></code>.{{file|configuration.nix|nix|<nowiki>
{ config, ... }: # ← Required to get the packages used by the currently configured kernel, including drivers
{
  # Last version that supports Kepler GPUs
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.legacy_470;
}
</nowiki>}}Nixpkgs does not endeavor to support all legacy branches since older, unmaintained legacy branches can become incompatible with newer kernel and X server versions, and at some point it becomes infeasible to patch them to cooperate with modern software. You can find the list of supported legacy branches under [https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/nvidia-x11/default.nix in the Nixpkgs repository].


= CUDA and using your GPU for compute =
==== Beta/production branches ====
See the [[CUDA]] wiki page!
By default, modules from the stable branch are used, which come from latest release available for the current architecture — while x86-64 and aarch64 systems follow the "New Feature Branch" releases as they remain actively developed, 32-bit x86 and ARM systems remain on the 390.xx branch as that is the last branch with 32-bit support.


= Using your GPU for graphics =
Instead of the stable branch, users can also switch to the beta branch, which has more new features and experimental changes at the risk of containing more bugs, or the production branch, a more conservative, well-tested stable version that is suitable for production use with minimum breakage, at the cost of being behind in terms of new features.  
Please note that, if you are setting up PRIME offloading, you '''must''' set the single value of "nvidia" even though it would be more conceptually correct to also include the driver for your other GPU. Doing otherwise will cause a broken xorg.conf to be generated. This is because NixOS doesn't ''actually'' handle multiple GPUs / GPU drivers properly, as per https://github.com/NixOS/nixpkgs/issues/108018.


== Determining the type of your GPU ==
Using the beta and production branches are similar to how one would use legacy branches:{{file|configuration.nix|nix|<nowiki>
{ config, ... }:
{
  #hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.stable; # Default
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.beta;
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.production;
}
</nowiki>}}


* MXM / output-providing card (shows as <code>VGA Controller</code> in <code>lspci</code>), i.e. graphics card in desktop computer or in some laptops
=== Nouveau ===
* muxless/non-MXM Optimus cards have no display outputs and show as <code>3D Controller</code> in <code>lspci</code> output, seen in most modern consumer laptops
Nouveau is a set of free and open-source drivers for NVIDIA GPUs that provide 2D/3D acceleration for all NVIDIA GPUs. Its use is in general not recommended due to its considerably worse performance compared to NVIDIA's kernel modules, as it does not support reclocking (changing the GPU clock frequency on-demand) for many NVIDIA GPUs<ref>https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-nvidia-linux-nouveau/998310-nouveau-persevered-in-2017-for-open-source-nvidia-but-2018-could-be-much-better#post998316</ref>. Nevertheless it remains a viable option for users who want a fully free and open-source operating system as it does not contain any proprietary components, unlike NVIDIA's kernel modules and userspace libraries.


MXM cards allow you to use the Nvidia card standalone, in Non-Optimus mode. Non-MXM cards ''require'' [https://en.wikipedia.org/wiki/Nvidia_Optimus Optimus], Nvidia's integrated-vs-discrete GPU switching technology.
Nouveau is enabled by default whenever graphics are enabled, and does not need any extra configuration.{{file|configuration.nix|nix|<nowiki>
{
  hardware.graphics.enable = true;
}
</nowiki>}}


== Non-Optimus mode ==
== Configuring ==


'''You need an MXM card''' (see above) for Non-Optimus mode. Follow [https://nixos.org/nixos/manual/#sec-x11-graphics-cards-nvidia NVIDIA Graphics Cards] section in official manual.
=== Power management ===


In case of laptop you may also need to use a BIOS option to select which card to use for the internal display.
=== Hybrid graphics with PRIME ===
Laptops often feature both an integrated GPU (iGPU) and a dedicated GPU (dGPU) in order to strive a balance between performance and power consumption ­— while the dGPU is used for performance-intensive tasks such as gaming, video editing, 3D rendering, compute jobs, etc., the iGPU can be used to render common 2D elements like application windows and the desktop environment.  


== Optimus ==
PRIME, therefore, is a technology developed to facilitate the cooperation between the two GPUs and is critical for the laptop's graphical performance. Depending on your needs, you can configure PRIME in one of three modes, which have different tradeoffs in terms of performance and battery life.


'''Mostly useful for laptops'''. There are currently two solutions available under NixOS, described in detail below:
==== Common setup ====
All PRIME configurations require setting the PCI bus IDs of the two GPUs. One easy way to do find their IDs is by running <code>lspci</code> from the <code>pciutils</code> package, and then finding devices that are classified as VGA controllers. After double checking that the listed devices are indeed your integrated and dedicated GPUs, you can then find the PCI IDs at the beginning of each line. Exact results may vary, but an example output might look like:<syntaxhighlight lang="console">
$ nix shell nixpkgs#pciutils -c "lspci" | grep "VGA"
0000:00:02.0 VGA compatible controller: Intel Corporation TigerLake-H GT1 [UHD Graphics] (rev 01)
0000:01:00.0 VGA compatible controller: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] (rev a1)
</syntaxhighlight>Before putting them into your configuration, however, '''they must first be reformatted''' — take the last three numbers, convert them from hexadecimal to decimal, remove the leading zeroes, concatenate them with colons, and then add a <code>PCI:</code> prefix. Then, they can be set under <code>intelBusId</code>, <code>nvidiaBusId</code>, or <code>amdgpuBusId</code> in <code>hardware.nvidia.prime</code>, depending on the manufacturer of the GPU:{{file|configuration.nix|nix|<nowiki>
{
  hardware.nvidia.prime = {
    intelBusId = "PCI:0:2:0";
    nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
  };
}
</nowiki>}}


* Official solution: Nvidia PRIME (in on-demand "offload" mode, and always-on "sync" mode)
==== Offload mode ====
* Previous open-source solution: Bumblebee (now deprecated)
{{Note|Offload mode is available '''since NixOS 20.09 and NVIDIA driver version 435.21''', and requires an NVIDIA GPU of the Turing generation, or newer and a compatible CPU — either an Intel CPU from the Coffee Lake generation or newer, or an AMD Ryzen. Offload mode is '''incompatible''' with sync mode.}}
Offload mode puts your dGPU to sleep and lets the iGPU handle all tasks, except if you call the dGPU specifically by "offloading" an application to it. For example, you can run your laptop normally and it will use the energy-efficient iGPU all day, and then you can offload a game from Steam onto the dGPU to make the dGPU run that game only. For many, this is the most desirable option.


=== Nvidia PRIME ===
To enable offload mode, set the <code>hardware.nvidia.prime.offload.enable</code> option to <code>true</code>:


Official solution by nvidia.  Currently, reverse PRIME does not work. The consequence of this is that if you have a special laptop configuration where external display ports are only exposed to the dedicated GPU, then running in offload mode  [https://archived.forum.manjaro.org/t/nvidia-render-offloading-help-getting-external-monitor-working/99430/23 will not allow you to use those display ports for external monitors]. If you wish to use the external monitors in that particular case, you have to use sync mode.
{{file|configuration.nix|nix|<nowiki>
 
{
==== offload mode ====
  hardware.nvidia.prime = {
 
    offload.enable = true;
'''Available since 20.09''' (see [https://github.com/NixOS/nixpkgs/pull/66601 #66601]).
   
 
    intelBusId = "PCI:0:2:0";
In this mode the Nvidia card is only activated on demand, however a Nvidia card of the Turing generation or newer and an Intel Coffee Lake chipset is required for a complete poweroff of the Nvidia card (see [https://discourse.nixos.org/t/how-to-use-nvidia-prime-offload-to-run-the-x-server-on-the-integrated-board/9091/19?u=moritzschaefer discussion]).
    nvidiaBusId = "PCI:1:0:0";
 
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
Offload mode is enabled by running your program(s) with specific environment variables, i.e., here's a sample script called <code>nvidia-offload</code> that you can run wrapped around your exacutable, for example <code>nvidia-offload glxgears</code>:
  };
 
}
{{file|nvidia-offload|bash|<nowiki>
</nowiki>}}When you want to run a program on the dGPU, you only need to set a few environment variables for the driver to recognize that offload mode should be used. If <code>hardware.nvidia.prime.offload.enableOffloadCmd</code> is set to true, NixOS will generate a wrapper script named <code>nvidia-offload</code> that sets the right variables for you:{{file|nvidia-offload|bash|<nowiki>
export __NV_PRIME_RENDER_OFFLOAD=1
export __NV_PRIME_RENDER_OFFLOAD=1
export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
Line 71: Line 99:
export __VK_LAYER_NV_optimus=NVIDIA_only
export __VK_LAYER_NV_optimus=NVIDIA_only
exec "$@"
exec "$@"
</nowiki>}}
</nowiki>}}If everything is configured correctly, then running a program like <code>glxgears</code> should use the iGPU, while running <code>nvidia-offload glxgears</code> should only use the dGPU.
 
==== Sync mode ====
{{Note|Sync mode is available '''since NixOS 19.03 and NVIDIA driver version 390.67''', and is '''incompatible''' with both offload and reverse sync modes. Sync mode also requires using a desktop manager that respects the <code>services.xserver.displayManager.setupCommands</code> option, including LightDM, GDM and SDDM.}}In sync mode, rendering is completely delegated to the dGPU, while the iGPU only displays the rendered framebuffers copied from the dGPU. Sync mode offers better performance and greatly reduces screen tearing, at the expense of higher power consumption since the dGPU will not go to sleep when not needed, as is the case in offload mode. Sync mode may also solve some issues with connecting a display in clamshell mode directly to the GPU.
 
To enable sync mode, set the <code>hardware.nvidia.prime.sync.enable</code> option to <code>true</code>:


To configure Offload mode, you firstly you need to enable the proprietary Nvidia driver:
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{
{
   services.xserver.videoDrivers = [ "nvidia" ];
   hardware.nvidia.prime = {
  ...
    sync.enable = true;
</nowiki>}}
   
    intelBusId = "PCI:0:2:0";
Note that on '''certain laptops''' and/or if you are using a custom kernel version, you may have issues with your NixOS system finding the primary display. In this case you should use <code>hardware.nvidia.modesetting.enable</code>, i.e.:
    nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
   };
}
{
  hardware.nvidia.modesetting.enable = true;
  services.xserver.videoDrivers = [ "nvidia" ];
   // ...
</nowiki>}}
</nowiki>}}
Then you need to setup the Bus ID's of the cards as seen below.
''Note: Bus ID is important and needs to be formatted properly''
The Nvidia driver expects the bus ID to be in decimal format;
There are two ways you can get the bus IDs, one is with <nowiki>lspci</nowiki>, which shows the bus IDs in hexadecimal format and the other with <nowiki>lshw</nowiki>, which shows it in decimal format, as wanted by nixos.
===== lspci =====
You can convert the value by:
* Stripping any leading zeros from the bus numbers or if the number is above 09, convert it to decimal and use that value.
* Replacing any full stops with colons.
* Prefix the final value with "PCI".
For example:
'''Output from lspci'''
<code>09:1f.0 VGA compatible controller: NVIDIA Corporation Device 1f91 (rev a1)</code>
'''Converted and correct format'''
<code>PCI:9:31:0</code>
===== lshw =====
If you don't have <nowiki>lshw</nowiki> installed, you can get it temporarily in a ephemeral shell by running:
<code>nix-shell -p lshw --run "lshw -c display"</code>
The two bus ID's will be in the corresponding "bus info" field.
For example:
<code>
bus info: pci@0000:01:00.0
</code>
and
<code>
bus info: pci@0000:00:02.0
</code>
Now you have to take everything after the first colon, and replace the <code>.</code> with another colon.
<code>
01:00:0
</code>
and


<code>
==== Reverse sync mode ====
00:02:0
{{Note|Reverse sync mode is available '''since NixOS 23.05 and NVIDIA driver version 460.39''' and is still an experimental, buggy feature<ref>https://forums.developer.nvidia.com/t/the-all-new-outputsink-feature-aka-reverse-prime/129828/67</ref>. '''Your mileage may vary.''' Reverse sync mode is '''incompatible''' with sync mode and requires using a desktop manager that respects the <code>services.xserver.displayManager.setupCommands</code> option, including LightDM, GDM and SDDM.}}The difference between regular sync mode and reverse sync mode is that the '''dGPU''' is configured as the primary output device, allowing displaying to external displays wired to it and not the iGPU (more common). 
</code>


A possible configuration:
To enable sync mode, set the <code>hardware.nvidia.prime.reverseSync.enable</code> option to <code>true</code>:


{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{ pkgs, ... }:
let
  nvidia-offload = pkgs.writeShellScriptBin "nvidia-offload" ''
    export __NV_PRIME_RENDER_OFFLOAD=1
    export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
    export __GLX_VENDOR_LIBRARY_NAME=nvidia
    export __VK_LAYER_NV_optimus=NVIDIA_only
    exec "$@"
  '';
in
{
{
  environment.systemPackages = [ nvidia-offload ];
  services.xserver.videoDrivers = [ "nvidia" ];
   hardware.nvidia.prime = {
   hardware.nvidia.prime = {
     offload.enable = true;
     reverseSync.enable = true;


    # Bus ID of the Intel GPU. You can find it using lspci, either under 3D or VGA
     intelBusId = "PCI:0:2:0";
     intelBusId = "PCI:0:2:0";
    # Bus ID of the NVIDIA GPU. You can find it using lspci, either under 3D or VGA
     nvidiaBusId = "PCI:1:0:0";
     nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
   };
   };
}
}
</nowiki>}}
</nowiki>}}


==== booting with an external display ====
== Tips and tricks ==


Most Optimus laptops have the HDMI port for an external display wired directly to the Nvidia chip, in which case you need a configuration to use the Nvidia driver directly without offload mode. Fortunately, NixOS has an amazing feature called specialisations which allows you to do this easily. Here is an example configuration:
=== Check nixos-hardware ===
The [https://github.com/NixOS/nixos-hardware nixos-hardware] project attempts to provide configurations that address specific hardware quirks for different devices. It is possible that someone already wrote a hardware configuration for your device and that usually takes care of drivers. If so, follow the upstream documentation to enable the required modules. 
 
=== Multiple boot configurations ===
Imagine you have a laptop that you mostly use in clamshell mode (docked, connected to an external display and plugged into a charger) but that you sometimes use on the go.
 
In clamshell mode, using PRIME Sync is likely to lead to better performance, external display support, etc., at the cost of potentially (but not always) lower battery life. However, when using the laptop on the go, you may prefer to use offload mode.
 
NixOS supports "specialisations", which allow you to automatically generate different boot profiles when rebuilding your system. We can, for example, enable PRIME sync by default, but also create a "on-the-go" specialization that disables PRIME sync and instead enables offload mode:


{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
   specialisation = {
{
    external-display.configuration = {
   specialisation.on-the-go.configuration = {
      system.nixos.tags = [ "external-display" ];
    system.nixos.tags = [ "on-the-go" ];
      hardware.nvidia.prime.offload.enable = lib.mkForce false;
    hardware.nvidia.prime = {
       hardware.nvidia.powerManagement.enable = lib.mkForce false;
      offload = {
        enable = lib.mkForce true;
        enableOffloadCmd = lib.mkForce true;
      };
       sync.enable = lib.mkForce false;  
     };
     };
   };
   };
}
</nowiki>}}
</nowiki>}}


Once you rebuild your configuration, an extra <code>external-display</code> configuration will be built and placed in your boot menu.
(You can also add other settings here totally unrelated to NVIDIA, such as power profiles, etc.)


To use this, boot your laptop with the lid open, choose the <code>external-display</code> configuration in the boot menu, and continue to keep the lid open until your desktop appears on the external display. At this point you can close the lid.
After rebuilding and rebooting, you'll see in your boot menu under each Generation an "on-the-go" option, which will let you boot into the on-to-go specialisation for that generation.


==== offloading steam ====
=== Using GPUs on non-NixOS ===
If you're using Nix-packaged software on a non-NixOS system, you'll need a workaround to get everything up-and-running. The [https://github.com/guibou/nixGL nixGL project] provides wrapper to use GL drivers on non-NixOS systems. You need to have GPU drivers installed on your distro (for kernel modules). With nixGL installed, you'll run <code>nixGL foobar</code> instead of  <code>foobar</code>.


First, add this to your <code>~/.bashrc</code> :
Note that nixGL is not specific to NVIDIA GPUs, and should work with just about any GPU.
 
<syntaxhighlight lang="bash">
export XDG_DATA_HOME="$HOME/.local/share"
</syntaxhighlight>   
.


For NixOS Steam run:
=== CUDA and using your GPU for compute ===
See the [[CUDA]] wiki page.


<syntaxhighlight lang="bash">
=== Running Specific NVIDIA Driver Versions ===
mkdir -p ~/.local/share/applications
sed 's/^Exec=/&nvidia-offload /' /run/current-system/sw/share/applications/steam.desktop > ~/.local/share/applications/steam.desktop
</syntaxhighlight>
.


For Flatpak Steam run:
To run a specific version of the NVIDIA driver in NixOS, you can customize your configuration by specifying the desired version along with the corresponding SHA256 hashes. Below is an example configuration for using NVIDIA driver version <code>555.58.02</code>:


<syntaxhighlight lang="bash">
{{file|/etc/nixos/nvidia.nix|nix|<nowiki>
mkdir -p ~/.local/share/applications
{ config, ... }:
sed 's/^Exec=/&nvidia-offload /' /var/lib/flatpak/exports/share/applications/com.valvesoftware.steam.desktop > ~/.local/share/applications/com.valvesoftware.steam.desktop
</syntaxhighlight>
.
 
Then restart your graphical environment session.
 
==== sync mode ====
 
In this mode the Nvidia card is turned on constantly, having impact on laptop battery and health.
 
Possible issues:
* Hangs of applications after resume from suspend
* Wrong DPI calculation (in this case provide dpi manually <code>services.xserver.dpi = 96;</code>)
* Black screen after system upgrade (e.g. <code>nixos-rebuild switch</code>; use <code>nixos-rebuild boot</code> instead and reboot)
* No video playback acceleration available (vaapi)
 
===== Example for NixOS 20.03 =====
 
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{
{
   hardware.nvidia.modesetting.enable = true;
   hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.mkDriver {
  services.xserver.videoDrivers = [ "nvidia" ];
     version = "555.58.02";
  hardware.nvidia.optimus_prime = {
     sha256_64bit = "sha256-xctt4TPRlOJ6r5S54h5W6PT6/3Zy2R4ASNFPu8TSHKM=";
     enable = true;
     sha256_aarch64 = "sha256-xctt4TPRlOJ6r5S54h5W6PT6/3Zy2R4ASNFPu8TSHKM=";
 
     openSha256 = "sha256-ZpuVZybW6CFN/gz9rx+UJvQ715FZnAOYfHn5jt5Z2C8=";
     # Bus ID of the NVIDIA GPU. You can find it using lspci, either under 3D or VGA
     settingsSha256 = "sha256-ZpuVZybW6CFN/gz9rx+UJvQ715FZnAOYfHn5jt5Z2C8=";
     nvidiaBusId = "PCI:1:0:0";
    persistencedSha256 = lib.fakeSha256;
 
     # Bus ID of the Intel GPU. You can find it using lspci, either under 3D or VGA
     intelBusId = "PCI:0:2:0";
   };
   };
}
};
</nowiki>}}
</nowiki>}}


===== Example for NixOS 20.09/unstable =====
In this configuration:


{{file|/etc/nixos/configuration.nix|nix|<nowiki>
* Replace <code>version</code> with the desired driver version.
{
* Update the SHA256 hashes to match the new version you want to use.
  services.xserver.videoDrivers = [ "nvidia" ];
* After updating the configuration, run <code>sudo nixos-rebuild switch</code> to apply the changes and load the specified NVIDIA driver version.


  hardware.nvidia.prime = {
This allows you to pin the specific driver version being used in your NixOS installation.
    sync.enable = true;


    # Bus ID of the NVIDIA GPU. You can find it using lspci, either under 3D or VGA
== Troubleshooting ==
    nvidiaBusId = "PCI:1:0:0";


    # Bus ID of the Intel GPU. You can find it using lspci, either under 3D or VGA
=== Booting to text mode ===
    intelBusId = "PCI:0:2:0";
If you encounter the problem of booting to text mode you might try adding the NVIDIA kernel module manually with:
  };
}
</nowiki>}}


=== Bumblebee ===
<syntaxHighlight lang="nix">
boot.initrd.kernelModules = [ "nvidia" ];
boot.extraModulePackages = [ config.boot.kernelPackages.nvidia_x11 ];
</syntaxHighlight>


Deprecated solution. You should use [[Nvidia#offload_mode|offload mode]] instead.
=== Screen tearing issues ===


Use option <syntaxHighlight lang="nix">hardware.bumblebee.enable = true;</syntaxHighlight>
First, try to switch to PRIME sync mode, as described above. If that doesn't work, try forcing a composition pipeline.


{{note|Forcing a full composition pipeline has been reported to reduce the performance of some OpenGL applications and may produce issues in WebGL. It also drastically increases the time the driver needs to clock down after load.}}


== Troubleshooting ==
=== Fix screen tearing ===
You may often incounter screen tearing or artifacts when using proprietary Nvidia drivers. You can fix that by forcing full composition pipeline.
{{note|This has been reported to reduce the performance of some OpenGL applications and may produce issues in WebGL. It also drastically increases the time the driver needs to clock down after load.}}
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
hardware.nvidia.forceFullCompositionPipeline = true;
hardware.nvidia.forceFullCompositionPipeline = true;
</nowiki>}}
</nowiki>}}


=== Fix app flickering with Picom ===
=== Flickering with Picom ===
{{file|~/.config/picom/picom.conf|conf|<nowiki>
{{file|~/.config/picom/picom.conf|conf|<nowiki>
unredir-if-possible = false;
unredir-if-possible = false;
Line 290: Line 227:
</nowiki>}}
</nowiki>}}


=== Fix graphical corruption on suspend/resume ===
=== Graphical corruption and system crashes on suspend/resume ===
By default only a small portion of VRAM is saved when suspending the system [https://download.nvidia.com/XFree86/Linux-x86_64/460.73.01/README/powermanagement.html]. This can cause graphical issues in some applications when resuming from suspend. To fix it, enable systemd-based suspend, which will save and restore all of VRAM:
<code>powerManagement.enable = true</code> can sometimes fix this, but is itself unstable and is known to cause suspend issues.
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
 
hardware.nvidia.powerManagement.enable = true;
If you have a modern NVIDIA GPU (Turing [https://en.wikipedia.org/wiki/Turing_(microarchitecture)#Products_using_Turing] or later), you may also want to investigate the <code>hardware.nvidia.powerManagement.finegrained</code> option: [https://download.nvidia.com/XFree86/Linux-x86_64/460.73.01/README/dynamicpowermanagement.html]
</nowiki>}}
If you have a modern Nvidia GPU (Turing [https://en.wikipedia.org/wiki/Turing_(microarchitecture)#Products_using_Turing] or later), you may also want to investigate the <code>hardware.nvidia.powerManagement.finegrained</code> option: [https://download.nvidia.com/XFree86/Linux-x86_64/460.73.01/README/dynamicpowermanagement.html]


=== Fix black screen on a system with an integrated GPU ===
=== Black screen or 'nothing works' on laptops ===
If you are using non-Optimus mode on a system with an integrated GPU, the kernel module <code>i915</code> for intel or <code>amdgpu</code> for AMD may interfere with the NVIDIA driver. This may result in a black screen when switching to the virtual terminal, or when exiting the X session. A possible workaround is to disable the integrated GPU by blacklisting the module, using the following configuration option (see also [https://discourse.nixos.org/t/nvidia-gpu-and-i915-kernel-module/21307/3]):
The kernel module<code>i915</code>for Intel or<code>amdgpu</code>for AMD may interfere with the NVIDIA driver. This may result in a black screen when switching to the virtual terminal, or when exiting the X session. A possible workaround is to disable the integrated GPU by blacklisting the module, using the following configuration option (see also [https://discourse.nixos.org/t/nvidia-gpu-and-i915-kernel-module/21307/3]):


<syntaxHighlight lang="nix">
<syntaxHighlight lang="nix">
Line 307: Line 242:
</syntaxHighlight>
</syntaxHighlight>


== Disabling ==
=== Kernel modules from NVIDIA ===
Normally, NVIDIA's kernel modules should be completely disabled by removing <code>"nvidia"</code> from <code>services.xserver.videoDrivers</code>. If that fails to work, you can also manually blacklist the corresponding kernel modules:<syntaxhighlight lang="nix">
{
  boot.blacklistedKernelModules = [
    "nvidia"
    "nvidiafb"
    "nvidia-drm"
    "nvidia-uvm"
    "nvidia-modeset"
  ];
}
</syntaxhighlight>
=== Nouveau ===
Nouveau can be disabled by blacklisting the <code>nouveau</code> kernel module:<syntaxhighlight lang="nix">
{
  boot.blacklistedKernelModules = [ "nouveau" ];
}
</syntaxhighlight>Note that disabling both NVIDIA kernel modules and Nouveau effectively disables the GPU entirely.
== Footnotes ==
[[Category:Video]]
[[Category:Video]]

Latest revision as of 16:12, 26 October 2024

This page attempts to cover everything related to the use of NVIDIA GPUs on NixOS.

Enabling

Kernel modules from NVIDIA

Kernel modules from NVIDIA offer better performance than other alternatives, but make the system unfree by requiring proprietary userspace libraries that can interface with the kernel modules. Users that want to have a fully free and open-source system should use Nouveau instead.

To enable them, add "nvidia" to the list of enabled video drivers defined by the services.xserver.videoDrivers option.

Note: hardware.graphics.enable was named hardware.opengl.enable until NixOS 24.11.
Note: Since driver version 560, you also will need to decide whether to use the open-source or proprietary modules by setting the hardware.nvidia.open option to either true or false respectively.

Open-source kernel modules are preferred over and planned to steadily replace proprietary modules[1], although they only support GPUs of the Turing architecture or newer (from GeForce RTX 20 series and GeForce GTX 16 series onwards). Data center GPUs starting from Grace Hopper or Blackwall must use open-source modules — proprietary modules are no longer supported.

If you're using proprietary modules, make sure to allow unfree software. The unfree NVIDIA packages include nvidia-x11, nvidia-settings, and nvidia-persistenced.
Warning: If you use a laptop with both dedicated and integrated GPUs, remember to configure PRIME in order to make your dedicated NVIDIA GPU work properly with your integrated GPU. Your configuration might not work if you skip this step.
configuration.nix
{
  hardware.graphics.enable = true;

  services.xserver.videoDrivers = [ "nvidia" ];
  hardware.nvidia.open = true; # Set to false for proprietary drivers
}

Legacy branches

GPUs of the Kepler architecture or older (most GeForce 600/700/800M cards and older) are no longer supported by latest proprietary modules. Instead, users of these GPUs must use legacy branches that may still receive updates, as long as the GPUs themselves remain supported by NVIDIA. You can find which legacy branch you need to use by searching for your GPU model on NVIDIA's official legacy driver support list.

To use legacy branches, you need to set the hardware.nvidia.package option to a package set named config.boot.kernelPackages.nvidiaPackages.legacy_<branch>.

configuration.nix
{ config, ... }: # ← Required to get the packages used by the currently configured kernel, including drivers
{ 
  # Last version that supports Kepler GPUs
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.legacy_470;
}

Nixpkgs does not endeavor to support all legacy branches since older, unmaintained legacy branches can become incompatible with newer kernel and X server versions, and at some point it becomes infeasible to patch them to cooperate with modern software. You can find the list of supported legacy branches under in the Nixpkgs repository.

Beta/production branches

By default, modules from the stable branch are used, which come from latest release available for the current architecture — while x86-64 and aarch64 systems follow the "New Feature Branch" releases as they remain actively developed, 32-bit x86 and ARM systems remain on the 390.xx branch as that is the last branch with 32-bit support.

Instead of the stable branch, users can also switch to the beta branch, which has more new features and experimental changes at the risk of containing more bugs, or the production branch, a more conservative, well-tested stable version that is suitable for production use with minimum breakage, at the cost of being behind in terms of new features.

Using the beta and production branches are similar to how one would use legacy branches:

configuration.nix
{ config, ... }:
{ 
  #hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.stable; # Default
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.beta;
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.production;
}

Nouveau

Nouveau is a set of free and open-source drivers for NVIDIA GPUs that provide 2D/3D acceleration for all NVIDIA GPUs. Its use is in general not recommended due to its considerably worse performance compared to NVIDIA's kernel modules, as it does not support reclocking (changing the GPU clock frequency on-demand) for many NVIDIA GPUs[2]. Nevertheless it remains a viable option for users who want a fully free and open-source operating system as it does not contain any proprietary components, unlike NVIDIA's kernel modules and userspace libraries.

Nouveau is enabled by default whenever graphics are enabled, and does not need any extra configuration.

configuration.nix
{
  hardware.graphics.enable = true;
}

Configuring

Power management

Hybrid graphics with PRIME

Laptops often feature both an integrated GPU (iGPU) and a dedicated GPU (dGPU) in order to strive a balance between performance and power consumption ­— while the dGPU is used for performance-intensive tasks such as gaming, video editing, 3D rendering, compute jobs, etc., the iGPU can be used to render common 2D elements like application windows and the desktop environment.

PRIME, therefore, is a technology developed to facilitate the cooperation between the two GPUs and is critical for the laptop's graphical performance. Depending on your needs, you can configure PRIME in one of three modes, which have different tradeoffs in terms of performance and battery life.

Common setup

All PRIME configurations require setting the PCI bus IDs of the two GPUs. One easy way to do find their IDs is by running lspci from the pciutils package, and then finding devices that are classified as VGA controllers. After double checking that the listed devices are indeed your integrated and dedicated GPUs, you can then find the PCI IDs at the beginning of each line. Exact results may vary, but an example output might look like:

$ nix shell nixpkgs#pciutils -c "lspci" | grep "VGA"
0000:00:02.0 VGA compatible controller: Intel Corporation TigerLake-H GT1 [UHD Graphics] (rev 01)
0000:01:00.0 VGA compatible controller: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] (rev a1)

Before putting them into your configuration, however, they must first be reformatted — take the last three numbers, convert them from hexadecimal to decimal, remove the leading zeroes, concatenate them with colons, and then add a PCI: prefix. Then, they can be set under intelBusId, nvidiaBusId, or amdgpuBusId in hardware.nvidia.prime, depending on the manufacturer of the GPU:

configuration.nix
{
  hardware.nvidia.prime = {
    intelBusId = "PCI:0:2:0";
    nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
  };
}

Offload mode

Note: Offload mode is available since NixOS 20.09 and NVIDIA driver version 435.21, and requires an NVIDIA GPU of the Turing generation, or newer and a compatible CPU — either an Intel CPU from the Coffee Lake generation or newer, or an AMD Ryzen. Offload mode is incompatible with sync mode.

Offload mode puts your dGPU to sleep and lets the iGPU handle all tasks, except if you call the dGPU specifically by "offloading" an application to it. For example, you can run your laptop normally and it will use the energy-efficient iGPU all day, and then you can offload a game from Steam onto the dGPU to make the dGPU run that game only. For many, this is the most desirable option.

To enable offload mode, set the hardware.nvidia.prime.offload.enable option to true:

configuration.nix
{
  hardware.nvidia.prime = {
    offload.enable = true;
    
    intelBusId = "PCI:0:2:0";
    nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
  };
}

When you want to run a program on the dGPU, you only need to set a few environment variables for the driver to recognize that offload mode should be used. If hardware.nvidia.prime.offload.enableOffloadCmd is set to true, NixOS will generate a wrapper script named nvidia-offload that sets the right variables for you:

nvidia-offload
export __NV_PRIME_RENDER_OFFLOAD=1
export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
export __GLX_VENDOR_LIBRARY_NAME=nvidia
export __VK_LAYER_NV_optimus=NVIDIA_only
exec "$@"

If everything is configured correctly, then running a program like glxgears should use the iGPU, while running nvidia-offload glxgears should only use the dGPU.

Sync mode

Note: Sync mode is available since NixOS 19.03 and NVIDIA driver version 390.67, and is incompatible with both offload and reverse sync modes. Sync mode also requires using a desktop manager that respects the services.xserver.displayManager.setupCommands option, including LightDM, GDM and SDDM.

In sync mode, rendering is completely delegated to the dGPU, while the iGPU only displays the rendered framebuffers copied from the dGPU. Sync mode offers better performance and greatly reduces screen tearing, at the expense of higher power consumption since the dGPU will not go to sleep when not needed, as is the case in offload mode. Sync mode may also solve some issues with connecting a display in clamshell mode directly to the GPU.

To enable sync mode, set the hardware.nvidia.prime.sync.enable option to true:

/etc/nixos/configuration.nix
{
  hardware.nvidia.prime = {
    sync.enable = true;
    
    intelBusId = "PCI:0:2:0";
    nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
  };
}

Reverse sync mode

Note: Reverse sync mode is available since NixOS 23.05 and NVIDIA driver version 460.39 and is still an experimental, buggy feature[3]. Your mileage may vary. Reverse sync mode is incompatible with sync mode and requires using a desktop manager that respects the services.xserver.displayManager.setupCommands option, including LightDM, GDM and SDDM.

The difference between regular sync mode and reverse sync mode is that the dGPU is configured as the primary output device, allowing displaying to external displays wired to it and not the iGPU (more common).

To enable sync mode, set the hardware.nvidia.prime.reverseSync.enable option to true:

/etc/nixos/configuration.nix
{
  hardware.nvidia.prime = {
    reverseSync.enable = true;

    intelBusId = "PCI:0:2:0";
    nvidiaBusId = "PCI:1:0:0";
    #amdgpuBusId = "PCI:54:0:0"; # If you have an AMD iGPU
  };
}

Tips and tricks

Check nixos-hardware

The nixos-hardware project attempts to provide configurations that address specific hardware quirks for different devices. It is possible that someone already wrote a hardware configuration for your device and that usually takes care of drivers. If so, follow the upstream documentation to enable the required modules.

Multiple boot configurations

Imagine you have a laptop that you mostly use in clamshell mode (docked, connected to an external display and plugged into a charger) but that you sometimes use on the go.

In clamshell mode, using PRIME Sync is likely to lead to better performance, external display support, etc., at the cost of potentially (but not always) lower battery life. However, when using the laptop on the go, you may prefer to use offload mode.

NixOS supports "specialisations", which allow you to automatically generate different boot profiles when rebuilding your system. We can, for example, enable PRIME sync by default, but also create a "on-the-go" specialization that disables PRIME sync and instead enables offload mode:

/etc/nixos/configuration.nix
{
  specialisation.on-the-go.configuration = {
    system.nixos.tags = [ "on-the-go" ];
    hardware.nvidia.prime = {
      offload = {
        enable = lib.mkForce true;
        enableOffloadCmd = lib.mkForce true;
      };
      sync.enable = lib.mkForce false;    
    };
  };
}

(You can also add other settings here totally unrelated to NVIDIA, such as power profiles, etc.)

After rebuilding and rebooting, you'll see in your boot menu under each Generation an "on-the-go" option, which will let you boot into the on-to-go specialisation for that generation.

Using GPUs on non-NixOS

If you're using Nix-packaged software on a non-NixOS system, you'll need a workaround to get everything up-and-running. The nixGL project provides wrapper to use GL drivers on non-NixOS systems. You need to have GPU drivers installed on your distro (for kernel modules). With nixGL installed, you'll run nixGL foobar instead of foobar.

Note that nixGL is not specific to NVIDIA GPUs, and should work with just about any GPU.

CUDA and using your GPU for compute

See the CUDA wiki page.

Running Specific NVIDIA Driver Versions

To run a specific version of the NVIDIA driver in NixOS, you can customize your configuration by specifying the desired version along with the corresponding SHA256 hashes. Below is an example configuration for using NVIDIA driver version 555.58.02:

/etc/nixos/nvidia.nix
{ config, ... }:
{
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.mkDriver {
    version = "555.58.02";
    sha256_64bit = "sha256-xctt4TPRlOJ6r5S54h5W6PT6/3Zy2R4ASNFPu8TSHKM=";
    sha256_aarch64 = "sha256-xctt4TPRlOJ6r5S54h5W6PT6/3Zy2R4ASNFPu8TSHKM=";
    openSha256 = "sha256-ZpuVZybW6CFN/gz9rx+UJvQ715FZnAOYfHn5jt5Z2C8=";
    settingsSha256 = "sha256-ZpuVZybW6CFN/gz9rx+UJvQ715FZnAOYfHn5jt5Z2C8=";
    persistencedSha256 = lib.fakeSha256;
  };
};

In this configuration:

  • Replace version with the desired driver version.
  • Update the SHA256 hashes to match the new version you want to use.
  • After updating the configuration, run sudo nixos-rebuild switch to apply the changes and load the specified NVIDIA driver version.

This allows you to pin the specific driver version being used in your NixOS installation.

Troubleshooting

Booting to text mode

If you encounter the problem of booting to text mode you might try adding the NVIDIA kernel module manually with:

boot.initrd.kernelModules = [ "nvidia" ];
boot.extraModulePackages = [ config.boot.kernelPackages.nvidia_x11 ];

Screen tearing issues

First, try to switch to PRIME sync mode, as described above. If that doesn't work, try forcing a composition pipeline.

Note: Forcing a full composition pipeline has been reported to reduce the performance of some OpenGL applications and may produce issues in WebGL. It also drastically increases the time the driver needs to clock down after load.
/etc/nixos/configuration.nix
hardware.nvidia.forceFullCompositionPipeline = true;

Flickering with Picom

~/.config/picom/picom.conf
unredir-if-possible = false;
backend = "xrender"; # try "glx" if xrender doesn't help
vsync = true;

Graphical corruption and system crashes on suspend/resume

powerManagement.enable = true can sometimes fix this, but is itself unstable and is known to cause suspend issues.

If you have a modern NVIDIA GPU (Turing [1] or later), you may also want to investigate the hardware.nvidia.powerManagement.finegrained option: [2]

Black screen or 'nothing works' on laptops

The kernel modulei915for Intel oramdgpufor AMD may interfere with the NVIDIA driver. This may result in a black screen when switching to the virtual terminal, or when exiting the X session. A possible workaround is to disable the integrated GPU by blacklisting the module, using the following configuration option (see also [3]):

# intel
boot.kernelParams = [ "module_blacklist=i915" ];
# AMD
boot.kernelParams = [ "module_blacklist=amdgpu" ];

Disabling

Kernel modules from NVIDIA

Normally, NVIDIA's kernel modules should be completely disabled by removing "nvidia" from services.xserver.videoDrivers. If that fails to work, you can also manually blacklist the corresponding kernel modules:

{ 
  boot.blacklistedKernelModules = [
    "nvidia"
    "nvidiafb"
    "nvidia-drm"
    "nvidia-uvm"
    "nvidia-modeset"
  ];
}

Nouveau

Nouveau can be disabled by blacklisting the nouveau kernel module:

{ 
  boot.blacklistedKernelModules = [ "nouveau" ];
}

Note that disabling both NVIDIA kernel modules and Nouveau effectively disables the GPU entirely.

Footnotes