Incus: Difference between revisions

(10 intermediate revisions by 9 users not shown)

Line 1:

~~Incus~~ is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's LXD, keeping the Apache-2.0 license.

[https://linuxcontainers.org/incus/ {{PAGENAME}}] (<s>[[wikipedia:en:LXC#LXD]]</s>) is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's [[LXD]], keeping the Apache-2.0 license.

This document aims to provide NixOS specific information related to Incus. For non-NixOS specific documentation, please see the upstream documentation: https://linuxcontainers.org/incus/docs/main/

Line 6:

The service can be enabled and started by adding the service to your NixOS configuration. It must still be initialized.

virtualisation.incus.enable = true;

To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to ~~logout and log back in~~.

networking.nftables.enable = true;

See [[#Networking/Firewall]] for more information on the latter option.

To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to reboot.

users.users.YOUR_USERNAME.extraGroups = ["incus-admin"];

You should now be able to use the incus client to talk to the server.<syntaxhighlight lang="shell-session">

Line 72:

Line 75:

== Networking/Firewall ==

When using Incus on NixOS, nftables is required to ensure broadest compatibility with other services that manage firewall rules ~~from release 24.05~~. Trying to use iptables will fail eval, and this can be fixed by switching to nftables and for simple firewalls should be a drop-in replacement for iptables.

When using Incus on NixOS, nftables is required to ensure broadest compatibility with other services that manage firewall rules. Trying to use iptables will fail eval, and this can be fixed by switching to nftables and for simple firewalls should be a drop-in replacement for iptables.<syntaxhighlight lang="nix">

networking.nftables.enable = true;

By default the NixOS firewall will block DHCP requests to the Incus network, meaning instances will not get an IPv4 address. ~~The simplest fix~~ for ~~this is to mark the~~ Incus ~~bridged interface as trusted~~. This interface name should match the name given during initialization or configured through the incus ~~command line~~.

</syntaxhighlight>

networking.firewall.trustedInterfaces = [ "incusbr0" ];

By default the NixOS firewall will block DHCP requests to the Incus network, meaning instances will not get an IPv4 address. Ensure you allow 53 for DNS and 67 for DHCPv4 on any Incus bridge network interfaces. This interface name should match the name given during initialization or configured through the incus interfaces.<syntaxhighlight lang="nix">

networking.firewall.interfaces.incusbr0.allowedTCPPorts = [

53

67

];

networking.firewall.interfaces.incusbr0.allowedUDPPorts = [

53

67

];

</syntaxhighlight>

OR, the entire interface can be trusted. <syntaxhighlight lang="nix">

networking.firewall.trustedInterfaces = [ "incusbr0" ];

</syntaxhighlight>

Prior to version [https://nixos.org/blog/announcements/2023/nixos-2311/ NixOS 23.11 "Tapir"] the default behavior of the NixOS nftables [[Firewall]] implementation was to flush the full ruleset at any change to the nftables rules. This behavior has since been changed; however, for back-portability, it still persists in configurations with a {{nixos:option|system.stateVersion}} value set prior to <code>"23.11"</code>. This often results in the Incus ruleset table (named "incus") being wiped, resulting in loss of connectivity across VMs and containers. To prevent this from occurring on affected NixOS instances, the new implementation has to be enabled manually.<syntaxhighlight lang="nix">

networking.nftables.flushRuleset = false;

</syntaxhighlight>

== NixOS Instances ==

Line 85:

Line 103:

A NixOS virtual machine is launched with the following.

incus launch images:nixos/unstable nixos -c security.secureboot=false

incus launch --vm images:nixos/unstable nixos -c security.secureboot=false

== NixOS Images ==

Line 103:

Line 122:

Container and VM images are built by Hydra as part of the [https://github.com/NixOS/nixpkgs/blob/master/nixos/release.nix NixOS release].

https://hydra.nixos.org/job/nixos/trunk-combined/nixos.~~lxdContainerImage~~.x86_64-linux

https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux

https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusVirtualMachineImage.x86_64-linux

~~https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdVirtualMachineImage.x86_64-linux~~

Line 115:

Line 135:

CI: https://jenkins.linuxcontainers.org/job/image-nixos/

=== Custom Images ===

== Custom Images ==

All the necessary build infrastructure exists in nixpkgs to build custom images.

=== VMs ===

All the necessary build infrastructure exists in nixpkgs to build custom images.

Define some NixOS systems.<syntaxhighlight lang="nix">

Line 136:

Line 160:

system = "x86_64-linux";

modules = [

"${inputs.nixpkgs}/nixos/modules/virtualisation/~~lxd~~-virtual-machine.nix"

"${inputs.nixpkgs}/nixos/modules/virtualisation/incus-virtual-machine.nix"

(

{ pkgs, ... }:

Line 148:

Line 172:

</syntaxhighlight>Then you can build the image and associated metadata.<syntaxhighlight lang="shell-session">

# nix build .#nixosConfigurations.~~container~~.config.system.build.~~squashfs~~ --print-out-paths

$ nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths

/nix/store/~~24djf2qlpkyh29va8z6pxrqp8x5z6xyv~~-nixos-~~lxc~~-image~~-x86_64-linux.img~~

/nix/store/znk28bp34bycb3h5k0byb61bwda23q5l-nixos-disk-image

# nix build .#nixosConfigurations.~~container~~.config.system.build.metadata --print-out-paths

$ nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths

/nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball

# nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths

</syntaxhighlight>Finally, you can manually import into an Incus storage pool and used to launch instances.<syntaxhighlight lang="bash">

/nix/store/~~znk28bp34bycb3h5k0byb61bwda23q5l~~-nixos-~~disk~~-image

$ incus image import --alias nixos-gen/custom/jellyfin /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball/tarball/nixos-system-x86_64-linux.tar.xz /nix/store/znk28bp34bycb3h5k0byb61bwda23q5l-nixos-disk-image/nixos.qcow2

</syntaxhighlight>To build and import the VM in one command, follow the steps below.<syntaxhighlight lang="bash">

$ incus image import --alias nixos-gen/custom/jellyfin $(nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths)/tarball/nixos-system-x86_64-linux.tar.xz $(nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths)/nixos.qcow2

# Image imported with fingerprint: ***

</syntaxhighlight>You can verify the import with the commands below. <syntaxhighlight lang="bash">

$ incus image list nixos/custom/vm

+------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+

+------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+

| nixos/custom/vm | 9d0d6f3df0cc | no | NixOS Uakari 24.05.20240513.a39a12a x86_64-linux | x86_64 | CONTAINER | 170.31MiB | 2024/05/21 09:21 EDT |

+------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+

$ incus launch nixos/custom/vm

Launching the instance

Instance name is: square-heron

$ incus shell square-heron

[root@nixos:~]# which vim

/run/current-system/sw/bin/vim

</syntaxhighlight>

=== Containers ===

$ nix build .#nixosConfigurations.container.config.system.build.squashfs --print-out-paths

/nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img

# nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths

$ nix build .#nixosConfigurations.container.config.system.build.metadata --print-out-paths

/nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball

</syntaxhighlight><syntaxhighlight lang="shell-session">

</syntaxhighlight>~~Finally, the image can be imported into an Incus storage pool and used to launch instances.~~<syntaxhighlight lang="shell-session">

$ incus image import --alias nixos/custom/container /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball/tarball/nixos-system-x86_64-linux.tar.xz /nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img

# incus image import --alias nixos/custom/container /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball/tarball/nixos-system-x86_64-linux.tar.xz /nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img

Image imported with fingerprint: 9d0d6f3df0cccec4da7ce4f69952bd389b6dd655fd9070e498f591aaffbb2cda

# incus image list nixos/custom/container

$ incus image list nixos/custom/container

+------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+

Line 171:

Line 225:

+------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+

# incus launch nixos/custom/container -c security.nesting=true

$ incus launch nixos/custom/container -c security.nesting=true

Launching the instance

Instance name is: square-heron

# incus shell square-heron

$ incus shell square-heron

[root@nixos:~]# which vim

[root@nixos:~] which vim

/run/current-system/sw/bin/vim

</syntaxhighlight>Or, the all in one command:<syntaxhighlight lang="bash">

incus image import --alias nixos/custom/vm $(nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths)/tarball/nixos-system-x86_64-linux.tar.xz $(nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths)/nixos.qcow2

</syntaxhighlight>

[[Category:Server]]

[[Category:Container]]

[[Category:Virtualization]]

@@ Line 1: / Line 1: @@
-Incus is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's LXD, keeping the Apache-2.0 license.
+[https://linuxcontainers.org/incus/ {{PAGENAME}}] (<s>[[wikipedia:en:LXC#LXD]]</s>) is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's [[LXD]], keeping the Apache-2.0 license.
 This document aims to provide NixOS specific information related to Incus. For non-NixOS specific documentation, please see the upstream documentation: https://linuxcontainers.org/incus/docs/main/
@@ Line 6: / Line 6: @@
 The service can be enabled and started by adding the service to your NixOS configuration. It must still be initialized.
   virtualisation.incus.enable = true;
-To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to logout and log back in.
+ networking.nftables.enable = true;
+See [[#Networking/Firewall]] for more information on the latter option.
+To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to reboot.
   users.users.YOUR_USERNAME.extraGroups = ["incus-admin"];
 You should now be able to use the incus client to talk to the server.<syntaxhighlight lang="shell-session">
@@ Line 72: / Line 75: @@
 == Networking/Firewall ==
-When using Incus on NixOS, nftables is required to ensure broadest compatibility with other services that manage firewall rules from release 24.05. Trying to use iptables will fail eval, and this can be fixed by switching to nftables and for simple firewalls should be a drop-in replacement for iptables.
+When using Incus on NixOS, nftables is required to ensure broadest compatibility with other services that manage firewall rules. Trying to use iptables will fail eval, and this can be fixed by switching to nftables and for simple firewalls should be a drop-in replacement for iptables.<syntaxhighlight lang="nix">
- networking.nftables.enable = true;
+networking.nftables.enable = true;
-By default the NixOS firewall will block DHCP requests to the Incus network, meaning instances will not get an IPv4 address. The simplest fix for this is to mark the Incus bridged interface as trusted. This interface name should match the name given during initialization or configured through the incus command line.
+</syntaxhighlight>
- networking.firewall.trustedInterfaces = [ "incusbr0" ];
+By default the NixOS firewall will block DHCP requests to the Incus network, meaning instances will not get an IPv4 address. Ensure you allow 53 for DNS and 67 for DHCPv4 on any Incus bridge network interfaces. This interface name should match the name given during initialization or configured through the incus interfaces.<syntaxhighlight lang="nix">
+networking.firewall.interfaces.incusbr0.allowedTCPPorts = [
+];
+networking.firewall.interfaces.incusbr0.allowedUDPPorts = [
+];
+</syntaxhighlight>
+OR, the entire interface can be trusted. <syntaxhighlight lang="nix">
+networking.firewall.trustedInterfaces = [ "incusbr0" ];
+</syntaxhighlight>
+Prior to version [https://nixos.org/blog/announcements/2023/nixos-2311/ NixOS 23.11 "Tapir"] the default behavior of the NixOS nftables [[Firewall]] implementation was to flush the full ruleset at any change to the nftables rules. This behavior has since been changed; however, for back-portability, it still persists in configurations with a {{nixos:option|system.stateVersion}} value set prior to <code>"23.11"</code>. This often results in the Incus ruleset table (named "incus") being wiped, resulting in loss of connectivity across VMs and containers. To prevent this from occurring on affected NixOS instances, the new implementation has to be enabled manually.<syntaxhighlight lang="nix">
+networking.nftables.flushRuleset = false;
+</syntaxhighlight>
 == NixOS Instances ==
@@ Line 85: / Line 103: @@
 A NixOS virtual machine is launched with the following.
-  incus launch images:nixos/unstable nixos -c security.secureboot=false
+  incus launch --vm images:nixos/unstable nixos -c security.secureboot=false
 == NixOS Images ==
@@ Line 103: / Line 122: @@
 Container and VM images are built by Hydra as part of the [https://github.com/NixOS/nixpkgs/blob/master/nixos/release.nix NixOS release].
-https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdContainerImage.x86_64-linux
+https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusContainerImage.x86_64-linux
+https://hydra.nixos.org/job/nixos/trunk-combined/nixos.incusVirtualMachineImage.x86_64-linux
-https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdVirtualMachineImage.x86_64-linux
@@ Line 115: / Line 135: @@
 CI: https://jenkins.linuxcontainers.org/job/image-nixos/
-=== Custom Images ===
+== Custom Images ==
-All the necessary build infrastructure exists in nixpkgs to build custom images.
+=== VMs ===
+All the necessary build infrastructure exists in nixpkgs to build custom images.
 Define some NixOS systems.<syntaxhighlight lang="nix">
@@ Line 136: / Line 160: @@
      system = "x86_64-linux";
      modules = [
-       "${inputs.nixpkgs}/nixos/modules/virtualisation/lxd-virtual-machine.nix"
+       "${inputs.nixpkgs}/nixos/modules/virtualisation/incus-virtual-machine.nix"
        (
          { pkgs, ... }:
@@ Line 148: / Line 172: @@
 </syntaxhighlight>Then you can build the image and associated metadata.<syntaxhighlight lang="shell-session">
-# nix build .#nixosConfigurations.container.config.system.build.squashfs --print-out-paths
+$ nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths
-/nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img
+/nix/store/znk28bp34bycb3h5k0byb61bwda23q5l-nixos-disk-image
-# nix build .#nixosConfigurations.container.config.system.build.metadata --print-out-paths
+$ nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths
 /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball
-# nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths
+</syntaxhighlight>Finally, you can manually import into an Incus storage pool and used to launch instances.<syntaxhighlight lang="bash">
-/nix/store/znk28bp34bycb3h5k0byb61bwda23q5l-nixos-disk-image
+$ incus image import --alias nixos-gen/custom/jellyfin /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball/tarball/nixos-system-x86_64-linux.tar.xz /nix/store/znk28bp34bycb3h5k0byb61bwda23q5l-nixos-disk-image/nixos.qcow2
+</syntaxhighlight>To build and import the VM in one command, follow the steps below.<syntaxhighlight lang="bash">
+$ incus image import --alias nixos-gen/custom/jellyfin $(nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths)/tarball/nixos-system-x86_64-linux.tar.xz $(nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths)/nixos.qcow2
+# Image imported with fingerprint: ***
+</syntaxhighlight>You can verify the import with the commands below. <syntaxhighlight lang="bash">
+$ incus image list nixos/custom/vm
++------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+
+|         ALIAS          | FINGERPRINT  | PUBLIC |                   DESCRIPTION                    | ARCHITECTURE |   TYPE    |   SIZE    |     UPLOAD DATE      |
++------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+
+| nixos/custom/vm | 9d0d6f3df0cc | no     | NixOS Uakari 24.05.20240513.a39a12a x86_64-linux | x86_64       | CONTAINER | 170.31MiB | 2024/05/21 09:21 EDT |
++------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+
+$ incus launch nixos/custom/vm
+Launching the instance
+Instance name is: square-heron
+$ incus shell square-heron
+[root@nixos:~]# which vim
+/run/current-system/sw/bin/vim
+</syntaxhighlight>
+=== Containers ===
+<syntaxhighlight lang="bash">
+$ nix build .#nixosConfigurations.container.config.system.build.squashfs --print-out-paths
+/nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img
-# nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths
+$ nix build .#nixosConfigurations.container.config.system.build.metadata --print-out-paths
 /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball
+</syntaxhighlight><syntaxhighlight lang="shell-session">
-</syntaxhighlight>Finally, the image can be imported into an Incus storage pool and used to launch instances.<syntaxhighlight lang="shell-session">
+$ incus image import --alias nixos/custom/container /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball/tarball/nixos-system-x86_64-linux.tar.xz /nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img
-# incus image import --alias nixos/custom/container /nix/store/2snjw9y8brfh5gia44jv6bhdhmmdydva-tarball/tarball/nixos-system-x86_64-linux.tar.xz /nix/store/24djf2qlpkyh29va8z6pxrqp8x5z6xyv-nixos-lxc-image-x86_64-linux.img
 Image imported with fingerprint: 9d0d6f3df0cccec4da7ce4f69952bd389b6dd655fd9070e498f591aaffbb2cda
-# incus image list nixos/custom/container
+$ incus image list nixos/custom/container
 +------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+
 |         ALIAS          | FINGERPRINT  | PUBLIC |                   DESCRIPTION                    | ARCHITECTURE |   TYPE    |   SIZE    |     UPLOAD DATE      |
@@ Line 171: / Line 225: @@
 +------------------------+--------------+--------+--------------------------------------------------+--------------+-----------+-----------+----------------------+
-# incus launch nixos/custom/container -c security.nesting=true
+$ incus launch nixos/custom/container -c security.nesting=true
 Launching the instance
 Instance name is: square-heron
-# incus shell square-heron
+$ incus shell square-heron
-[root@nixos:~]# which vim
+[root@nixos:~] which vim
 /run/current-system/sw/bin/vim
+</syntaxhighlight>Or, the all in one command:<syntaxhighlight lang="bash">
+incus image import --alias nixos/custom/vm $(nix build .#nixosConfigurations.vm.config.system.build.metadata --print-out-paths)/tarball/nixos-system-x86_64-linux.tar.xz $(nix build .#nixosConfigurations.vm.config.system.build.qemuImage --print-out-paths)/nixos.qcow2
 </syntaxhighlight>
 [[Category:Server]]
 [[Category:Container]]
+[[Category:Virtualization]]