Incus: Difference between revisions

From NixOS Wiki
Klinger (talk | contribs)
added install, initialize, network/firewall. cleaned up image and instance info.
Line 1: Line 1:
Incus is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's LXD.
Incus is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's LXD, created by those who started LXD.


Linux containers provides installation instructions, including for nixos [https://linuxcontainers.org/incus/docs/main/installing/]. Wide variety of linux distro images are available, including for NixOS [https://images.linuxcontainers.org/].
This document aims to provide NixOS specific information related to Incus. For non-NixOS specific documentation, please see the upstream documentation: https://linuxcontainers.org/incus/docs/main/


== NixOS image ==
== Installation ==
The service can be enabled and started by adding the service to your NixOS configuration. It must still be initialized.
virtualisation.incus.enable = true;
To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to logout and log back in.
users.users.YOUR_USERNAME.extraGroups = ["incus-admin"];
You should now be able to use the incus client to talk to the server.<syntaxhighlight lang="shell-session">
[root@nixos:/etc/nixos]# incus version
If this is your first time running Incus on this machine, you should also run: incus admin init
To start your first container, try: incus launch images:ubuntu/22.04
Or for a virtual machine: incus launch images:ubuntu/22.04 --vm


There is a pre-built NixOS image available at https://images.linuxcontainers.org/ providing both stable and unstable NixOS distributions.
Client version: 6.0.0
Server version: 6.0.0


To list available nixos images you can use <tt>incus image list</tt> command:
</syntaxhighlight>


<pre>
== Initialization ==
$ incus image list images:nixos
As you can see in the above code block, adding the Incus service will provide a working instance of the server, but is not sufficient on its own to have a complete setup.
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
 
|            ALIAS            | FINGERPRINT  | PUBLIC |              DESCRIPTION              | ARCHITECTURE |      TYPE      |  SIZE    |         UPLOAD DATE         |
For more complex setups, please refer to https://linuxcontainers.org/incus/docs/main/howto/initialize/
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
 
| nixos/23.11 (3 more)          | 4a84a70fb432 | yes    | Nixos 23.11 amd64 (20240216_01:02)    | x86_64      | CONTAINER      | 128.94MiB | Feb 16, 2024 at 12:00am (UTC) |
=== Minimal ===
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
The simplest way to initialize, Incus will provide a basic directory backed storage pool and a bridged NAT network with DHCP.
| nixos/23.11 (3 more)          | cee1d901cca2 | yes    | Nixos 23.11 amd64 (20240216_01:02)    | x86_64      | VIRTUAL-MACHINE | 467.54MiB | Feb 16, 2024 at 12:00am (UTC) |
incus admin init --minimal
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
 
| nixos/23.11/arm64 (1 more)    | 0881cc69ae39 | yes    | Nixos 23.11 arm64 (20240216_01:03)    | aarch64      | VIRTUAL-MACHINE | 474.89MiB | Feb 16, 2024 at 12:00am (UTC) |
=== Preseed ===
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
NixOS has an option for providing a preseed to Incus, as documented in the initialize link above. This is a declarative initialization, with the caveat that Incus preseed will never remove a resource created. Here is an example that is similar to the Minimal initialization option.<syntaxhighlight lang="nix">
| nixos/23.11/arm64 (1 more)    | c436a2e0a0e3 | yes    | Nixos 23.11 arm64 (20240216_01:03)    | aarch64      | CONTAINER      | 123.78MiB | Feb 16, 2024 at 12:00am (UTC) |
virtualisation.incus.preseed = {
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
networks = [
| nixos/unstable (3 more)      | 711ad413f1d4 | yes    | Nixos unstable amd64 (20240216_01:03) | x86_64       | VIRTUAL-MACHINE | 467.31MiB | Feb 16, 2024 at 12:00am (UTC) |
    {
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
      config = {
| nixos/unstable (3 more)      | b15681a4f2cf | yes    | Nixos unstable amd64 (20240216_01:03) | x86_64      | CONTAINER      | 174.39MiB | Feb 16, 2024 at 12:00am (UTC) |
        "ipv4.address" = "10.0.100.1/24";
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
        "ipv4.nat" = "true";
| nixos/unstable/arm64 (1 more) | 95242674aa96 | yes    | Nixos unstable arm64 (20240216_01:03) | aarch64      | VIRTUAL-MACHINE | 475.38MiB | Feb 16, 2024 at 12:00am (UTC) |
      };
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
      name = "incusbr0";
| nixos/unstable/arm64 (1 more) | c0a02e8fd464 | yes    | Nixos unstable arm64 (20240216_01:03) | aarch64      | CONTAINER      | 168.44MiB | Feb 16, 2024 at 12:00am (UTC) |
      type = "bridge";
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+-------------------------------+
    }
</pre>
  ];
  profiles = [
    {
      devices = {
        eth0 = {
          name = "eth0";
          network = "incusbr0";
          type = "nic";
        };
        root = {
          path = "/";
          pool = "default";
          size = "35GiB";
          type = "disk";
        };
      };
      name = "default";
    }
  ];
  storage_pools = [
    {
      config = {
        source = "/var/lib/incus/storage-pools/default";
      };
      driver = "dir";
      name = "default";
    }
  ];
};
</syntaxhighlight>
 
== Networking/Firewall ==
When using Incus on NixOS, nftables is required to ensure broadest compatibility with other services that manage firewall rules from release 24.05. Trying to use iptables will fail eval, and this can be fixed by switching to nftables and for simple firewalls should be a drop-in replacement for iptables.
networking.nftables.enable = true;
By default the NixOS firewall will block DHCP requests to the Incus network, meaning instances will not get an IPv4 address. The simplest fix for this is to mark the Incus bridged interface as trusted. This interface name should match the name given during initialization or configured through the incus command line.
networking.firewall.trustedInterfaces = [ "incusbr0" ];
 
== NixOS Images ==
 
=== Pre-built Images ===
NixOS images are available at https://images.linuxcontainers.org/ providing VM and Container images for both stable and unstable NixOS.<syntaxhighlight lang="shell-session">
[root@nixos:/etc/nixos]# incus image list images:nixos
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+----------------------+
|            ALIAS            | FINGERPRINT  | PUBLIC |              DESCRIPTION              | ARCHITECTURE |      TYPE      |  SIZE    |     UPLOAD DATE     |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+----------------------+
| nixos/23.11 (3 more)          | 1e606df4d91a | yes    | Nixos 23.11 amd64 (20240521_01:02)    | x86_64      | CONTAINER      | 124.84MiB | 2024/05/21 00:00 UTC |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+----------------------+
| nixos/23.11 (3 more)          | a96494ff3c46 | yes    | Nixos 23.11 amd64 (20240521_01:02)    | x86_64      | VIRTUAL-MACHINE | 452.43MiB | 2024/05/21 00:00 UTC |
 
</syntaxhighlight>
 
==== Creation ====
Container and VM images are built by Hydra as part of the [https://github.com/NixOS/nixpkgs/blob/master/nixos/release.nix NixOS release].
 
https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdContainerImage.x86_64-linux
 
https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdVirtualMachineImage.x86_64-linux
 
 
The LXC Image Server then consumes them and repackages them using their CI.


To launch a new NixOS container use the following command:
Definition: https://github.com/lxc/lxc-ci/blob/main/jenkins/jobs/image-nixos.yaml


<pre>
CI: https://jenkins.linuxcontainers.org/job/image-nixos/
incus launch images:nixos/unstable nixos -c security.nesting=true
</pre>


<tt>security.nesting=true</tt> is needed for nix to work correctly.
=== Custom Images ===


Your new instance should be running:
== NixOS Instances ==
To launch a new NixOS container use the following command.


<pre>
<pre>
$ incus list
incus launch images:nixos/unstable nixos -c security.nesting=true
+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME  |  STATE  |        IPV4        |                    IPV6                      |  TYPE    | SNAPSHOTS |
+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| nixos | RUNNING | 10.227.60.142 (eth0) | fd42:49ed:2bf3:f0ad:216:3eff:fe73:47bd (eth0) | CONTAINER | 0        |
+-------+---------+----------------------+-----------------------------------------------+-----------+-----------+
 
$ incus exec nixos -- bash
</pre>
</pre>


After that you can run a root shell inside container using <tt>incus exec nixos -- bash</tt>. From there you probably want to add a new user and use ssh to connect via the IP that you can find using <tt>incus list</tt> command. If for some reason instance does not get an IP, it's most likely issue with firewalld and you need to add <tt>incusbr0</tt> to allowed firewall zone.
A NixOS virtual machine is launched with the following.
 
incus launch images:nixos/unstable nixos -c security.secureboot=false
Image is built using linuxcontainers CI job [https://jenkins.linuxcontainers.org/job/image-nixos/], the source of which can be found here [https://github.com/lxc/lxc-ci/blob/main/jenkins/jobs/image-nixos.yaml]. As we can see jenkins CI job downloads <tt>lxdVirtualMachineImage</tt> derivation produced by hydra, which produces qcow2 image file. Configuration for nixos image can be found here [https://github.com/NixOS/nixpkgs/blob/master/nixos/maintainers/scripts/lxd/lxd-virtual-machine-image.nix].




[[Category:Server]]
[[Category:Server]]
[[Category:Container]]
[[Category:Container]]

Revision as of 12:53, 21 May 2024

Incus is a next generation system container and virtual machine manager. It is a community driven alternative to Canonical's LXD, created by those who started LXD.

This document aims to provide NixOS specific information related to Incus. For non-NixOS specific documentation, please see the upstream documentation: https://linuxcontainers.org/incus/docs/main/

Installation

The service can be enabled and started by adding the service to your NixOS configuration. It must still be initialized.

virtualisation.incus.enable = true;

To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to logout and log back in.

users.users.YOUR_USERNAME.extraGroups = ["incus-admin"];

You should now be able to use the incus client to talk to the server.

[root@nixos:/etc/nixos]# incus version
If this is your first time running Incus on this machine, you should also run: incus admin init
To start your first container, try: incus launch images:ubuntu/22.04
Or for a virtual machine: incus launch images:ubuntu/22.04 --vm

Client version: 6.0.0
Server version: 6.0.0

Initialization

As you can see in the above code block, adding the Incus service will provide a working instance of the server, but is not sufficient on its own to have a complete setup.

For more complex setups, please refer to https://linuxcontainers.org/incus/docs/main/howto/initialize/

Minimal

The simplest way to initialize, Incus will provide a basic directory backed storage pool and a bridged NAT network with DHCP.

incus admin init --minimal

Preseed

NixOS has an option for providing a preseed to Incus, as documented in the initialize link above. This is a declarative initialization, with the caveat that Incus preseed will never remove a resource created. Here is an example that is similar to the Minimal initialization option.

virtualisation.incus.preseed = {
 networks = [
    {
      config = {
        "ipv4.address" = "10.0.100.1/24";
        "ipv4.nat" = "true";
      };
      name = "incusbr0";
      type = "bridge";
    }
  ];
  profiles = [
    {
      devices = {
        eth0 = {
          name = "eth0";
          network = "incusbr0";
          type = "nic";
        };
        root = {
          path = "/";
          pool = "default";
          size = "35GiB";
          type = "disk";
        };
      };
      name = "default";
    }
  ];
  storage_pools = [
    {
      config = {
        source = "/var/lib/incus/storage-pools/default";
      };
      driver = "dir";
      name = "default";
    }
  ];
};

Networking/Firewall

When using Incus on NixOS, nftables is required to ensure broadest compatibility with other services that manage firewall rules from release 24.05. Trying to use iptables will fail eval, and this can be fixed by switching to nftables and for simple firewalls should be a drop-in replacement for iptables.

networking.nftables.enable = true;

By default the NixOS firewall will block DHCP requests to the Incus network, meaning instances will not get an IPv4 address. The simplest fix for this is to mark the Incus bridged interface as trusted. This interface name should match the name given during initialization or configured through the incus command line.

networking.firewall.trustedInterfaces = [ "incusbr0" ];

NixOS Images

Pre-built Images

NixOS images are available at https://images.linuxcontainers.org/ providing VM and Container images for both stable and unstable NixOS.

[root@nixos:/etc/nixos]# incus image list images:nixos
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+----------------------+
|             ALIAS             | FINGERPRINT  | PUBLIC |              DESCRIPTION              | ARCHITECTURE |      TYPE       |   SIZE    |     UPLOAD DATE      |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+----------------------+
| nixos/23.11 (3 more)          | 1e606df4d91a | yes    | Nixos 23.11 amd64 (20240521_01:02)    | x86_64       | CONTAINER       | 124.84MiB | 2024/05/21 00:00 UTC |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+-----------+----------------------+
| nixos/23.11 (3 more)          | a96494ff3c46 | yes    | Nixos 23.11 amd64 (20240521_01:02)    | x86_64       | VIRTUAL-MACHINE | 452.43MiB | 2024/05/21 00:00 UTC |

Creation

Container and VM images are built by Hydra as part of the NixOS release.

https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdContainerImage.x86_64-linux

https://hydra.nixos.org/job/nixos/trunk-combined/nixos.lxdVirtualMachineImage.x86_64-linux


The LXC Image Server then consumes them and repackages them using their CI.

Definition: https://github.com/lxc/lxc-ci/blob/main/jenkins/jobs/image-nixos.yaml

CI: https://jenkins.linuxcontainers.org/job/image-nixos/

Custom Images

NixOS Instances

To launch a new NixOS container use the following command.

incus launch images:nixos/unstable nixos -c security.nesting=true

A NixOS virtual machine is launched with the following.

incus launch images:nixos/unstable nixos -c security.secureboot=false