Official NixOS Wiki

NixOS Containers: Difference between revisions

← Older edit
DHCP (talk | contribs)
232 edits
VisualWikitext
mention microVMs as an alternative
DHCP (talk | contribs)
232 edits
m update system.stateVersion to 26.05
 
(18 intermediate revisions by 9 users not shown)
Line 1: Line 1:
== Native NixOS containers ==
Setup native [https://wiki.archlinux.org/title/systemd-nspawn systemd-nspawn] containers, which are running NixOS and are configured and managed by NixOS using the <code>containers</code> directive.


It is possible to configure native [https://wiki.archlinux.org/title/systemd-nspawn systemd-nspawn] containers, which are running NixOS and are configured and managed by NixOS using the <code>containers</code> directive.
See [[Docker]] page for OCI container (Docker, Podman) configuration.


=== Configuration ===
=== Configuration ===


The following example creates a container called <code>nextcloud</code> running the web application [[Nextcloud]]. It will start automatically at boot and has its private network subnet.
The following example creates a container called webserver running a httpd web server. It will start automatically at boot and has its private network subnet.
 
{{file|3=networking.nat = {
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
networking.nat = {
   enable = true;
   enable = true;
  # Use "ve-*" when using nftables instead of iptables
   internalInterfaces = ["ve-+"];
   internalInterfaces = ["ve-+"];
   externalInterface = "ens3";
   externalInterface = "ens3";
Line 16: Line 15:
};
};


containers.nextcloud = {
containers.webserver = {
   autoStart = true;
   autoStart = true;
   privateNetwork = true;
   privateNetwork = true;
Line 25: Line 24:
   config = { config, pkgs, lib, ... }: {
   config = { config, pkgs, lib, ... }: {


     services.nextcloud = {
     services.httpd = {
       enable = true;
       enable = true;
       package = pkgs.nextcloud28;
       adminAddr = "admin@example.org";
      hostName = "localhost";
      config.adminpassFile = "${pkgs.writeText "adminpass" "test123"}"; # DON'T DO THIS IN PRODUCTION - the password file will be world-readable in the Nix Store!
     };
     };


     system.stateVersion = "23.11";
     networking = {
      firewall.allowedTCPPorts = [ 80 ];


    networking = {
      firewall = {
        enable = true;
        allowedTCPPorts = [ 80 ];
      };
       # Use systemd-resolved inside the container
       # Use systemd-resolved inside the container
       # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
       # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
Line 46: Line 39:
     services.resolved.enable = true;
     services.resolved.enable = true;


    system.stateVersion = "26.05";
   };
   };
};
};|name=/etc/nixos/configuration.nix|lang=nix}}
</nowiki>}}


In order to reach the web application on the host system, we have to open [[Firewall]] port 80 and also configure NAT through <code>networking.nat</code>. The web service of the container will be available at http://192.168.100.11
In order to reach the web application on the host system, we have to open [[Firewall]] port 80 and also configure NAT through <code>networking.nat</code>. The web service of the container will be available at http://192.168.100.11
Line 60: Line 53:
'''NAT (Network Address Translation)'''
'''NAT (Network Address Translation)'''


<syntaxhighlight lang="nix">
In order to allow the container to connect to the internet, you have to configure NAT through <code>networking.nat</code>.
</syntaxhighlight>
{{File|3=networking.nat = {
  enable = true;
  # Use "ve-*" when using nftables instead of iptables
  internalInterfaces = ["ve-+"];
  externalInterface = "ens3";
  # Lazy IPv6 connectivity for the container
  enableIPv6 = true;
};|name=/etc/nixos/configuration.nix|lang=nix}}'''Bridge'''


'''Bridge'''
Connect a container to a bridge using Network Manager interfaces:
 
{{File|3=networking = {
<syntaxhighlight lang="nix">
networking = {
   bridges.br0.interfaces = [ "eth0s31f6" ]; # Adjust interface accordingly
   bridges.br0.interfaces = [ "eth0s31f6" ]; # Adjust interface accordingly
    
    
Line 87: Line 85:
   localAddress = "192.168.100.5/24";
   localAddress = "192.168.100.5/24";
   config = { };
   config = { };
};
};|name=/etc/nixos/configuration.nix|lang=nix}}'''Without privateNetwork (simpler)'''
</syntaxhighlight>
 
If the service can be accessed by changing its port, the private network is not needed necessarily. Be careful to not use occupied ports. This example runs an [[Actual]] server on port 3003. It can be accessed through the host at <code>http://localhost:3003</code>. Since <code>privateNetwork</code> is not defined, it defaults to <code>false</code>.
 
{{File|3=containers.actualContainer = {
  autoStart = true;
  config = {...}: {
    services.actual = {
      enable = true;
      settings.port = 3003;
    };
  };
};|name=/etc/nixos/configuration.nix|lang=nix}}


=== Usage ===
=== Usage ===
Line 99: Line 108:
Checking the status of the container
Checking the status of the container
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# systemctl status container@nextcloud
# systemctl status container@webserver
</syntaxhighlight>
</syntaxhighlight>


Login into the container
Login into the container
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# nixos-container root-login nextcloud
# nixos-container root-login webserver
</syntaxhighlight>
</syntaxhighlight>


Start or stop a container
Start or stop a container
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# nixos-container start nextcloud
# nixos-container start webserver
# nixos-container stop nextcloud
# nixos-container stop webserver
</syntaxhighlight>
</syntaxhighlight>


Destroy a container including its file system
Destroy a container including its file system
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# nixos-container destroy nextcloud
# nixos-container destroy webserver
</syntaxhighlight>
</syntaxhighlight>


View log for container
<syntaxhighlight lang="console">
# journalctl -M webserver
</syntaxhighlight>
Further informations are available in the {{manual:nixos|sec=#ch-containers|chapter=NixOS manual}}.
Further informations are available in the {{manual:nixos|sec=#ch-containers|chapter=NixOS manual}}.


== Declarative OCI containers (Docker/Podman) ==
== Tips and tricks ==
 
=== Define and create nixos-container from a Flake file ===
 
We can define and create a custom container called <code>container</code> from a file stored as <code>flake.nix</code>. In this case we use the unstable branch of the nixpkgs repository as a source.
{{File|3={
  inputs.nixpkgs.url = "nixpkgs/nixos-unstable";
 
  outputs = { self, nixpkgs }: {
 
    nixosConfigurations.container = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ({ pkgs, ... }: {
            boot.isContainer = true;
 
            networking.firewall.allowedTCPPorts = [ 80 ];
 
            services.httpd = {
              enable = true;
              adminAddr = "morty@example.org";
            };
          })
        ];
    };
 
  };
}|name=/etc/nixos/configuration.nix|lang=nix}}
 
To create and run that container, enter following commands. In this example the <code>flake.nix</code> file is in the same directory.
<syntaxhighlight lang="console">
# nixos-container create flake-test --flake .
host IP is 10.233.4.1, container IP is 10.233.4.2
 
# nixos-container start flake-test
</syntaxhighlight>


=== Example config ===
=== Use agenix secrets in container ===
<syntaxhighlight lang="nixos">
{ config, pkgs, ... }:


To add <code>agenix</code> secrets to a container bind mount the <code>ssh-host.key</code> and import the <code>agenix.nixosModule</code> and set <code>age.identityPaths</code> [https://discourse.nixos.org/t/secrets-inside-nixos-containers/34403/6 Source]
{{File|3={ agenix, ... }:
{
{
   config.virtualisation.oci-containers.containers = {
 
     hackagecompare = {
   containers."withSecret" = {
       image = "chrissound/hackagecomparestats-webserver:latest";
 
       ports = ["127.0.0.1:3010:3010"];
    # pass the private key to the container for agenix to decrypt the secret
      volumes = [
    bindMounts."/etc/ssh/ssh_host_ed25519_key".isReadOnly = true;
        "/root/hackagecompare/packageStatistics.json:/root/hackagecompare/packageStatistics.json"
 
       ];
     config =
       cmd = [
      {
         "--base-url"
        config,
        "\"/hackagecompare\""
        lib,
        pkgs,
        ...
      }:
       {
        imports = [ agenix.nixosModules.default ]; # import agenix-module into the nixos-container
 
        age.identityPaths = [ "/etc/ssh/ssh_host_ed25519_key" ]; # isn't set automatically when openssh is not setup
        # import the secret
        age.secrets."secret-name" = {
          file = ../secrets/secret.age;
        };
       };
  };
}|name=/etc/nixos/configuration.nix|lang=nix}}
 
=== Bridge together two nixos-containers ===
'''Target:'''
 
Create two containers, both with <code>privateNetwork = true;</code>:
 
* <code>containerA</code> at 192.168.100.2
** which will access <code>containerB</code>
* <code>containerB</code> at 192.168.100.3
** which runs an httpd server at http://localhost:80
 
They should be connected with a bridge <code>br0</code> and both should have internet address.
 
Assuming Network Manager is used, so the introduction of <code>systemd.network</code> should not interfere with the rest of the setup.
 
'''Configuration:'''
 
Create and configure the internet connection and the bridge:
{{File|3=# Give containers access to the internet
networking.nat = {
  enable = true;
  internalInterfaces = [ "br0" ]; # Connect the bridge to the internet
  externalInterface = "wlp5s0";  # Adjust according to your internet interface
  # Lazy IPv6 connectivity for the container
  enableIPv6 = true;
};
 
# Both systemd-networkd and NetworkManager can exist in parallel on the same machine,
# when they manage a distinct set of interfaces.
# If upstream connectivity is managed by NetworkManager (for example, NM handles wifi and networkd does VM networking),
# set systemd.network.wait-online.enable to false so that boot isn't blocked on connectivity that networkd will never provide.
# https://wiki.nixos.org/wiki/Systemd/networkd#When_to_use
systemd.network = {
  enable = true;
  wait-online.enable = false;
  netdevs = {
      # Create the bridge interface
      # Each interface is stored as a seperate file under /etc/systemd/network by default
      # <number>-name is required so that it is not overwritten by other configurations of the same name
      # It is recommended that each filename is prefixed with a number smaller than "70" (e.g. 10-eth0.network).
       # https://www.freedesktop.org/software/systemd/man/latest/systemd.netdev.html
       "20-br0" = {
         netdevConfig = {
          Kind = "bridge";
          Name = "br0";
          # Sets a pre-determined mac address
          # Leave empty if you want the system to auto-assign a mac address to the bridge
          # MACAddress = "10:00:00:00:00:01";
        };
      };
  };
  networks = {
    # Configure the bridge for its desired function
    "40-br0" = {
      matchConfig.Name = "br0";
      # The address of the bridge
      # /29 is the netmask, it creates 2^(32-29) = 8 subnets
      # 2 are reserved (first and last), as Network Address and Broadcast Address
      # The bridge already takes up one subnet, so 3 addresses are already reserved
      # To bridge 2 networks, you need a netmask of <=29 for IPv4
      # https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=29&cip=192.168.100.1&ctype=ipv4&x=Calculate
      # 192.168.100.0 - 192.168.100.7
      # Network Address  Usable Host Range              Broadcast Address
      # 192.168.100.0    192.168.100.1 - 192.168.100.6  192.168.100.7
      address = [
        "192.168.100.1/29"
       ];
       ];
      # bridgeConfig = {};
      # Disable address autoconfig when no IP configuration is required
      # networkConfig.LinkLocalAddressing = "no";
      # linkConfig = {
        # or "routable" with IP addresses configured
        # RequiredForOnline = "carrier";
      # };
     };
     };
   };
   };
}
};|name=/etc/nixos/configuration.nix|lang=nix}}
</syntaxhighlight>
 
Create and configure <code>containerA</code>:
{{File|3=containers.containerA = {
  autoStart = true;
  privateNetwork = true;
  hostBridge = "br0";
  # hostAddress = "192.168.100.1";  # Not used when using hostBridge
  localAddress = "192.168.100.2/29"; # Should have the netmask if hostBridge is used
  config =
    { config, pkgs, lib, ... }:
    {
      system.stateVersion = "26.05";
     
      networking = {
        # Changes the gateway to the Network Address of the bridge, so that it has access to the internet
        # The bridge has access to the internet
        defaultGateway = {
          address = "192.168.100.1";
        };
      };
     
      networking = {
        # Use systemd-resolved inside the container
        # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
        useHostResolvConf = lib.mkForce false;
      };
     
      services.resolved.enable = true;
    };
};|name=/etc/nixos/configuration.nix|lang=nix}}


=== Usage ===
Create and configure <code>containerB</code>:
NixOS uses Podman to run OCI containers. Note that these are '''user-specific''', so running commands with or without sudo can change your output.
{{File|3=containers.containerB = {
  autoStart = true;
  privateNetwork = true;
  hostBridge = "br0";
  # hostAddress = "192.168.100.1";  # Not used when using hostBridge
  localAddress = "192.168.100.3/29"; # Should have the netmask if hostBridge is used
  config =
    { config, pkgs, lib, ... }:
    {
      # test http server that uses port :80
      services.httpd = {
        enable = true;
      };
     
      system.stateVersion = "26.05";
     
      networking = {
        # Changes the gateway to the Network Address of the bridge, so that it has access to the internet
        # The bridge has access to the internet
        defaultGateway = {
          address = "192.168.100.1";
        };
      };
     
      networking = {
        firewall.allowedTCPPorts = [ 80 ];
        # Use systemd-resolved inside the container
        # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
        useHostResolvConf = lib.mkForce false;
      };


      services.resolved.enable = true;
    };
};|name=/etc/nixos/configuration.nix|lang=nix}}


List containers
You can test the connection between <code>containerA</code> and <code>containerB</code> by loggining into <code>containerA</code> and pinging <code>containerB</code>, curling to <code>containerB</code>'s httpd server or pinging an internet website:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# podman ps
# nixos-container root-login containerA
[root@containerA:~]# ping 192.168.100.3 -c3      # Ping containerB
[root@containerA:~]# curl http://192.168.100.3:80 # Curl to containerB's httpd server
[root@containerA:~]# ping nixos.org -c3          # Ping an internet website
</syntaxhighlight>
</syntaxhighlight>


Update image
You can test the connection between the host machine and <code>containerA</code> or <code>containerB</code> by pinging <code>containerA</code>,  pinging <code>containerB</code> and curling to <code>containerB</code>'s httpd server:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# podman restart hackagecompare
$ ping 192.168.100.2 -c3      # Ping containerA
$ ping 192.168.100.3 -c3      # Ping containerB
$ curl http://192.168.100.3:80 # Curl to containerB's httpd server
</syntaxhighlight>
</syntaxhighlight>


List images
Note that with the command <code>ip address</code>, even if the interfaces of the containers are displayed (<code>vb-containerA</code> and <code>vb-containerB</code>), they only have a MAC address assigned, they do not have a separate ip address displayed. For extra configuring, maybe use the option <code>containers.<name>.extraVeths</code>.
<syntaxhighlight lang="console">
# podman ls
</syntaxhighlight>Remove container<syntaxhighlight lang="console">
# podman rm hackagecompare
</syntaxhighlight>


Remove image
Made with help of the <code>systemd.network</code> wiki page<ref>[[Systemd/networkd]]</ref> and this discourse post<ref>https://discourse.nixos.org/t/how-to-connect-two-or-more-nixos-containers-together-their-internet-ports/77674/9?u=blastboomstrice</ref>.
<syntaxhighlight lang="console">
# podman rmi c0d9a5f58afe
</syntaxhighlight>Update image<syntaxhighlight lang="console">
# podman pull chrissound/hackagecomparestats-webserver:latest
</syntaxhighlight>Run interactive shell in running container<syntaxhighlight lang="console">
# podman exec -ti $ContainerId /bin/sh
</syntaxhighlight>


== Troubleshooting ==
== Troubleshooting ==


=== I have changed the host's channel and some services are no longer functional ===
=== I have changed the host's channel and some services are no longer functional ===
'''Symptoms:'''
'''Symptoms:'''
* Lost data in PostgreSQL database
* Lost data in PostgreSQL database
Line 190: Line 378:
* [https://blog.beardhatcode.be/2020/12/Declarative-Nixos-Containers.html Blog Article - Declarative NixOS Containers]
* [https://blog.beardhatcode.be/2020/12/Declarative-Nixos-Containers.html Blog Article - Declarative NixOS Containers]
* [https://discourse.nixos.org/t/extra-container-run-declarative-containers-without-full-system-rebuilds/511 NixOS Discourse - Extra-container: Run declarative containers without full system rebuilds]
* [https://discourse.nixos.org/t/extra-container-run-declarative-containers-without-full-system-rebuilds/511 NixOS Discourse - Extra-container: Run declarative containers without full system rebuilds]
* [https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/virtualization/nixos-container/nixos-container.pl Nixpkgs - nixos-container.pl]
* [https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/ni/nixos-container/nixos-container.pl Nixpkgs - nixos-container.pl]
* [https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/nixos-containers.nix Nixpkgs - nixos-containers.nix]
* [https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/nixos-containers.nix Nixpkgs - nixos-containers.nix]
* [https://nixcademy.com/2023/08/29/nixos-nspawn/ nixos-nspawn]
* [https://nixcademy.com/2023/08/29/nixos-nspawn/ nixos-nspawn]
Retrieved from "https://wiki.nixos.org/wiki/NixOS_Containers"