NixOS Containers: Difference between revisions

From NixOS Wiki
update stateVersion
Add infrmation how to get agenix secrets into the container
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Native NixOS containers ==
Setup native [https://wiki.archlinux.org/title/systemd-nspawn systemd-nspawn] containers, which are running NixOS and are configured and managed by NixOS using the <code>containers</code> directive.


It is possible to configure native [https://wiki.archlinux.org/title/systemd-nspawn systemd-nspawn] containers, which are running NixOS and are configured and managed by NixOS using the <code>containers</code> directive.
See [[Docker]] page for OCI container (Docker, Podman) configuration.


=== Configuration ===
=== Configuration ===


The following example creates a container called <code>nextcloud</code> running the web application [[Nextcloud]]. It will start automatically at boot and has its private network subnet.
The following example creates a container called webserver running a httpd web server. It will start automatically at boot and has its private network subnet.


{{file|/etc/nixos/configuration.nix|nix|<nowiki>
{{file|/etc/nixos/configuration.nix|nix|<nowiki>
Line 16: Line 16:
};
};


containers.nextcloud = {
containers.webserver = {
   autoStart = true;
   autoStart = true;
   privateNetwork = true;
   privateNetwork = true;
Line 25: Line 25:
   config = { config, pkgs, lib, ... }: {
   config = { config, pkgs, lib, ... }: {


     services.nextcloud = {
     services.httpd = {
       enable = true;
       enable = true;
       package = pkgs.nextcloud28;
       adminAddr = "admin@example.org";
      hostName = "localhost";
      config.adminpassFile = "${pkgs.writeText "adminpass" "test123"}"; # DON'T DO THIS IN PRODUCTION - the password file will be world-readable in the Nix Store!
     };
     };


     system.stateVersion = "24.11";
     networking = {
      firewall.allowedTCPPorts = [ 80 ];


    networking = {
      firewall = {
        enable = true;
        allowedTCPPorts = [ 80 ];
      };
       # Use systemd-resolved inside the container
       # Use systemd-resolved inside the container
       # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
       # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
Line 46: Line 40:
     services.resolved.enable = true;
     services.resolved.enable = true;


    system.stateVersion = "24.11";
   };
   };
};
};
Line 99: Line 94:
Checking the status of the container
Checking the status of the container
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# systemctl status container@nextcloud
# systemctl status container@webserver
</syntaxhighlight>
</syntaxhighlight>


Login into the container
Login into the container
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# nixos-container root-login nextcloud
# nixos-container root-login webserver
</syntaxhighlight>
</syntaxhighlight>


Start or stop a container
Start or stop a container
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# nixos-container start nextcloud
# nixos-container start webserver
# nixos-container stop nextcloud
# nixos-container stop webserver
</syntaxhighlight>
</syntaxhighlight>


Destroy a container including its file system
Destroy a container including its file system
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
# nixos-container destroy nextcloud
# nixos-container destroy webserver
</syntaxhighlight>
</syntaxhighlight>


Further informations are available in the {{manual:nixos|sec=#ch-containers|chapter=NixOS manual}}.
Further informations are available in the {{manual:nixos|sec=#ch-containers|chapter=NixOS manual}}.


== Declarative OCI containers (Docker/Podman) ==
== Tips and tricks ==
 
==== Define and create nixos-container from a Flake file ====
We can define and create a custom container called <code>container</code> from a file stored as <code>flake.nix</code>. In this case we use the unstable branch of the nixpkgs repository as a source.<syntaxhighlight lang="nix">
{
  inputs.nixpkgs.url = "nixpkgs/nixos-unstable";
 
  outputs = { self, nixpkgs }: {
 
    nixosConfigurations.container = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ({ pkgs, ... }: {
            boot.isContainer = true;


=== Example config ===
            networking.firewall.allowedTCPPorts = [ 80 ];
<syntaxhighlight lang="nixos">
{ config, pkgs, ... }:


{
            services.httpd = {
  config.virtualisation.oci-containers.containers = {
              enable = true;
    hackagecompare = {
              adminAddr = "morty@example.org";
      image = "chrissound/hackagecomparestats-webserver:latest";
            };
      ports = ["127.0.0.1:3010:3010"];
          })
      volumes = [
         ];
        "/root/hackagecompare/packageStatistics.json:/root/hackagecompare/packageStatistics.json"
      ];
      cmd = [
        "--base-url"
         "\"/hackagecompare\""
      ];
     };
     };
   };
   };
}
}
</syntaxhighlight>To create and run that container, enter following commands. In this example the <code>flake.nix</code> file is in the same directory.<syntaxhighlight lang="console">
# nixos-container create flake-test --flake .
host IP is 10.233.4.1, container IP is 10.233.4.2
# nixos-container start flake-test
</syntaxhighlight>
</syntaxhighlight>


=== Usage ===
==== Use agenix secrets in container ====
NixOS uses Podman to run OCI containers. Note that these are '''user-specific''', so running commands with or without sudo can change your output.
To add <code>agenix</code> secrets to a container bind mount the <code>ssh-host.key</code> and import the <code>agenix.nixosModule</code> and set <code>age.identityPaths</code> [https://discourse.nixos.org/t/secrets-inside-nixos-containers/34403/6 Source]<syntaxhighlight lang="nix">
{ agenix, ... }:
{


  containers."withSecret" = {


List containers
    # pass the private key to the container for agenix to decrypt the secret
<syntaxhighlight lang="console">
    bindMounts."/etc/ssh/ssh_host_ed25519_key".isReadOnly = true;
# podman ps
</syntaxhighlight>


Update image
    config =
<syntaxhighlight lang="console">
      {
# podman restart hackagecompare
        config,
</syntaxhighlight>
        lib,
        pkgs,
        ...
      }:
      {
        imports = [ agenix.nixosModules.default ]; # import agenix-module into the nixos-container


List images
        age.identityPaths = [ "/etc/ssh/ssh_host_ed25519_key" ]; # isn't set automatically when openssh is not setup
<syntaxhighlight lang="console">
        # import the secret
# podman ls
        age.secrets."secret-name" = {
</syntaxhighlight>Remove container<syntaxhighlight lang="console">
          file = ../secrets/secret.age;
# podman rm hackagecompare
        };
</syntaxhighlight>
      };
 
  };
Remove image
}
<syntaxhighlight lang="console">
# podman rmi c0d9a5f58afe
</syntaxhighlight>Update image<syntaxhighlight lang="console">
# podman pull chrissound/hackagecomparestats-webserver:latest
</syntaxhighlight>Run interactive shell in running container<syntaxhighlight lang="console">
# podman exec -ti $ContainerId /bin/sh
</syntaxhighlight>
</syntaxhighlight>


== Troubleshooting ==
== Troubleshooting ==


=== I have changed the host's channel and some services are no longer functional ===
==== I have changed the host's channel and some services are no longer functional ====
 
'''Symptoms:'''
'''Symptoms:'''
* Lost data in PostgreSQL database
* Lost data in PostgreSQL database

Latest revision as of 21:45, 8 January 2025

Setup native systemd-nspawn containers, which are running NixOS and are configured and managed by NixOS using the containers directive.

See Docker page for OCI container (Docker, Podman) configuration.

Configuration

The following example creates a container called webserver running a httpd web server. It will start automatically at boot and has its private network subnet.

/etc/nixos/configuration.nix
networking.nat = {
  enable = true;
  internalInterfaces = ["ve-+"];
  externalInterface = "ens3";
  # Lazy IPv6 connectivity for the container
  enableIPv6 = true;
};

containers.webserver = {
  autoStart = true;
  privateNetwork = true;
  hostAddress = "192.168.100.10";
  localAddress = "192.168.100.11";
  hostAddress6 = "fc00::1";
  localAddress6 = "fc00::2";
  config = { config, pkgs, lib, ... }: {

    services.httpd = {
      enable = true;
      adminAddr = "admin@example.org";
    };

    networking = {
      firewall.allowedTCPPorts = [ 80 ];

      # Use systemd-resolved inside the container
      # Workaround for bug https://github.com/NixOS/nixpkgs/issues/162686
      useHostResolvConf = lib.mkForce false;
    };
    
    services.resolved.enable = true;

    system.stateVersion = "24.11";
  };
};

In order to reach the web application on the host system, we have to open Firewall port 80 and also configure NAT through networking.nat. The web service of the container will be available at http://192.168.100.11

Networking

By default, if privateNetwork is not set, the container shares the network with the host, enabling it to bind any port on any interface. However, when privateNetwork is set to true, the container gains its private virtual eth0 and ve-<container_name> on the host. This isolation is beneficial when you want the container to have its dedicated networking stack.

NAT (Network Address Translation)

Bridge

networking = {
  bridges.br0.interfaces = [ "eth0s31f6" ]; # Adjust interface accordingly
  
  # Get bridge-ip with DHCP
  useDHCP = false;
  interfaces."br0".useDHCP = true;

  # Set bridge-ip static
  interfaces."br0".ipv4.addresses = [{
    address = "192.168.100.3";
    prefixLength = 24;
  }];
  defaultGateway = "192.168.100.1";
  nameservers = [ "192.168.100.1" ];
};

containers.<name> = {
  privateNetwork = true;
  hostBridge = "br0"; # Specify the bridge name
  localAddress = "192.168.100.5/24";
  config = { };
};

Usage

List containers

# machinectl list

Checking the status of the container

# systemctl status container@webserver

Login into the container

# nixos-container root-login webserver

Start or stop a container

# nixos-container start webserver
# nixos-container stop webserver

Destroy a container including its file system

# nixos-container destroy webserver

Further informations are available in the NixOS Manual, NixOS manual.

Tips and tricks

Define and create nixos-container from a Flake file

We can define and create a custom container called container from a file stored as flake.nix. In this case we use the unstable branch of the nixpkgs repository as a source.

{
  inputs.nixpkgs.url = "nixpkgs/nixos-unstable";

  outputs = { self, nixpkgs }: {

    nixosConfigurations.container = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ({ pkgs, ... }: {
            boot.isContainer = true;

            networking.firewall.allowedTCPPorts = [ 80 ];

            services.httpd = {
              enable = true;
              adminAddr = "morty@example.org";
            };
          })
        ];
    };

  };
}

To create and run that container, enter following commands. In this example the flake.nix file is in the same directory.

# nixos-container create flake-test --flake .
host IP is 10.233.4.1, container IP is 10.233.4.2

# nixos-container start flake-test

Use agenix secrets in container

To add agenix secrets to a container bind mount the ssh-host.key and import the agenix.nixosModule and set age.identityPaths Source

{ agenix, ... }:
{

  containers."withSecret" = {

    # pass the private key to the container for agenix to decrypt the secret
    bindMounts."/etc/ssh/ssh_host_ed25519_key".isReadOnly = true;

    config =
      {
        config,
        lib,
        pkgs,
        ...
      }:
      {
        imports = [ agenix.nixosModules.default ]; # import agenix-module into the nixos-container

        age.identityPaths = [ "/etc/ssh/ssh_host_ed25519_key" ]; # isn't set automatically when openssh is not setup
        # import the secret
        age.secrets."secret-name" = {
          file = ../secrets/secret.age;
        };
      };
  };
}

Troubleshooting

I have changed the host's channel and some services are no longer functional

Symptoms:

  • Lost data in PostgreSQL database
  • MySQL has changed its path, where it creates the database

Solution

If you did not have a system.stateVersion option set inside your declarative container configuration, it will use the default one for the channel. Your data might be safe, if you did nothing meanwhile. Add the missing system.stateVersion to your container, rebuild, and possibly stop/start the container.

See also