K3s: Difference between revisions

From NixOS Wiki
imported>Vater
mNo edit summary
imported>Azazel75
add flannel config
Line 29: Line 29:


<syntaxHighlight lang=nix>
<syntaxHighlight lang=nix>
   virtualisation.containerd.enable = true;
   virtualisation.containerd = {
    enable = true;
    settings =
      let
        fullCNIPlugins = pkgs.buildEnv {
          name = "full-cni";
          paths = with pkgs;[
            cni-plugins
            cni-plugin-flannel
          ];
        };
      in {
        plugins."io.containerd.grpc.v1.cri".cni = {
          bin_dir = "${fullCNIPlugins}/bin";
          conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d/";
        };
      };
  };
   # TODO describe how to enable zfs snapshotter in containerd
   # TODO describe how to enable zfs snapshotter in containerd
   services.k3s.extraFlags = toString [
   services.k3s.extraFlags = toString [

Revision as of 14:45, 3 May 2022

K3s is a simplified version of Kubernetes. It bundles all components for a kubernetes cluster into a few of small binaries.

Single node setup

{
  # This is required so that pod can reach the API server (running on port 6443 by default)
  networking.firewall.allowedTCPPorts = [ 6443 ];
  services.k3s.enable = true;
  services.k3s.role = "server";
  services.k3s.extraFlags = toString [
    # "--kubelet-arg=v=4" # Optionally add additional args to k3s
  ];
  environment.systemPackages = [ pkgs.k3s ];
}

After enabling, you can access your cluster through sudo k3s kubectl i.e. sudo k3s kubectl cluster-info, or by using the generated kubeconfig file in /etc/rancher/k3s/k3s.yaml

Multi-node setup

See this real world example. You might want to ignore some parts of it i.e. the monitoring as this is specific to our setup. The K3s server needs to import modules/k3s/server.nix and an agent modules/k3s/agent.nix. Tipp: You might run into issues with coredns not being reachable from agent nodes. Right now we disable the NixOS firewall all together until we find a better solution.

ZFS support

K3s's builtin containerd does not support the zfs snapshotter. However it is possible to configure it to use an external containerd:

  virtualisation.containerd = {
    enable = true;
    settings =
      let
        fullCNIPlugins = pkgs.buildEnv {
          name = "full-cni";
          paths = with pkgs;[
            cni-plugins
            cni-plugin-flannel
          ];
        };
      in {
        plugins."io.containerd.grpc.v1.cri".cni = {
          bin_dir = "${fullCNIPlugins}/bin";
          conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d/";
        };
      };
  };
  # TODO describe how to enable zfs snapshotter in containerd
  services.k3s.extraFlags = toString [
    "--container-runtime-endpoint unix:///run/containerd/containerd.sock"
  ];