K3s: Difference between revisions

From NixOS Wiki
imported>Mcsaucy
document that you need ports 2379 and 2380 for etcd peers and clients if using HA with etcd
imported>Georgiancamarasan
m Fixed minor spellcheck issues
Line 26: Line 26:
See this [https://github.com/Mic92/doctor-cluster-config/tree/master/modules/k3s real world example]. You might want to ignore some parts of it i.e. the monitoring as this is specific to our setup.
See this [https://github.com/Mic92/doctor-cluster-config/tree/master/modules/k3s real world example]. You might want to ignore some parts of it i.e. the monitoring as this is specific to our setup.
The K3s server needs to import <code>modules/k3s/server.nix</code> and an agent <code>modules/k3s/agent.nix</code>.
The K3s server needs to import <code>modules/k3s/server.nix</code> and an agent <code>modules/k3s/agent.nix</code>.
Tipp: You might run into issues with coredns not being reachable from agent nodes. Right now we disable the NixOS firewall all together until we find a better solution.
Tip: You might run into issues with coredns not being reachable from agent nodes. Right now, we disable the NixOS firewall all together until we find a better solution.


== ZFS support ==
== ZFS support ==


K3s's builtin containerd does not support the zfs snapshotter. However it is possible to configure it to use an external containerd:
K3s's builtin containerd does not support the zfs snapshotter. However, it is possible to configure it to use an external containerd:


<syntaxHighlight lang=nix>
<syntaxHighlight lang=nix>
Line 59: Line 59:
== Network policies ==
== Network policies ==


The current k3s derivation doesn't include <code>ipset</code> package which is required by the network policy controller.
The current k3s derivation doesn't include <code>ipset</code> package, which is required by the network policy controller.


k3s logs
k3s logs
Line 66: Line 66:
</syntaxHighlight>
</syntaxHighlight>


There is an open pull request to fix it https://github.com/NixOS/nixpkgs/pull/176520#pullrequestreview-1304593562. Until then the package can be added to k3s's path as follow
There is an open pull request to fix it https://github.com/NixOS/nixpkgs/pull/176520#pullrequestreview-1304593562. Until then, the package can be added to k3s's path as follows
<syntaxHighlight lang=nix>
<syntaxHighlight lang=nix>
   systemd.services.k3s.path = [ pkgs.ipset ];
   systemd.services.k3s.path = [ pkgs.ipset ];
Line 75: Line 75:
=== Raspberry Pi not working ===
=== Raspberry Pi not working ===


If the k3s.service/k3s server does not start and gives you th error <code>FATA[0000] failed to find memory cgroup (v2)</code> Here's the github issue: https://github.com/k3s-io/k3s/issues/2067 .
If the k3s.service/k3s server does not start and gives you the error <code>FATA[0000] failed to find memory cgroup (v2)</code> Here's the github issue: https://github.com/k3s-io/k3s/issues/2067 .


To fix the problem you can add these things to your configuration.nix.
To fix the problem, you can add these things to your configuration.nix.


<source lang="nix">  boot.kernelParams = [
<source lang="nix">  boot.kernelParams = [

Revision as of 11:45, 26 October 2023

K3s is a simplified version of Kubernetes. It bundles all components for a kubernetes cluster into a few of small binaries.

Single node setup

{

  networking.firewall.allowedTCPPorts = [
    6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
    # 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
    # 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
  ];
  services.k3s.enable = true;
  services.k3s.role = "server";
  services.k3s.extraFlags = toString [
    # "--kubelet-arg=v=4" # Optionally add additional args to k3s
  ];
  environment.systemPackages = [ pkgs.k3s ];
}

After enabling, you can access your cluster through sudo k3s kubectl i.e. sudo k3s kubectl cluster-info, or by using the generated kubeconfig file in /etc/rancher/k3s/k3s.yaml

Multi-node setup

See this real world example. You might want to ignore some parts of it i.e. the monitoring as this is specific to our setup. The K3s server needs to import modules/k3s/server.nix and an agent modules/k3s/agent.nix. Tip: You might run into issues with coredns not being reachable from agent nodes. Right now, we disable the NixOS firewall all together until we find a better solution.

ZFS support

K3s's builtin containerd does not support the zfs snapshotter. However, it is possible to configure it to use an external containerd:

  virtualisation.containerd = {
    enable = true;
    settings =
      let
        fullCNIPlugins = pkgs.buildEnv {
          name = "full-cni";
          paths = with pkgs;[
            cni-plugins
            cni-plugin-flannel
          ];
        };
      in {
        plugins."io.containerd.grpc.v1.cri".cni = {
          bin_dir = "${fullCNIPlugins}/bin";
          conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d/";
        };
      };
  };
  # TODO describe how to enable zfs snapshotter in containerd
  services.k3s.extraFlags = toString [
    "--container-runtime-endpoint unix:///run/containerd/containerd.sock"
  ];

Network policies

The current k3s derivation doesn't include ipset package, which is required by the network policy controller.

k3s logs

level=warning msg="Skipping network policy controller start, ipset unavailable: ipset utility not found"

There is an open pull request to fix it https://github.com/NixOS/nixpkgs/pull/176520#pullrequestreview-1304593562. Until then, the package can be added to k3s's path as follows

  systemd.services.k3s.path = [ pkgs.ipset ];

Troubleshooting

Raspberry Pi not working

If the k3s.service/k3s server does not start and gives you the error FATA[0000] failed to find memory cgroup (v2) Here's the github issue: https://github.com/k3s-io/k3s/issues/2067 .

To fix the problem, you can add these things to your configuration.nix.

  boot.kernelParams = [
    "cgroup_enable=cpuset" "cgroup_memory=1" "cgroup_enable=memory"
  ];