K3s: Difference between revisions

From NixOS Wiki
imported>Mcsaucy
document that you need ports 2379 and 2380 for etcd peers and clients if using HA with etcd
Klinger (talk | contribs)
Category:Container. Link. Description.
 
(28 intermediate revisions by 11 users not shown)
Line 1: Line 1:
K3s is a simplified version of [[Kubernetes]]. It bundles all components for a kubernetes cluster into a few of small binaries.
[https://k3s.io/ K3s] is a simplified Kubernetes version that bundles Kubernetes cluster components into a few small binaries optimized for Edge and IoT devices.  


== Single node setup ==


<syntaxHighlight lang=nix>
NixOS's K3s documentation is available at:
{


  networking.firewall.allowedTCPPorts = [
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/cluster/k3s/README.md
    6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
    # 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
    # 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
  ];
  services.k3s.enable = true;
  services.k3s.role = "server";
  services.k3s.extraFlags = toString [
    # "--kubelet-arg=v=4" # Optionally add additional args to k3s
  ];
  environment.systemPackages = [ pkgs.k3s ];
}
</syntaxHighlight>


After enabling, you can access your cluster through <code>sudo k3s kubectl</code> i.e. <code>sudo k3s kubectl cluster-info</code>, or by using the generated kubeconfig file in <code>/etc/rancher/k3s/k3s.yaml</code>
[[Category:Container]]
 
== Multi-node setup ==
 
See this [https://github.com/Mic92/doctor-cluster-config/tree/master/modules/k3s real world example]. You might want to ignore some parts of it i.e. the monitoring as this is specific to our setup.
The K3s server needs to import <code>modules/k3s/server.nix</code> and an agent <code>modules/k3s/agent.nix</code>.
Tipp: You might run into issues with coredns not being reachable from agent nodes. Right now we disable the NixOS firewall all together until we find a better solution.
 
== ZFS support ==
 
K3s's builtin containerd does not support the zfs snapshotter. However it is possible to configure it to use an external containerd:
 
<syntaxHighlight lang=nix>
  virtualisation.containerd = {
    enable = true;
    settings =
      let
        fullCNIPlugins = pkgs.buildEnv {
          name = "full-cni";
          paths = with pkgs;[
            cni-plugins
            cni-plugin-flannel
          ];
        };
      in {
        plugins."io.containerd.grpc.v1.cri".cni = {
          bin_dir = "${fullCNIPlugins}/bin";
          conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d/";
        };
      };
  };
  # TODO describe how to enable zfs snapshotter in containerd
  services.k3s.extraFlags = toString [
    "--container-runtime-endpoint unix:///run/containerd/containerd.sock"
  ];
</syntaxHighlight>
 
== Network policies ==
 
The current k3s derivation doesn't include <code>ipset</code> package which is required by the network policy controller.
 
k3s logs
<syntaxHighlight lang=text>
level=warning msg="Skipping network policy controller start, ipset unavailable: ipset utility not found"
</syntaxHighlight>
 
There is an open pull request to fix it https://github.com/NixOS/nixpkgs/pull/176520#pullrequestreview-1304593562. Until then the package can be added to k3s's path as follow
<syntaxHighlight lang=nix>
  systemd.services.k3s.path = [ pkgs.ipset ];
</syntaxHighlight>
 
== Troubleshooting ==
 
=== Raspberry Pi not working ===
 
If the k3s.service/k3s server does not start and gives you th error <code>FATA[0000] failed to find memory cgroup (v2)</code> Here's the github issue: https://github.com/k3s-io/k3s/issues/2067 .
 
To fix the problem you can add these things to your configuration.nix.
 
<source lang="nix">  boot.kernelParams = [
    "cgroup_enable=cpuset" "cgroup_memory=1" "cgroup_enable=memory"
  ];
</source>
 
 
[[Category:Applications]]
[[Category:Server]]
[[Category:orchestration]]

Latest revision as of 21:54, 18 June 2024

K3s is a simplified Kubernetes version that bundles Kubernetes cluster components into a few small binaries optimized for Edge and IoT devices.


NixOS's K3s documentation is available at:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/cluster/k3s/README.md