Kubernetes: Difference between revisions
No edit summary |
m fix styling |
||
| (6 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
[https://kubernetes.io/ Kubernetes] is an open-source container orchestration system for automating software deployment, scaling, and management. | |||
This wiki article extends the documentation in [https://nixos.org/manual/nixos/stable/#sec-kubernetes NixOS manual]. | |||
== [[wikipedia:en:KISS principle|KISS]] == | == [[wikipedia:en:KISS principle|KISS]] == | ||
| Line 13: | Line 17: | ||
Caveats: | Caveats: | ||
* | * This was only tested on <code>20.09pre215024.e97dfe73bba (Nightingale)</code> (<code>unstable</code>) | ||
* | * This is probably not best-practice | ||
** | ** For a production-grade cluster you shouldn't use <code>easyCerts</code> | ||
* If you experience inability to reach service CIDR from pods, disable firewall via <code>networking.firewall.enable = false;</code> or otherwise make sure that it doesn't interfere with packet forwarding. | * If you experience inability to reach service CIDR from pods, disable firewall via <code>networking.firewall.enable = false;</code> or otherwise make sure that it doesn't interfere with packet forwarding. | ||
* Make sure to set <code>docker0</code> in promiscuous mode <code>ip link set docker0 promisc on</code> | * Make sure to set <code>docker0</code> in promiscuous mode <code>ip link set docker0 promisc on</code> | ||
| Line 23: | Line 27: | ||
Add to your <code>configuration.nix</code>: | Add to your <code>configuration.nix</code>: | ||
<syntaxhighlight lang=nix> | <syntaxhighlight lang="nix"> | ||
{ config, pkgs, ... }: | { config, pkgs, ... }: | ||
let | let | ||
# When using easyCerts=true the IP | # When using 'easyCerts = true;', the IP address must resolve to the master at the time of creation. | ||
# In this case, set 'kubeMasterIP = "127.0.0.1";'. Otherwise, you may encounter the following issue: https://github.com/NixOS/nixpkgs/issues/59364. | |||
kubeMasterIP = "10.1.1.2"; | kubeMasterIP = "10.1.1.2"; | ||
kubeMasterHostname = "api.kube"; | kubeMasterHostname = "api.kube"; | ||
| Line 133: | Line 137: | ||
According to the [https://github.com/NixOS/nixpkgs/blob/18ff53d7656636aa440b2f73d2da788b785e6a9c/nixos/tests/kubernetes/rbac.nix#L118 NixOS tests], make your Node join the cluster: | According to the [https://github.com/NixOS/nixpkgs/blob/18ff53d7656636aa440b2f73d2da788b785e6a9c/nixos/tests/kubernetes/rbac.nix#L118 NixOS tests], make your Node join the cluster: | ||
On the master, grab the apitoken | |||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
cat /var/lib/kubernetes/secrets/apitoken.secret | cat /var/lib/kubernetes/secrets/apitoken.secret | ||
</syntaxhighlight> | </syntaxhighlight> | ||
On the node, join the node with | |||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
echo TOKEN | nixos-kubernetes-node-join | echo TOKEN | nixos-kubernetes-node-join | ||
| Line 226: | Line 230: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== | === Reset to a clean state === | ||
Sometimes it helps to have a clean state on all instances: | Sometimes it helps to have a clean state on all instances: | ||
| Line 244: | Line 248: | ||
To do so, I found it necessary to change a few things (tested with <code>rook v1.2</code>): | To do so, I found it necessary to change a few things (tested with <code>rook v1.2</code>): | ||
* | * You need the <code>ceph</code> kernel module: <code>boot.kernelModules = [ "ceph" ];</code> | ||
* | * Change the root dir of the kubelet: <code>kubelet.extraOpts = "--root-dir=/var/lib/kubelet";</code> | ||
* | * Reboot all your nodes | ||
* | * Continue with [https://rook.io/docs/rook/v1.2/ceph-quickstart.html the official quickstart guide] | ||
* | * In <code>operator.yaml</code>, help the CSI plugins find the hosts' ceph kernel modules by adding (or uncommenting -- they're in the example config) these entries: | ||
CSI_CEPHFS_PLUGIN_VOLUME: | | CSI_CEPHFS_PLUGIN_VOLUME: | | ||
- name: lib-modules | - name: lib-modules | ||
| Line 332: | Line 336: | ||
There are various community projects aimed at facilitating working with Kubernetes combined with Nix: | There are various community projects aimed at facilitating working with Kubernetes combined with Nix: | ||
* [https://github.com/saschagrunert/kubernix kubernix]: simple setup of development clusters using Nix | * [https://github.com/saschagrunert/kubernix kubernix]: simple setup of development clusters using Nix | ||
* [https://kubenix.org/ kubenix]: [https://github.com/hall/kubenix GitHub (updated 2023)] | |||
* [https://kubenix.org/ kubenix] | |||
* [https://github.com/justinas/nixos-ha-kubernetes nixos-ha-kubernetes] | * [https://github.com/justinas/nixos-ha-kubernetes nixos-ha-kubernetes] | ||
* [https://github.com/nix-community/nixhelm nixhelm]: generates nix expressions from a selection of helm charts | |||
* [https://github.com/reMarkable/helmfile-nix helmfile-nix]: wrapper around [[Helm and Helmfile|Helmfile]] to allow writing helmfiles in the nix language | |||
== References == | == References == | ||
* [https://github.com/NixOS/nixpkgs/issues/39327 Issue #39327]: | * [https://github.com/NixOS/nixpkgs/issues/39327 Issue #39327]: Kubernetes support is missing some documentation | ||
* [https://discourse.nixos.org/t/kubernetes-using-multiple-nodes-with-latest-unstable/3936 NixOS Discourse]: Using multiple nodes on unstable | * [https://discourse.nixos.org/t/kubernetes-using-multiple-nodes-with-latest-unstable/3936 NixOS Discourse]: Using multiple nodes on unstable | ||
* [https://kubernetes.io/docs/home/ Kubernetes docs] | * [https://kubernetes.io/docs/home/ Kubernetes docs] | ||
| Line 347: | Line 352: | ||
[[Category:Applications]] | [[Category:Applications]] | ||
[[Category:Server]] | [[Category:Server]] | ||
[[Category: | [[Category:Container]] | ||
[[Category:NixOS Manual]] | |||