Kubernetes: Difference between revisions
Add more nix-based kubernetes tools |
m fix styling |
||
| Line 17: | Line 17: | ||
Caveats: | Caveats: | ||
* | * This was only tested on <code>20.09pre215024.e97dfe73bba (Nightingale)</code> (<code>unstable</code>) | ||
* | * This is probably not best-practice | ||
** | ** For a production-grade cluster you shouldn't use <code>easyCerts</code> | ||
* If you experience inability to reach service CIDR from pods, disable firewall via <code>networking.firewall.enable = false;</code> or otherwise make sure that it doesn't interfere with packet forwarding. | * If you experience inability to reach service CIDR from pods, disable firewall via <code>networking.firewall.enable = false;</code> or otherwise make sure that it doesn't interfere with packet forwarding. | ||
* Make sure to set <code>docker0</code> in promiscuous mode <code>ip link set docker0 promisc on</code> | * Make sure to set <code>docker0</code> in promiscuous mode <code>ip link set docker0 promisc on</code> | ||
| Line 137: | Line 137: | ||
According to the [https://github.com/NixOS/nixpkgs/blob/18ff53d7656636aa440b2f73d2da788b785e6a9c/nixos/tests/kubernetes/rbac.nix#L118 NixOS tests], make your Node join the cluster: | According to the [https://github.com/NixOS/nixpkgs/blob/18ff53d7656636aa440b2f73d2da788b785e6a9c/nixos/tests/kubernetes/rbac.nix#L118 NixOS tests], make your Node join the cluster: | ||
On the master, grab the apitoken | |||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
cat /var/lib/kubernetes/secrets/apitoken.secret | cat /var/lib/kubernetes/secrets/apitoken.secret | ||
</syntaxhighlight> | </syntaxhighlight> | ||
On the node, join the node with | |||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
echo TOKEN | nixos-kubernetes-node-join | echo TOKEN | nixos-kubernetes-node-join | ||
| Line 230: | Line 230: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== | === Reset to a clean state === | ||
Sometimes it helps to have a clean state on all instances: | Sometimes it helps to have a clean state on all instances: | ||
| Line 248: | Line 248: | ||
To do so, I found it necessary to change a few things (tested with <code>rook v1.2</code>): | To do so, I found it necessary to change a few things (tested with <code>rook v1.2</code>): | ||
* | * You need the <code>ceph</code> kernel module: <code>boot.kernelModules = [ "ceph" ];</code> | ||
* | * Change the root dir of the kubelet: <code>kubelet.extraOpts = "--root-dir=/var/lib/kubelet";</code> | ||
* | * Reboot all your nodes | ||
* | * Continue with [https://rook.io/docs/rook/v1.2/ceph-quickstart.html the official quickstart guide] | ||
* | * In <code>operator.yaml</code>, help the CSI plugins find the hosts' ceph kernel modules by adding (or uncommenting -- they're in the example config) these entries: | ||
CSI_CEPHFS_PLUGIN_VOLUME: | | CSI_CEPHFS_PLUGIN_VOLUME: | | ||
- name: lib-modules | - name: lib-modules | ||
| Line 336: | Line 336: | ||
There are various community projects aimed at facilitating working with Kubernetes combined with Nix: | There are various community projects aimed at facilitating working with Kubernetes combined with Nix: | ||
* [https://github.com/saschagrunert/kubernix kubernix]: simple setup of development clusters using Nix | * [https://github.com/saschagrunert/kubernix kubernix]: simple setup of development clusters using Nix | ||
* [https://kubenix.org/ kubenix] | * [https://kubenix.org/ kubenix]: [https://github.com/hall/kubenix GitHub (updated 2023)] | ||
* [https://github.com/justinas/nixos-ha-kubernetes nixos-ha-kubernetes] | * [https://github.com/justinas/nixos-ha-kubernetes nixos-ha-kubernetes] | ||
* [https://github.com/nix-community/nixhelm nixhelm] | * [https://github.com/nix-community/nixhelm nixhelm]: generates nix expressions from a selection of helm charts | ||
* [https://github.com/reMarkable/helmfile-nix helmfile-nix] | * [https://github.com/reMarkable/helmfile-nix helmfile-nix]: wrapper around [[Helm and Helmfile|Helmfile]] to allow writing helmfiles in the nix language | ||
== References == | == References == | ||
* [https://github.com/NixOS/nixpkgs/issues/39327 Issue #39327]: | * [https://github.com/NixOS/nixpkgs/issues/39327 Issue #39327]: Kubernetes support is missing some documentation | ||
* [https://discourse.nixos.org/t/kubernetes-using-multiple-nodes-with-latest-unstable/3936 NixOS Discourse]: Using multiple nodes on unstable | * [https://discourse.nixos.org/t/kubernetes-using-multiple-nodes-with-latest-unstable/3936 NixOS Discourse]: Using multiple nodes on unstable | ||
* [https://kubernetes.io/docs/home/ Kubernetes docs] | * [https://kubernetes.io/docs/home/ Kubernetes docs] | ||