Ceph

The Ceph Nix package has been hard-pressed to keep up with Ceph, as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.

Here is a quick collection of commands I used on a 3-node Ceph mesh.

Describe your ceph user, alongside your normal login user:

  users.users = {
    mesh = { isNormalUser = true; extraGroups = [ "wheel" "docker" ]; };
    ceph = { isNormalUser = true; extraGroups = [ "wheel" "ceph" ]; };
  };
  users.groups.ceph = {};

Be sure you rebuild so you can assign some paths to the ceph user.

Run uuidgen and assign the response as your fsid; describe your Ceph nodes:

  services.ceph = {
    global.fsid = "4b687c5c-5a20-4a77-8774-487989fd0bc7";
    osd = {
      enable = true;
      daemons = ["0"];
    };
    mon = {
      enable = false;
      extraConfig = {
        "mon initial members" = "mesh-a,mesh-b,mesh-c";
        "mon host" = "10.0.0.11,10.0.0.12,10.0.0.13";
      };
    };
  };

Make your OSD volume; run these commands on each node: (based on https://docs.ceph.com/en/quincy/install/manual-deployment/ )

export IP=<your-node-IP-on-local-LAN>
export FSID=4b687c5c-5a20-4a77-8774-487989fd0bc7

sudo -u ceph mkdir -p /etc/ceph
sudo -u ceph mkdir -p /var/lib/ceph/bootstrap-osd
sudo -u ceph mkdir -p /tmp/monmap
sudo -u ceph mkdir -p /var/lib/ceph/mon/ceph-$(hostname)
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-mon-$(hostname)

# Make a keyring!
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
sudo mkdir -p /var/lib/ceph/bootstrap-osd && sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
sudo chown ceph:ceph /tmp/ceph.mon.keyring

# Make a monitor!
sudo monmaptool --create --add mesh-a $IP --fsid $FSID /tmp/monmap
sudo -u ceph ceph-mon --mkfs -i mon-$(hostname) --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

Bind all Ceph OSD shares using systemd (based on u/imspacekitteh's example):

  systemd.services.ceph-mesh = {
    enable = true;
    description = "Ceph OSD Bindings";
    unitConfig = {
      After = "local-fs.target";
      Wants = "local-fs.target";
    };
    serviceConfig = {
      Type = "oneshot";
      KillMode = "none";
      Environment = "CEPH_VOLUME_TIMEOUT=10000 PATH=$PATH:/run/current-system/sw/bin/";
      ExecStart = "/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT /run/current-system/sw/bin/ceph-volume lvm activate --all --no-systemd'";
      TimeoutSec = 0;
    };
    wantedBy = ["multi-user.target"];
  };

Though these commands seem reliable enough, there are some issues...

mesh@mesh-c:~/.build/ > sudo ceph -s
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! I'd assumed the Nixpkgs module should make that, based on all the extraConfig options supplied.

So I ran in sequence:

sudo su
export FSID="4b687c5c-5a20-4a77-8774-487989fd0bc7"
echo "
[global]
fsid=$FSID
mon initial members = mesh-a
mon host = 10.0.0.11
cluster network = 10.0.0.0/24
" > /etc/ceph/ceph.conf
exit
cat /etc/ceph/ceph.conf

This should be minimally enough to load Ceph, and we can come back and nixify this soon. The problem remains...

mesh@mesh-a:~/.build/ > sudo systemctl restart ceph-mesh

mesh@mesh-a:~/.build/ > sudo systemctl status ceph-mesh
○ ceph-mesh.service - Ceph OSD Bindings
     Loaded: loaded (/etc/systemd/system/ceph-mesh.service; enabled; preset: enabled)
     Active: inactive (dead) since Tue 2023-12-19 16:12:51 EST; 4s ago
    Process: 37570 ExecStart=/bin/sh -c timeout $CEPH_VOLUME_TIMEOUT /run/current-system/sw/bin/ceph-volume lvm activate --all --no-systemd (code=exited, status=0/SUCCESS)
   Main PID: 37570 (code=exited, status=0/SUCCESS)
         IP: 0B in, 0B out
        CPU: 162ms

Dec 19 16:12:51 mesh-a systemd[1]: Starting Ceph OSD Bindings...
Dec 19 16:12:51 mesh-a sh[37571]: --> Was unable to find any OSDs to activate
Dec 19 16:12:51 mesh-a sh[37571]: --> Verify OSDs are present with "ceph-volume lvm list"
Dec 19 16:12:51 mesh-a systemd[1]: ceph-mesh.service: Deactivated successfully.
Dec 19 16:12:51 mesh-a systemd[1]: Finished Ceph OSD Bindings.


mesh@mesh-a:~/.build/ > sudo ceph-volume lvm list
No valid Ceph lvm devices found

Here is a summary of records produced inside /var/lib/ceph:

mesh@mesh-c:~/.build/ > nix-shell -p tree

[nix-shell:~/.build]$ sudo -u ceph tree /var/lib/ceph
/var/lib/ceph
├── bootstrap-osd
│   └── ceph.keyring
└── mon
    ├── ceph-mesh-c
    └── ceph-mon-mesh-c
        ├── keyring
        ├── kv_backend
        └── store.db
            ├── 000004.log
            ├── CURRENT
            ├── IDENTITY
            ├── LOCK
            ├── MANIFEST-000005
            └── OPTIONS-000007

6 directories, 9 files

Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online. Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of 2023-12) Ceph is on v19.