Official NixOS Wiki

Ceph: Difference between revisions

← Older edit
VisualWikitext
imported>C4lliope
Nearly able to make a volume.
imported>Nh2
nixpkgs Ceph has had current versions for a while; soften SeaweedFS plug
 
(4 intermediate revisions by one other user not shown)
Line 1: Line 1:
The Ceph Nix package has been hard-pressed to keep up with Ceph, as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.
The below wiki instructions still have some troubles; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences.
 
Another distributed filesystem alternative you may evaluate is [https://github.com/seaweedfs/seaweedfs SeaweedFS].


----
----


Here is a quick collection of commands I used on a 3-node Ceph mesh.
Here is a quick collection of commands I used on a 3-node Ceph mesh.
The examples have been reduced to a single node, `mesh-a`, for simplicity.


Describe your ceph user, alongside your normal login user:
Describe your ceph user, alongside your normal login user:
Line 29: Line 32:
       enable = false;
       enable = false;
       extraConfig = {
       extraConfig = {
         "mon initial members" = "mesh-a,mesh-b,mesh-c";
         "mon initial members" = "mesh-a";
         "mon host" = "10.0.0.11,10.0.0.12,10.0.0.13";
         "mon host" = "10.0.0.11";
       };
       };
     };
     };
Line 37: Line 40:
</syntaxhighlight>
</syntaxhighlight>


Make your OSD volume; run these commands on each node:
Some preparation is needed so Ceph can run the monitors.
(based on https://docs.ceph.com/en/quincy/install/manual-deployment/ )
You'll need to run these commands on each node
(based on https://docs.ceph.com/en/quincy/install/manual-deployment/ ):


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 44: Line 48:
export FSID=4b687c5c-5a20-4a77-8774-487989fd0bc7
export FSID=4b687c5c-5a20-4a77-8774-487989fd0bc7


# Make your paths!
sudo -u ceph mkdir -p /etc/ceph
sudo -u ceph mkdir -p /etc/ceph
sudo -u ceph mkdir -p /var/lib/ceph/bootstrap-osd
sudo -u ceph mkdir -p /var/lib/ceph/bootstrap-osd
Line 61: Line 66:
</syntaxhighlight>
</syntaxhighlight>


Bind all Ceph OSD shares using systemd (based on <code>u/imspacekitteh</code>'s example):
Prepare <code>systemd</code> to bind all Ceph OSD shares (based on <code>u/imspacekitteh</code>'s example, see links):


<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
Line 89: Line 94:
</pre>
</pre>


Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! I'd assumed the Nixpkgs module should make that, based on all the <code>extraConfig</code> options supplied.
Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! The Nixpkgs module should be upgraded to handle this someday, based on our supplied <code>extraConfig</code> options.


So make the necessary config; this should be minimally enough to load Ceph, and we can come back and nixify this soon:
Bypass the error by making the necessary config; this should be minimally enough to load Ceph:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 125: Line 130:
Dec 19 16:12:51 mesh-a systemd[1]: ceph-mesh.service: Deactivated successfully.
Dec 19 16:12:51 mesh-a systemd[1]: ceph-mesh.service: Deactivated successfully.
Dec 19 16:12:51 mesh-a systemd[1]: Finished Ceph OSD Bindings.
Dec 19 16:12:51 mesh-a systemd[1]: Finished Ceph OSD Bindings.


mesh@mesh-a:~/.build/ > sudo ceph-volume lvm list
mesh@mesh-a:~/.build/ > sudo ceph-volume lvm list
Line 146: Line 150:


Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online.
Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online.
Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of 2023-12) Ceph is on v19.
Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of 2023-12) Ceph is on v18.


* https://d.moonfire.us/blog/2022/12/10/ceph-and-nixos/
* https://d.moonfire.us/blog/2022/12/10/ceph-and-nixos/
* https://github.com/NixOS/nixpkgs/issues/147801
* https://github.com/NixOS/nixpkgs/issues/147801
* https://www.reddit.com/r/ceph/comments/14otjyo/ceph_on_nixos/
* https://www.reddit.com/r/ceph/comments/14otjyo/ceph_on_nixos/
Retrieved from "https://wiki.nixos.org/wiki/Ceph"