Ceph - Revision history

imported>Nh2: nixpkgs Ceph has had current versions for a while; soften SeaweedFS plug

2024-02-17T04:57:18Z

nixpkgs Ceph has had current versions for a while; soften SeaweedFS plug

← Older revision		Revision as of 04:57, 17 February 2024
Line 1:		Line 1:
	The ~~Ceph Nix package has been hard-pressed to keep up with [https://ceph.io Ceph], as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution~~; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences~~, if you made progress running a modern Ceph version~~.		The below wiki instructions still have some troubles; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences.

	~~Also, the year~~ is ~~2024. You probably should use~~ [https://github.com/seaweedfs/seaweedfs SeaweedFS].		Another distributed filesystem alternative you may evaluate is [https://github.com/seaweedfs/seaweedfs SeaweedFS].

	----		----
Line 150:		Line 150:

	Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online.		Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online.
	Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of 2023-12) Ceph is on ~~v19~~.		Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of 2023-12) Ceph is on v18.

	* https://d.moonfire.us/blog/2022/12/10/ceph-and-nixos/		* https://d.moonfire.us/blog/2022/12/10/ceph-and-nixos/
	* https://github.com/NixOS/nixpkgs/issues/147801		* https://github.com/NixOS/nixpkgs/issues/147801
	* https://www.reddit.com/r/ceph/comments/14otjyo/ceph_on_nixos/		* https://www.reddit.com/r/ceph/comments/14otjyo/ceph_on_nixos/

imported>C4lliope: Recommend SeaweedFS.

2023-12-28T20:09:50Z

Recommend SeaweedFS.

← Older revision		Revision as of 20:09, 28 December 2023
Line 1:		Line 1:
	The Ceph Nix package has been hard-pressed to keep up with [https://ceph.io Ceph], as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.		The Ceph Nix package has been hard-pressed to keep up with [https://ceph.io Ceph], as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.

			Also, the year is 2024. You probably should use [https://github.com/seaweedfs/seaweedfs SeaweedFS].

	----		----

imported>C4lliope: Add Ceph link

2023-12-21T23:42:47Z

Add Ceph link

← Older revision		Revision as of 23:42, 21 December 2023
Line 1:		Line 1:
	The Ceph Nix package has been hard-pressed to keep up with Ceph, as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.		The Ceph Nix package has been hard-pressed to keep up with [https://ceph.io Ceph], as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.

	----		----

imported>C4lliope: Make example only single-node.

2023-12-19T21:39:59Z

Make example only single-node.

← Older revision		Revision as of 21:39, 19 December 2023
Line 4:		Line 4:

	Here is a quick collection of commands I used on a 3-node Ceph mesh.		Here is a quick collection of commands I used on a 3-node Ceph mesh.
			The examples have been reduced to a single node, `mesh-a`, for simplicity.

	Describe your ceph user, alongside your normal login user:		Describe your ceph user, alongside your normal login user:
Line 29:		Line 30:
	enable = false;		enable = false;
	extraConfig = {		extraConfig = {
	"mon initial members" = "mesh-a~~,mesh-b,mesh-c~~";		"mon initial members" = "mesh-a";
	"mon host" = "10.0.0.11~~,10.0.0.12,10.0.0.13~~";		"mon host" = "10.0.0.11";
	};		};
	};		};

imported>C4lliope: Clean up.

2023-12-19T21:35:09Z

Clean up.

← Older revision		Revision as of 21:35, 19 December 2023
Line 37:		Line 37:
	</syntaxhighlight>		</syntaxhighlight>

	~~Make your OSD volume;~~ run these commands on each node:		Some preparation is needed so Ceph can run the monitors.
	(based on https://docs.ceph.com/en/quincy/install/manual-deployment/ )		You'll need to run these commands on each node
			(based on https://docs.ceph.com/en/quincy/install/manual-deployment/ ):

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 44:		Line 45:
	export FSID=4b687c5c-5a20-4a77-8774-487989fd0bc7		export FSID=4b687c5c-5a20-4a77-8774-487989fd0bc7

			# Make your paths!
	sudo -u ceph mkdir -p /etc/ceph		sudo -u ceph mkdir -p /etc/ceph
	sudo -u ceph mkdir -p /var/lib/ceph/bootstrap-osd		sudo -u ceph mkdir -p /var/lib/ceph/bootstrap-osd
Line 61:		Line 63:
	</syntaxhighlight>		</syntaxhighlight>

	~~Bind~~ all Ceph OSD shares ~~using systemd~~ (based on <code>u/imspacekitteh</code>'s example):		Prepare <code>systemd</code> to bind all Ceph OSD shares (based on <code>u/imspacekitteh</code>'s example, see links):

	<syntaxhighlight lang="nix">		<syntaxhighlight lang="nix">
Line 89:		Line 91:
	</pre>		</pre>

	Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! ~~I'd assumed the~~ Nixpkgs module should ~~make that~~, based on ~~all the~~ <code>extraConfig</code> options ~~supplied~~.		Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! The Nixpkgs module should be upgraded to handle this someday, based on our supplied <code>extraConfig</code> options.

	~~So make~~ the necessary config; this should be minimally enough to load Ceph~~, and we can come back and nixify this soon~~:		Bypass the error by making the necessary config; this should be minimally enough to load Ceph:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 125:		Line 127:
	Dec 19 16:12:51 mesh-a systemd[1]: ceph-mesh.service: Deactivated successfully.		Dec 19 16:12:51 mesh-a systemd[1]: ceph-mesh.service: Deactivated successfully.
	Dec 19 16:12:51 mesh-a systemd[1]: Finished Ceph OSD Bindings.		Dec 19 16:12:51 mesh-a systemd[1]: Finished Ceph OSD Bindings.


	mesh@mesh-a:~/.build/ > sudo ceph-volume lvm list		mesh@mesh-a:~/.build/ > sudo ceph-volume lvm list

imported>C4lliope: Nearly able to make a volume.

2023-12-19T21:29:22Z

Nearly able to make a volume.

← Older revision		Revision as of 21:29, 19 December 2023
Line 91:		Line 91:
	Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! I'd assumed the Nixpkgs module should make that, based on all the <code>extraConfig</code> options supplied.		Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! I'd assumed the Nixpkgs module should make that, based on all the <code>extraConfig</code> options supplied.

	So ~~I ran in sequence~~:		So make the necessary config; this should be minimally enough to load Ceph, and we can come back and nixify this soon:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	sudo su		sudo su -c "echo '
	~~export FSID="4b687c5c~~-~~5a20-4a77-8774-487989fd0bc7~~"
	echo "
	[global]		[global]
	fsid=$FSID		fsid=$FSID
Line 102:		Line 100:
	mon host = 10.0.0.11		mon host = 10.0.0.11
	cluster network = 10.0.0.0/24		cluster network = 10.0.0.0/24
	" > /etc/ceph/ceph.conf		' > /etc/ceph/ceph.conf" -
	~~exit~~
			# Double-check!
	cat /etc/ceph/ceph.conf		cat /etc/ceph/ceph.conf
	</syntaxhighlight>		</syntaxhighlight>

	~~This should be minimally enough~~ to ~~load Ceph, and we can come back and nixify this soon.~~		We can now see that Ceph is ready for us to make a volume!
	~~The problem remains...~~

	<pre>		<pre>
Line 133:		Line 131:
	</pre>		</pre>

	~~Here is a summary of records produced inside <code>/var/lib/ceph</code>:~~		Go ahead and make one.

	<pre>		<pre>
	mesh@mesh-c:~/.build/ > ~~nix~~-~~shell~~ -~~p tree~~		mesh@mesh-a:~/.build/ > sudo ceph-volume lvm create --data /dev/nvme0n1p4 --no-systemd
			Running command: /nix/store/x645fiz9vzkkwyf08agprl9h25fkqw7g-ceph-18.2.0/bin/ceph-authtool --gen-print-key
	[nix-~~shell~~:~/~~.build]$ sudo~~ -u ceph ~~tree /var~~/~~lib~~/ceph		Running command: /nix/store/x645fiz9vzkkwyf08agprl9h25fkqw7g-ceph-18.2.0/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a0d4c3db-cc9c-416a-b209-72b9bad3be40
	/var/lib/ceph		</pre>
	~~├──~~ bootstrap-osd
	~~│ └──~~ ceph.keyring
	~~└── mon~~
	~~├── ceph~~-~~mesh~~-c
	~~└── ceph~~-~~mon~~-~~mesh~~-c
	~~├── keyring~~
	~~├── kv_backend~~
	~~└── store.db~~
	~~├── 000004.log~~
	~~├── CURRENT~~
	~~├── IDENTITY~~
	~~├── LOCK~~
	~~├── MANIFEST~~-~~000005~~
	~~└── OPTIONS-000007~~

	~~6 directories~~, ~~9 files~~		OOPS! This command hangs endlessly, as does <code>sudo ceph -s</code>.
	</~~pre~~>

			Your help is needed to make more progress here!

	----		----

imported>C4lliope: More digging....

2023-12-19T21:16:09Z

More digging....

@@ Line 50: / Line 50: @@
 sudo -u ceph mkdir /var/lib/ceph/mon/ceph-mon-$(hostname)
 sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
 sudo mkdir -p /var/lib/ceph/bootstrap-osd && sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
 sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
 sudo chown ceph:ceph /tmp/ceph.mon.keyring
 sudo monmaptool --create --add mesh-a $IP --fsid $FSID /tmp/monmap
 sudo -u ceph ceph-mon --mkfs -i mon-$(hostname) --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
@@ Line 86: / Line 89: @@
 </pre>
-nothing has appeared inside <code>lsblk</code>, so as an inexperienced Ceph user I can only assume additional actions are necessary.
+Clearly, Ceph is concerned that the `/etc/ceph/ceph.conf` file is missing. So am I! I'd assumed the Nixpkgs module should make that, based on all the <code>extraConfig</code> options supplied.
 Here is a summary of records produced inside <code>/var/lib/ceph</code>:

imported>C4lliope at 20:55, 19 December 2023

2023-12-19T20:55:42Z

← Older revision		Revision as of 20:55, 19 December 2023
Line 117:		Line 117:

	Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online.		Many users aspire to run Ceph on NixOS, and recommend varying approaches in different forums online.
	Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of ~~now~~) Ceph is on v19.		Here is a collection of links that can lead you along, though please consider; these experiences come from older versions of Ceph, such as v10, while (as of 2023-12) Ceph is on v19.

	* https://d.moonfire.us/blog/2022/12/10/ceph-and-nixos/		* https://d.moonfire.us/blog/2022/12/10/ceph-and-nixos/
	* https://github.com/NixOS/nixpkgs/issues/147801		* https://github.com/NixOS/nixpkgs/issues/147801
	* https://www.reddit.com/r/ceph/comments/14otjyo/ceph_on_nixos/		* https://www.reddit.com/r/ceph/comments/14otjyo/ceph_on_nixos/

imported>C4lliope at 20:54, 19 December 2023

2023-12-19T20:54:47Z

← Older revision		Revision as of 20:54, 19 December 2023
Line 17:		Line 17:
	Be sure you rebuild so you can assign some paths to the <code>ceph</code> user.		Be sure you rebuild so you can assign some paths to the <code>ceph</code> user.

	~~Make a UUID using~~ <code>uuidgen</code> and describe your Ceph nodes:		Run <code>uuidgen</code> and assign the response as your <code>fsid</code>; describe your Ceph nodes:

	<syntaxhighlight lang="nix">		<syntaxhighlight lang="nix">

imported>C4lliope: Rearrange and add shell command.

2023-12-19T20:53:10Z

Rearrange and add shell command.

← Older revision		Revision as of 20:53, 19 December 2023
Line 1:		Line 1:
	The Ceph Nix package has been hard-pressed to keep up with Ceph, as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version. Here is a quick collection of commands I used on a 3-node Ceph mesh.		The Ceph Nix package has been hard-pressed to keep up with Ceph, as filesystem concerns are a larger challenge on NixOS than on other Linux lineages. This is a problem seeking a solution; read ahead to see some of the remaining issues this guide should address. Please make a Wiki account and add your experiences, if you made progress running a modern Ceph version.

			----

			Here is a quick collection of commands I used on a 3-node Ceph mesh.

	Describe your ceph user, alongside your normal login user:		Describe your ceph user, alongside your normal login user: