Messing Around With Ceph (2/N)

Alright... all of our VMs have been prepped for Ceph per the preflight checklist. Let's set up a filesystem! For what follows I'll (mostly) be using the Storage Cluster Quick Start as a guide. Commands are executed on the admin node as the ceph-deploy user unless otherwise noted.

Gather all the host keys ahead of time to avoid the "do you want to add this key" prompts:

ssh-keyscan admin node1 node2 node3 client > /home/ceph-deploy/.ssh/known_hosts

Install the ceph-deploy utility (and dependency python-setuptools):

sudo yum install -y ceph-deploy python-setuptools

Write out an initial configuration to the monitor nodes:

ceph-deploy new node1 node2 node3

At this point nothing is running, and nothing much has been installed. Next step is to install all the Ceph packages everywhere:

ceph-deploy install admin node1 node2 node3 client

The first time I did this, after following the preflight checklist, I got the following error:

[ceph_deploy][ERROR ] RuntimeError: NoSectionError: No section: 'ceph'

Apparently there's a clash between what the documentation says and what the deployment automation expects; both of them want to put repo config in the file ceph.repo. If you hit the error above you can fix it via

for host in node1 node2 node3 admin client; do
  vagrant ssh $host -- sudo mv /etc/yum.repos.d/ceph.repo /etc/yum.repos.d/ceph-deploy.repo;
done

At this point the Ceph binaries are now installed everywhere, which means that we can start bringing up services. The monitor daemons come online first, which makes sense when you recall that their job is to house a consistent, distributed view of all the cluster state. Here we go:

ceph-deploy mon create-initial

If you watch the spew to console when you run this command you'll notice that the script brings up the monitor daemons on all the nodes originally specified via ceph-deploy new. Once the daemons are running the script then goes back and waits until the daemons are reporting a quorum. Finally, it spits out a bunch of keying/auth material into the current directory. Install the keying material (and some config files) on the admin and cluster nodes:

ceph-deploy admin admin node1 node2 node3

With the config and keys installed you can now check the status of the cluster (which must be done as root so you can read the keys):

[ceph-deploy@admin ~]$ sudo ceph status
  cluster:
    id:     8a7aac93-aa5a-4328-8684-8392a2e9ce8f
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node1,node2,node3
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

This shows that the cluster is up, that the monitors have a quorum, and that nothing else has been installed so far. All of which is exactly as we'd expect.

Next up, deploy the manager daemons:

ceph-deploy mgr create node1 node2 node3

What does the cluster look like now?

[ceph-deploy@admin ~]$ sudo ceph status
  cluster:
    id:     8a7aac93-aa5a-4328-8684-8392a2e9ce8f
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node1,node2,node3
    mgr: node1(active), standbys: node2, node3
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

The manager daemons are all up and running, with 1 node active and two acting as standbys. The manager daemons don't have an active/active mode, so this is as expected as well.

The last thing we'll tackle in this post is bringing up the OSDs, which are the things that actually store the data, online. Here you go:

for host in node1 node2 node3; do
  ceph-deploy osd create --data /dev/sdb $host;
done

This brings up an OSD on each cluster node, configured to use its second disk (/dev/sdb) for storage. Here's what our cluster looks like now:

[ceph-deploy@admin ~]$ sudo ceph status
  cluster:
    id:     8a7aac93-aa5a-4328-8684-8392a2e9ce8f
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node1,node2,node3
    mgr: node1(active), standbys: node2, node3
    osd: 3 osds: 3 up, 3 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   41 MiB used, 2.9 GiB / 3.0 GiB avail
    pgs:

All the OSDs are reporting as up, again as expected.

And we'll leave it at that for the time being. Let's recap what we've done so far. We:

1. Generated an initial config for the monitor nodes.
2. Installed the Ceph binaries everywhere.
3. Brough up the initial set of mon daemons.
4. Distributed config and keys.
5. Brought up mgr daemons.
6. Brought up OSDs, configured to use the second drive in each VM for storage.

At this point the cluster is a fully functional object store, but isn't yet ready for use as a DFS. Stay tuned for our next exciting installment where we'll get the FS stuff up and running.