How to install Ceph cluster (version Firefly 0.80.x)

Welcome to the Ceph wiki for the INFN Bari Cloud School!

With this procedure you will deploy a Ceph (version Firefly 0.80.x) cluster on Ubuntu 12.04 LTS made by one node running a metadata server, a monitor and acts also as admin node and 3 OSD nodes. The admin node will be "node01" meanwhile the 3 OSDs will be "node02", "node03" and "node04" (NOTE: you need a proper kernel version and the node clocks have to be synchronized, follow the Ceph official documentation for more details).

The following image shows the cluster architecture that you are going to deploy for the testing environment:

On every cluster node create a "ceph" user and set to it a new password:

sudo useradd -d /home/ceph -m ceph

sudo passwd ceph

To provide full privileges to the user, on every cluster node add the following to /etc/sudoers.d/ceph:

echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph

And change permissions in this way:

sudo chmod 0440 /etc/sudoers.d/ceph

Configure your admin node with password-less SSH access to each node running Ceph daemons (leave the passphrase empty). On your admin node node01, become ceph user and generate the ssh key:

# su - ceph
$ /bin/bash
$ ssh-keygen -t dsa

You will have output like this:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph/.ssh/id_rsa):
Created directory '/home/ceph/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ceph/.ssh/id_rsa.
Your public key has been saved in /home/ceph/.ssh/id_rsa.pub.

Copy the key to each cluster node and test the password-less access:

ssh-copy-id ceph@node02

ssh-copy-id ceph@node03

ssh-copy-id ceph@node04

ssh ceph@node02

ssh ceph@node03

ssh ceph@node04

on the admin node edit the .ssh/config file for the ceph user, the config file should be like this:

Host node01 node02 node03 node04 
Hostname %h
User ceph

On the admin node add repository key for Ceph repository (we are assuming that you are running a ubuntu/debian based distribution):

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -

Add the repository to the sources:

echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

Intall the ceph-deploy tool:

sudo apt-get update && sudo apt-get install ceph-deploy

On the admin node make a directory (a kind of working directory) into ceph user's home directory, this directory will be useful because will contain all the files generated by ceph-deploy tool:

Using the ceph user, run the following commands:

mkdir ceph-cluster

cd ceph-cluster/

From the admin node, into the working directory, run this command to install ceph on all the cluster nodes:

ceph-deploy install node01 node02 node03 noed04

Check the installed version on every node:

ceph -v

From the admin node, into the working directory, run this command in order to indicate to ceph-deploy which nodes will be the initial monitors:

ceph-deploy new node01

Always from the working directory run the following command to create the initial monitors:

ceph-deploy mon create-initial

If you get a warning message like this: "[ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['node01']" run the following command:

ceph-deploy gatherkeys node01

From the working directory run this command to let node01 to be the admin node (you can add more admin nodes)

ceph-deploy admin node01

On each admin node run:

sudo chmod +r /etc/ceph/ceph.client.admin.keyring

On the admin node, into the working directory, for every future OSD node list the available disks:

ceph-deploy disk list node02

ceph-deploy disk list node03

ceph-deploy disk list node04

Then prepare the OSD devices (NOTE: YOU WILL LOSE DATA STORED ON THESE DISKS). This command will make on every disk a partition for the journal and a partition for data.

ceph-deploy disk zap node02:sdb

ceph-deploy disk zap node03:sdb

ceph-deploy disk zap node04:sdb

Then gather useful keys to deploy the OSDs, on the admin node run (if you did not do it previously):

ceph-deploy gatherkeys node01

Create the OSDs, from the admin node run:

ceph-deploy osd create node02:sdb

ceph-deploy osd create node03:sdb

ceph-deploy osd create node04:sdb

From the admin node check the cluster status and the OSD tree map:

ceph status

ceph osd tree

Check the size replica with:

ceph osd dump | grep replica

If the size is not set on 3 and min_size is not set on value 2 do the following commands to set replica 3 for each pool and min_size (minimum amount of active PG to do r/w operations):

ceph osd pool set data size 3

ceph osd pool set metadata size 3

ceph osd pool set rbd size 3

ceph osd pool set data min_size 2

ceph osd pool set metadata min_size 2

ceph osd pool set rbd min_size 2

At this point check the cluster status with the following command:

ceph status

If you have a warning like this: "health HEALTH_WARN 192 pgs stuck unclean" you have to restart the Ceph OSD daemons, so on every OSD node run:

sudo restart ceph-osd-all

Wait a bit and then check if the status is "active+clean" with the usual ceph status command.

Create the MDS, from the admin node run:

ceph-deploy mds create node01

Now check if the MDS has been deployed correctly:

ceph mds dump

Ceph prepares some default storage pools named "rbd", "data" and "metadata". You have to indicate the proper number of placement group to use for every pool. To compute a good amount of placement group for one pool use the following formula (NOTE: #replica is the desired number of replicas for one pool):

(#OSD daemons * 100) / #replica

Round the result to the next power of 2: in our case will be (3*100)/3 = 100, rounded to 128. So run the following commands from the admin node to set the right amount of placement groups for the default pools (NOTE: you also have to indicate for each pool how many available placement groups to use for storing purposes setting the pgp_num variable). Usually the creation of new PGs needs time, therefore wait a bit if you receive some error messages:

ceph osd pool set data pg_num 128

ceph osd pool set data pgp_num 128

ceph osd pool set metadata pg_num 128

ceph osd pool set metadata pgp_num 128

ceph osd pool set rbd pg_num 128

ceph osd pool set rbd pgp_num 128

Finally check the cluster status:

ceph status

If it's "active+clean" you can start using your new ceph cluster, otherwise is preferable to debug and get the active+clean status.

To try to store one file into "data" pool use a command like this (in this case in our admin node):

rados put {object-name} {file-path} --pool=data

Do this command to check if the file has been stored into the pool "data":

rados ls -p data

To mount ceph FS with FUSE, first install ceph-fuse with the command

apt-get install ceph-fuse

and then mount it, running:

ceph-fuse -m {monitor_hostname:6789} {mount_point_path}

Note that the mount_point_path must exist before you can mount the ceph filesystem. In our case the mountpoint is the directory /ceph-fs that we create with

sudo mkdir /ceph-fs

Note: configure the mount in /etc/fstab:

#ceph-fuse
id=admin  /ceph-fs  fuse.ceph defaults 0 0