Proxmox building ceph and ceph module

8 August 2014 6:52 PM Linux, Storage, Virtualization , , , , ,

Introduction

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability – See more at: http://ceph.com

Ceph’s RADOS Block Device to be used for VM disks. The Ceph storage services are usually hosted on external, dedicated storage nodes. Such storage clusters can sum up to several hundreds of nodes, providing petabytes of storage capacity.

For smaller deployments, it is also possible to run Ceph services directly on your Proxmox VE nodes. Recent hardware has plenty of CPU power and RAM, so running storage services and VMs on same node is possible.

This articles describes how to setup and run Ceph storage services directly on Proxmox VE nodes.

Advantages

  • Easy setup and management with CLI and GUI support on Proxmox VE
  • Thin provisioning
  • Snapshots support
  • Self healing
  • No single point of failure
  • Scalable to the exabyte level
  • Setup pools with different performance and redundancy characteristics
  • Data is replicated, making it fault tolerant
  • Runs on economical commodity hardware
  • No need for hardware RAID controllers
  • Easy management
  • Open source

Recommended hardware

You need at least three identical servers for the redundant setup. Here is the specifications of one of our test lab clusters with Proxmox VE and Ceph (three nodes):

  • Dual Xeon E5-2620v2, 32 GB RAM, Intel S2600CP mainboard, Intel RMM, Chenbro 2U chassis with eight 3.5” hot-swap drive bays, 2 fixed 2.5″ SSD bays
  • 10 GBit network for Ceph traffic (one Intel X540-T2 in each server, one 10Gb switch – Netgear XS712T)
  • Single enterprise class SSD for the Proxmox VE installation (because we run Ceph monitors there and quite a lot of logs), we use one Intel DC S3500 80 GB per host.
  • Single, fast and reliable enterprise class SSD for Ceph Journal. Just for this test lab cluster, we used instead some Samsung SSD 840 PRO with 240 GB
  • SATA disk for storing the data (OSDs), use at least 4 disks/OSDs per server, more OSD disks are faster. We use four Seagate Constellation ES.3 SATA 6Gb (4TB model) per server.

N.B. enterprise class SSD means also that they have “power loss data protection”, like intel DC S3500 or DC S3700 have

This setup delivers 48 TB storage. By using the redundancy of 3, you can store up to 16 TB (100%). But to be prepared for failed disks and hosts, you should never fill up your storage with 100 %.

By adding more disks, the storage can be expanded up to 96 TB just by plugging in additional disks into the free drive bays. Of course, you can add more servers too as soon as your business is growing.

If you do not want to run virtual machines and Ceph on the same host, you can just add more Proxmox VE nodes and use these for running the guests and the others just for the storage.

Installation of Proxmox VE

Before you start with Ceph, you need a working Proxmox VE cluster with 3 nodes (or more). We install Proxmox VE on a fast and reliable enterprise class SSD, so we can use all bays for OSD data. Just follow the well known instructions on Installation and Proxmox VE 2.0 Cluster.

Note:

Use ext4 if you install on SSD (at the boot prompt of the installation ISO you can specify parameters, e.g. “linux ext4 swapsize=4”).

Network for Ceph

All nodes need access to a separate 1Gb network interface, exclusively used for Ceph. We use network 192.162.100.0/24 for this tutorial.

It is highly recommended to use 10Gb for that network to avoid performance problems. Bonding can be used to increase availability.

First node

The network setup (ceph private network) from our first node contains:

# nano /etc/network/interfaces
auto eth2
iface eth2 inet static
  address  192.168.100.1
  netmask  255.255.255.0

Second node

The network setup (ceph private network) from our second node contains:

# nano /etc/network/interfaces
auto eth2
iface eth2 inet static
  address  192.168.100.2
  netmask  255.255.255.0

Third node

The network setup (ceph private network) from our third node contains:

# nano /etc/network/interfaces
auto eth2
iface eth2 inet static
  address  192.168.100.3
  netmask  255.255.255.0

Installation of Ceph packages

I will come back to code a min. just memo first

echo -e "deb http://download.proxmox.com/debian wheezy pve-no-subscription" >> /etc/apt/sources.list;

apt-get install xfslibs-dev libboost-filesystem-dev libboost-thread-dev libboost-system-dev uuid-dev libc-dev libc6-dev xfsprogs uuid-dev libkeyutils-dev libgoogle-perftools-dev libatomic-ops-dev libaio-dev libgdata-common libgdata13 libsnappy-dev libleveldb-dev debhelper cdbs libexpat1-dev libatomic-ops-dev fakeroot dpkg-dev g++ libicu-dev subversion cmake libboost-dev build-essential  autotools-dev autoconf automake cdbs gcc g++ git libboost-dev libedit-dev libssl-dev libtool libfcgi libfcgi-dev libfuse-dev linux-kernel-headers libcrypto++-dev libcrypto++ libexpat1-dev pkg-config libcurl4-gnutls-dev libkeyutils-dev libaio-dev libboost-thread-dev libboost-all-dev; 

uname -a;
apt-get install apt-get install pve-headers-(same as version above);

git clone --recursive https://github.com/ceph/ceph.git;
cd ceph;
./autogen.sh;
./configure;
make && make install;

Note: this draft version