This guide was created to walk through the process of setting up a MySQL cluster that uses DRBD for the shared filesystem.

System Requirements

• You will need two unused partitions(one per cluster node) of roughly the same size. Identical size is best otherwise you will only be able to utilize as much space as the smallest partition.
  
• DRBD can be used without redhat cluster services but it is not recommended as DRBD partitions are meant to be used like SAN space and should only be mounted to a single system at a time (exception being Oracle RAC which we do not deploy on DRBD yet.)
  
• A dedicated network connection exclusively for DRBD traffic is recommended so synchronization does not have to compete with any other traffic.
  
  • port 7788 - 7799 opened on this dedicated network connection (shouldn't be an issues since they should be on the same subnet.)

Network Requirements

• Before you proceed, you need to make sure that Multicast networking is enabled on the VLAN you are configuring your cluster on. It's not configured by default in most DC's. Submit a request to networking before proceeding.
  
• You must enable firewall rules to your Ilo/Drac device.
  
  • Drac 6 = telnet port (23)
    
  • HP Ilo / Ilo2 = https (443)
    
  • HP Ilo MP = SSH (22) + Telnet (23)
    
  • HP Ilo3 = ipmi (623 UDP/TCP)
    
  • Drac 5 = SSH (22) + Telnet (23)
    See Here https://access.redhat.com/kb/docs/DOC-30004

RedHat Cluster Services Installation

1) update all packages on the server

• Redhat 4 - up2date -u
  
• Redhat 5 - yum update

2) Subscribe the server to the Redhat Cluster Services channel

• Login to the RedHat Network (username and password can be obtained from another SE or your manager)
  
  • https://rhn.redhat.com
    • When you're logged in find the server you want to install redhat clustering on.
      
    • Click on "Alter Channel Subscriptions" when you have the page for the server open.
      
    • Under "Additional Services Channels for Red Hat" check the box that says "RHEL Clustering"
      
    • click the "Change Subscriptions" button at the bottom.

3) Install the Redhat Clustering Services

• RHEL4: up2date --installall=rhel-x86_64-es-4-cluster (for 64bit)
  
  • If the above errors on kernel packages, it's because the kernel updates are set to be skipped.
    
    • sudo vi /etc/sysconfig/rhn/up2date
      
    • change pkgSkipList=kernel*; to pkgSkipList=
    • run up2date command again

• RHEL5: SSH into the server and install the RedHat Cluster Suite.
  
  • sudo yum groupinstall Clustering

4) Disable kernel updates

• RHEL4: vi /etc/sysconfig/rhn/up2date

• pkgSkipList=kernel*;

• RHEL5: vi /etc/yum.conf

• exclude=kernel*

MySQL Installation - Confirm with your friendly dba you are ok to proceed with this step, and these directions are still valid.

Directions are provided below and can also be found at this link:

https://wiki.opsource.net/display/OPSDBA/Procedure+-+MySQL+standalone+install

The packages necessary for mysql installation can be found at:

http://www.mysql.com/downloads/mysql/

Package List:
MySQL-client-community
MySQL-devel-community
MySQL-server-community
MySQL-shared-compat

After uploading the packages to your servers and selecting the appropriate my.cnf file from the kontrollkit package attached to the MySQL wiki page referenced above, perform the following:

• cp my.cnf /etc/
  
• rpm -ivh *.rpm -
  Preparing... ########################################### 100%
  1:MySQL-shared-compat ########################################### [OPSDBA: 25%]
  2:MySQL-client-community ########################################### [OPSDBA: 50%]
  3:MySQL-devel-community ########################################### [OPSDBA: 75%]
  4:MySQL-server-community ########################################### 100%

Starting MySQL.[OPSDBA: OK ]
Giving mysqld 2 seconds to start

• chkconfig mysql off
  
• /etc/init.d/mysql stop
  Shutting down MySQL..... [OPSDBA: OK ]

• chown mysql:mysql /home/mysql-backups
  
• mv /var/lib/mysql/* /data/mysql-server/
  
• rmdir /var/lib/mysql
  
• mkdir -p /data/mysql-server
  
• mkdir /data/mysql-backups
  
• chown -R mysql:mysql /data/mysql-backups
  
• ln -s /data/mysql-backups /home/mysql-backups
  
• chown mysql:mysql /home/mysql-backups
  
• chown -R mysql:mysql /data/mysql-server
  
• ln -s /data/mysql-server /var/lib/mysql
  
• chown mysql:mysql /var/lib/mysql
  
• /etc/init.d/mysql start
  Starting MySQL. [OPSDBA: OK ]

Confirm that mysql starts on it's own as expected...

• /etc/init.d/mysql stop

DRBD Installation

Download DRBD

• Until recently RHEL compatible DRBD rpm packages were not available. Although Redhat still does not provide DRBD packages. CentOS, a 100% binary compatible OS to Redhat now provides them on their repositories. Removing the previous need to compile your own RPM's, or install from source.
  
  • Download the release compatible to your kernel from http://mirror.centos.org/centos/5/extras/x86_64/RPMS/ - check your kernel type and bit(32/64) before downloading Consult http://www.drbd.org/download/mainline/ to determine what version of DRBD is best suited to your kernel version. In this scenario used for this installation and howto, kernel version and related DRBD package versions are provided below:

• uname -a
  Linux lhradobcndb02p.ood.ops 2.6.18-194.26.1.el5 #1

• rpm -qa | grep -i drbd

drbd83-8.3.8-1.el5.centos
kmod-drbd83-8.3.8-1.el5.centos

• rpm -ivh *.centos.*.rpm

Confirm the drbd kernel module has been loaded.

• lsmod | grep drbd

You may need to reboot to ensure the module is loaded.

Configuring the Filesystem for DRBD

http://www.drbd.org/docs/install/

• Select partitions that are the same size on each server to be your DRBD partition.

• fdisk /dev/cciss/c0d1 - this will vary depending on your server hardware
  
• partprobe - rescan the bus w/o reboot
  
• ls /dev/cciss/c0d1* - confirm it exists
  c0d1 c0d1p1
  
• pvcreate /dev/cciss/c0d1p1
  Physical volume "/dev/cciss/c0d1p1" successfully created

• vgcreate vg0 /dev/cciss/c0d1p1
  Volume group "vg0" successfully created
  
• vgdisplay vg0 | grep "Total PE" - This is used to determine total Physical Extents. So, the entire disk could be used for the Logical Volume
  Total PE 104986
  
• lvcreate -l 104986 vg0 -n lvol0
  Logical volume "lvol0" created

• • It's important that these are new, unformatted partitions as DRBD will give an error when trying to initialize the disks for DRBD if the partition is formatted.
• write the drbd configuration file.
  
  • By default there is a /etc/drbd.conf created on install.
    

    include "drbd.d/global_common.conf";

    include "drbd.d/*.res";

  • Any additional config files should be placed in /etc/drbd.d/ and have an extension of .res to be included in the configurations.
    
  • The only changes you should make in include drbd.d/global_common.conf is:
    

    syncer {

    rate 110M;

    # rate after al-extents use-rle cpu-mask verify-alg csums-alg

    }

  • This limits the speed of the syncer to 110M which is the maximum for a dedicate 1GB connection.
    
  • here is a sample config for a resource. These files need to be identical on each peer. If you have more than one resource they will need to have different ports specified in their resource configuration files.
    

    resource mysqldata0 {

    on lhradobcndb01p.ood.ops {

    device    /dev/drbd1;

    disk      /dev/mapper/vg0-lvol0;

    address   10.120.111.21:7789;

    meta-disk internal;

    }

    on lhradobcndb02p.ood.ops {

    device    /dev/drbd1;

    disk      /dev/mapper/vg0-lvol0;

    address   10.120.111.22:7789;

    meta-disk internal;

    }

    }

  • This configuration calls the resource "mysqldata0" and the drbd device is "/dev/drbd1" This is the device that is mounted. disk is the partition to be used for DRBD, address is that dedicated network IP(if applicable) and port, and meta-disk internal means that the DRBD meta-data will be on /dev/cciss/c0d1p1. You can have your meta data located elsewhere but internal is recommended. If you're wanting to make an existing disk a DRBD disk you can keep the data intact by putting the meta-data elsewhere but that is beyond the scope of this document.
    
  • Once your resource configuration is in place on each server you can begin initializing DRBD. This can be done with one command which runs the four commands below and must be completed on each server independently. ***NOTE the first command has been found to be inconsistent from server to server. So, the recommended approach is to run each command individually.
    

    In this case, resource = mysqldata0

    drbdadm up resource

    • • first meta-data needs to be created
        

        drbdadm create-md resource

      • second the DRBD resource needs to be attached
        

        drbdadm attach resource

      • third setup synchronization
        

        drbdadm syncer resource

      • fourth connect to the other peer
        

        drbdadm connect resource

    • You can check the status of DRBD to see if everything was successful
      

      cat /proc/drbd

      version: 8.3.0 (api:88/proto:86-89)

      GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by buildsystem@linbit, 2008-12-18 16:02:26

      1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---

      ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:200768

      The Inconsistent/Inconsistent disk state is expected at this point.
      

    • you can now begin disk synchronization. You will need to decide which disk will be the initial source. If you're trying to preserve data, it's important that you select the server that has the data to preserve. On the server that is to be the syncer source run the following command:
      

      drbdadm -- --overwrite-data-of-peer primary resource

    • You can monitor the syncer progress by using cat /proc/drbd. While syncing you can format, mount, and begin working with the resource on the primary node though you will have reduced performance until syncing is completed.

• mkfs.ext3 /dev/drbd1

Once synchronization is complete, testing needs to be performed.

• Node1 - mount /dev/drbd1 /data
  
• Node1 - for i in $(seq 1 5) ; do dd if=/dev/zero of=/data/file$i bs=1M count=100;done
  
• Node1 - umount /data ; drbdadm secondary mysqldata0
  
• Node2 - drbdadm primary mysqldata0 ; mount /dev/drbd1 /data
  
• Node2 - ls /data/ - should output 'file1 file2 file3 file4 file5' - If so, great data was replicated. Next step:
  
• Node2 - rm /data/file2 ; dd if=/dev/zero of=/data/file6 bs=100M count=2
  
• Node2 - umount /data ; drbdadm secondary mysqldata0
  
• Node1 - drbdadm primary mysqldata0 ; mount /dev/drbd1 /data
  
• Node1 - ls /data - should output - 'file1 file3 file4 file5 file6' - If so, DRBD is working
  
• chkconfig drbd on

Configure RedHat Cluster Services

RedHat cluster services are setup in /etc/cluster/cluster.conf
Here is a sample cluster.conf configured for DRBD and MySQL failover.

Click here to expand...

We'll break this down by section:

<?xml version="1.0"?>

<cluster alias="sjcopscl01p" config_version="1" name="sjcopscl01p">

<fence_daemon post_fail_delay="5" post_join_delay="3"/>

This is going to be fairly standard for all deploiyments. You'll only want to change the alias and the name. Whatever you change it to will need to be consistent throughout the document. For every change you make you will need to increment the config_version. This makes it easier to update the configuration to every cluster node from the node you're editing. Note, if you have 2 different clusters on the same network, make sure you DON'T copy/paste the config line for line. If they are 2 unique clusters, ensure you name the cluster with 2 different "name=" stanzas (names).

<clusternodes>

<clusternode name="sjcopsdb04p" nodeid="1" votes="1">

<fence>

<method name="1">

<device name="sjcopsdb04p-ilo"/>

</method>

</fence>

</clusternode>

<clusternode name="sjcopsdb05p" nodeid="2" votes="1">

<fence>

<method name="1">

<device name="sjcopsdb05p-ilo"/>

</method>

</fence>

</clusternode>

</clusternodes>

This is the cluster node section. This is where you define the cluster nodes and how the will be fenced. The node-id for each server will need to be unique. Note the device name in each fence section and it will need be the same later in the document. You can name the node anything you want, but the server name is usually best. If you start to test the 'fencedevice_agent' (ex. /sbin/fence_drac5 -a "10.128.47.226" -c "admin1->" -l "<username>" -p "<password>") and you find that telnet breaks, and SSH works (seems to be needed with rhel4 and the Dell idrac6 logins), you then need to add an additional item to the 'clusternodes' section. Under the ilo/drac clusternode definition, you must define 'secure="1"'. This tells the software to use SSH, instead of telnet.  See example below.

<clusternodes>

<clusternode name="sjctricdb08p.ood.ops" nodeid="1" votes="1">

<fence>

<method name="1">

<device name="sjctricdb08p-ilo" secure="1"/>

</method>

<cman expected_votes="1" two_node="1"/>

<fencedevices>

<fencedevice agent="fence_ilo" ipaddr="10.128.32.87" login="ilofence" name="sjcopsdb04p-ilo" passwd="opsapass"/>

<fencedevice agent="fence_ilo" ipaddr="10.128.32.142" login="ilofence" name="sjcopsdb05p-ilo" passwd="opsapass"/>

</fencedevices>

The cman line shouldn't need to be change at all.
The fencedevices section defines how cluster services can fence the device. This config is unique to servers that have ilo. For a dell DRAC config you will need to change the "agent." the ipaddr is the ilo ip address. for login and passwd, you will need to create the login and password in ilo so cluster services can access ilo if needed. note the fencedevice name as you will need to use it later in the document. It's most intuitive to use servername-ilo or servername-drac depending on the setup.

NOTE: The 'fencedevice_agent' needs to be set appropriately to the ilo/Drac (remote login/reboot) device/method. The 2 common ones we use are "ilo" for the HPs, and "drac5" seems to work for the newer Dell rX10 series (Drac6 OS). Again, a good source of information on this is https://access.redhat.com/kb/docs/DOC-30004. In there, you will see that if you are using DRAC6 OS, you should be using "ipmilan" as your fencedevice_agent. Two attempts at getting that to work haven't been successful, but enabling telnet and using the "drac5" agent seems to work.

<failoverdomains>

<failoverdomain name="mysql-cluster" ordered="1" restricted="0">

<failoverdomainnode name="sjcopsdb04p" priority="1"/>

<failoverdomainnode name="sjcopsdb05p" priority="2"/>

</failoverdomain>

</failoverdomains>

This is where you can define the clustered service. In this case it's a mysql-cluster. You can name it whatever you want, the more descriptive the better. The failoverdomainnode names should match the clusternode names defined earlier. If you have multiple clustered services you will have multiple failoverdomain entries.

<resources>

<ip address="10.128.1.35" monitor_link="1"/>

<fs device="/dev/drbd/by-res/mysqldata0" force_fsck="0" force_unmount="0" fsid="12345" fstype="ext3" mountpoint="/data" name="mysqldata0" options="noatime"/>

<mysql config_file="/etc/my.cnf" listen_address="10.128.1.35" name="mysql"/>

</resources>

This defines the resources available. In this case, there is an ip address, a filesystem and mysql. MySQL can actually be called by the init.d script or called directly. In this case it is called directly and is tied to the ip address that is part of the cluster.

<service autostart="1" name="mysql-cluster">

<ip ref="10.128.1.35"/>

<drbd name="drbd-mysql" resource="mysqldata0">

<fs ref="mysqldata0"/>

<mysql ref="mysql"/>

</drbd>

</service>

This is where the services are called for the mysql-cluster. The order matters. If these were put in the wrong order mysql could fail to start as the ip address would not be assigned. This specific config uses drbd and the config used here is taken from the drbd recommendation on their site.