Online data relocation in a clustered environment under RedHat 4

In this article I will talk about online data relocation in a clustered environment.
Under clustered environment I mean Red Hat Cluster Suite running on RHEL 4.9.
This article was inspired by one of our clients requesting to migrate data from one of the physical volumes to another physical volume without unmounting file system. At first look I thought this was going to be easy, but clustered environment plus really outdated version of RedHat made this task a bit more complex than I originaly thought.

Production system that I needed to migrate is running
kernel 2.6.9-101.ELsmp
lvm2 with lvm2-cluster
All of the file systems reside on top of lvm2
Some of them are ext3 and some are gfs
Physical volumes look like this

# pvs
  PV                VG       Fmt  Attr PSize   PFree
  /dev/mapper/lun00 vgarc    lvm2 a-     1.05T      0
  /dev/mapper/lun01 vgredo   lvm2 a-   198.00G      0
  /dev/mapper/lun02 vgglobal lvm2 a-   850.00G 351.00G
  /dev/mapper/lun03 vgdb     lvm2 a-   990.00G      0
  /dev/mapper/lun05 vgbackup lvm2 a-     5.01T      0
  /dev/mapper/lun30 vgglobal lvm2 a-   150.00G      0

Notice how lun02 and lun30 are both in the vgglobal volume group?
So this is what the request was for.
Client wanted to move physical extents from lun30 to lun02 and then destroy lun30.

In theory this task can be as simple as

pvmove -i 5 /dev/mapper/lun30 /dev/mapper/lun02
vgreduce vgglobal /dev/mapper/lun30

But according to RedHat 4 Cluster Logical Volume Manager documentation online migration is unsupported in a cluster

Because the pvmove command uses mirroring, it is not cluster-aware and needs exclusive access to a volume. For information on activating logical volumes on individual nodes in a cluster, see Section 4.8, “Activating Logical Volumes on Individual Nodes in a Cluster”.

However RedHat 5 Logical Volume Manager Administration
has a note that online migration is possible

In order to perform a pvmove operation in a cluster, you should ensure that the cmirror and cmirror-kmod packages are installed and that the cmirror service is running. The cmirror-kmod package that must be installed depends on the kernel that is running. For example, if the running kernel is kernel-largesmp, it is necessary to have cmirror-kmod-largesmp for the corresponding kernel version.

My next move was to head to CentOS Vault and check if there is anything similar to cmirror and cmirror-kmod in the 4.x branch. And there was something promising, 2 cmirror packages and a series of cmirror-kernel-* packages for different kernel flavors (regular kernel, smp, largesmp, XenU). The latest cmirror package http://vault.centos.org/4.9/csgfs/x86_64/RPMS/cmirror-1.0.2-1.el4_8.x86_64.rpm is kernel independent and all it provides is the init.d script to load and unload module

# rpm -qpl cmirror-1.0.2-1.el4_8.x86_64.rpm
/etc/rc.d/init.d/cmirror

But cmirror-kernel* packages containing the actual dm-cmirror.ko module are kernel dependent so I had to find one that was built for my kernel – 2.6.9-101.ELsmp

yum provides "/lib/modules/2.6.9-101.ELsmp/kernel/cluster/dm-cmirror.ko"

yelds nothing. May be there is one on the RedHat portal, but in the CentOS vault there is none matching my particular kernel.
Since I didn’t feel like chasing people within my company who have RedHat portal access I decided to explore possibility to build this module from sources.
Latest source rpm for this module can be found here http://vault.centos.org/4.9/csgfs/SRPMS/cmirror-kernel-2.6.9-43.19.el4.src.rpm
So let’s install it

 rpm -ivh http://vault.centos.org/4.9/csgfs/SRPMS/cmirror-kernel-2.6.9-43.19.el4.src.rpm

And before building it let’s modify specs file a bit, or else it will ask you to have many kernel dependencies for regule, largesmp and xen versions.
So in the file /usr/src/redhat/SPECS/cmirror-kernel.spec let’s modify these lines

%define buildlargesmp 1
%define buildxen 1
%define kernel_version 2.6.9-103.EL

to these

%define buildlargesmp 0
%define buildxen 0
%define kernel_version 2.6.9-101.EL

Ok, now it’s time to build rpm

rpmbuild -bb /usr/src/redhat/SPECS/cmirror-kernel.spec

After build is finished let’s install it

rpm -ivh /usr/src/redhat/RPMS/x86_64/cmirror-kernel-smp-2.6.9-43.19.x86_64.rpm

And now it’s time to load module and test pvmove.

Since it wasn’t wise to do all above steps and testing on the actual production system I ended up creating Red Hat Cluster with 2 nodes under virtual box (well, actually I created a CentOS cluster and matched kernel and package versions to production system). Presented 2 LUNs via iSCSI from the main host and these became my 2 PVs.
Below are results of the simulation

[root@node-1 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  node-1                                   Online, Local, rgmanager
  node-2                                   Online, rgmanager

[root@node-1 ~]# uname -a
Linux node-1.lab 2.6.9-101.ELsmp #1 SMP Thu Jul 21 17:46:19 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@node-1 ~]# iscsi-ls -l
*******************************************************************************
SFNet iSCSI Driver Version ...4:0.1.11-7(14-Apr-2008)
*******************************************************************************
TARGET NAME             : iqn.2013-02.com.example.ubuntu:lun1
TARGET ALIAS            :
HOST ID                 : 2
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 192.168.0.118:3260,1
SESSION STATUS          : ESTABLISHED AT Sat Feb 16 09:32:12 CST 2013
SESSION ID              : ISID 00023d000001 TSIH 300

DEVICE DETAILS:
---------------
LUN ID : 0
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI SCSI revision: 05

LUN ID : 1
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 05
  page83 type1: IET 00010001
2:0:0:1: expected length 24, got length 20
  page80: 0a
  Device: /dev/sdb
LUN ID : 2
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 05
  page83 type1: IET 00010002
2:0:0:2: expected length 24, got length 20
  page80: 0a
  Device: /dev/sdc
*******************************************************************************

[root@node-1 ~]# pvs
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a-     7.88G      0
  /dev/sdb   vgglobal   lvm2 a-   808.00M 484.00M             ### ISCSI LUN 1
  /dev/sdc   vgglobal   lvm2 a-   140.00M  12.00M             ### ISCSI LUN 2

[root@node-1 ~]# lvs
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  LogVol00 VolGroup00 -wi-ao   6.91G
  LogVol01 VolGroup00 -wi-ao 992.00M
  lvrepnb  vgglobal   -wi-ao 128.00M                           ### We will be moving physical extents of this logical volume
  mygfs    vgglobal   -wi-ao 324.00M

[root@node-1 ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/mapper/vgglobal-mygfs on /mygfs type gfs (rw)               ### This is gfs file system mounted on both nodes simultaneously
/dev/mapper/vgglobal-lvrepnb on /mypackagefs type ext3 (rw)      ### This is ext3 file system that we will attempt to move

Now let’s try to attempt pvmove without cmirror-kernel module.

[root@node-1 ~]# pvmove -i 5 -n vgglobal/lvrepnb /dev/sdc /dev/sdb
FATAL: Could not open '/lib/modules/2.6.9-101.ELsmp/kernel/cluster/dm-cmirror.ko': No such file or directory
  /sbin/modprobe failed: 1
  Error locking on node node-1: Volume is busy on another node
  Failed to activate lvrepnb

Now let’s install cmirror-kernel package that we built from source before and cmirror package.
Make sure to install on all nodes in a cluster!

rpm -ivh http://vault.centos.org/4.9/csgfs/x86_64/RPMS/cmirror-1.0.2-1.el4_8.x86_64.rpm
rpm -ivh /usr/src/redhat/RPMS/x86_64/cmirror-kernel-smp-2.6.9-43.19.x86_64.rpm
service cmirror start

And let’s attempt pvmove once again

[root@node-1 x86_64]# pvmove -i 5 -n vgglobal/lvrepnb /dev/sdc /dev/sdb
  /dev/sdc: Moved: 12.5%
  /dev/sdc: Moved: 25.0%
  /dev/sdc: Moved: 34.4%
  /dev/sdc: Moved: 46.9%
  /dev/sdc: Moved: 56.2%
  /dev/sdc: Moved: 68.8%
  /dev/sdc: Moved: 78.1%
  /dev/sdc: Moved: 90.6%
  /dev/sdc: Moved: 100.0%
[root@node-1 x86_64]# pvs
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a-     7.88G      0
  /dev/sdb   vgglobal   lvm2 a-   808.00M 356.00M
  /dev/sdc   vgglobal   lvm2 a-   140.00M 140.00M

During the pvmove I unmounted vgglobal/lvrepnb file system that was being moved and mounted it on node-2 and then mounted it back on node-1. Consistency of the files was not compromised after move.

This entry was posted in RedHat and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *