CentOS / Fedora Linux Live CD over cobbler PXE boot

This post will cover configuration of cobbler 2.6.0 running on Cent OS 6.8 for booting Live Fedora Core CD via PXE.

Make sure to adjust SELinux variables to match below settings

[root@nfsserver images]# getsebool -a | grep -i cobbler
cobbler_anon_write --> on
cobbler_can_network_connect --> on
cobbler_use_cifs --> off
cobbler_use_nfs --> off
httpd_can_network_connect_cobbler --> on
httpd_serve_cobbler_files --> on

setsebool -P httpd_can_network_connect on

Mount Live ISO file and copy initrd.img and vmlinuz files from Live ISO to /var/lib/tftpboot/images/Live/

mkdir -p /var/lib/tftpboot/images/Live/
mount -o loop /root/Fedora-Workstation-Live-x86_64-25-1.3.iso /mnt
cp /mnt/isolinux/vmlinuz /mnt/isolinux/initrd.img /var/lib/tftpboot/images/Live/

Define new menu item for Live distribution under /var/lib/tftpboot/pxelinux.cfg/default
Refer to these links for details:

        kernel /images/Live/vmlinuz
        MENU LABEL Live
        append initrd=/images/Live/initrd.img root=live: rd.live.image
        ipappend 2

In the above example we are downloading complete iso file over network from cobbler server. So we need to place iso file under /var/www/html/ and set correct SELinux security context

cp Fedora-Workstation-Live-x86_64-25-1.3.iso /var/www/html/
chcon --reference=index.html Fedora-Workstation-Live-x86_64-25-1.3.iso 

Note that above doesn’t work with CentOS6.x initrd, since initrd that comes with the CentOS 6 live iso doesn’t include module livenet.
For that reason if above method is attempted with CentOS6.x Live ISO – you will get an error during boot

dracut: FATAL: Don't know how to handle 'root=live:http:

Refer to below bug


Posted in Linux, RedHat | Tagged | Leave a comment

Connect to oracle database from php with oci

If you need to query oracle database from php – you will need to setup oci8 driver.
In this post I will talk about what it takes to set it up on CentOS 6.4 server running apache 2.2 with enabled SELinux.

Section 1. Install Oracle Instant Client.

Head to http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html and chose your Linux distribution type (64 bit or x86)

Since I’m on 64 bit after I click on the link for x64 I’m presented with multiple choices here


I will be downloading and installing following packages:

  • oracle-instantclient12.1-basic-
  • oracle-instantclient12.1-devel-
  • oracle-instantclient12.1-sqlplus-

After installation I will need to setup several environmental variables.

I add following lines at the end of my /etc/bashrc

export ORACLE_HOME=/usr/lib/oracle/12.1/client64
export LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib

I will also need to create directory for $TNS_ADMIN

mkdir /usr/lib/oracle/12.1/client64/admin

And will need to place tnsnames.ora under this directory which lists my database connections.
Actually this step is optional if you will be using so called Easy Connect string
So below is my tnsnames.ora file


mydb =
      (ADDRESS = (PROTOCOL = TCP)(HOST = = 1521))
      (SERVICE_NAME = mydb)

Section 2. Install and configure oci8 driver

Logout and login as root to propagate new environment variables that we created in the previous step.
You should be able to see them now

env|grep -i oracle

Now let’s install some php packages

yum install php-pear php-devel
pear download pecl/oci8

At this point you should have in your directory file oci8-2.0.6.tgz
where 2.0.6 is a version number and it can be different from what I have now at the time of writing this.
Let’s uncompress it and install

tar -xzvf oci8-2.0.6.tgz
cd oci8-2.0.6
./configure --with-oci8=shared,instantclient,/usr/lib/oracle/12.1/client64/lib
make install

At the end of /etc/php.ini file add these 2 lines


And restart apache

service httpd restart

Section 3. Test and reconfigure SELinux

Let’s test it.
Under /var/www/html directory create file index.php with the sample connection to our database.
Below sample code is taken from http://www.php.net/manual/en/oci8.examples.php
Notice how on line 3 I’m using Easy Connect string, so if you are using Easy Connect string you don’t need to have tnsnames.ora file.

<!--?php <br ?-->
$conn = oci_connect('hr', 'welcome', '');
if (!$conn) {
    $e = oci_error();
    trigger_error(htmlentities($e['message'], ENT_QUOTES), E_USER_ERROR);

// Prepare the statement
$stid = oci_parse($conn, 'SELECT * FROM departments');
if (!$stid) {
    $e = oci_error($conn);
    trigger_error(htmlentities($e['message'], ENT_QUOTES), E_USER_ERROR);

// Perform the logic of the query
$r = oci_execute($stid);
if (!$r) {
    $e = oci_error($stid);
    trigger_error(htmlentities($e['message'], ENT_QUOTES), E_USER_ERROR);

// Fetch the results of the query
print "</pre>
\n";while ($row = oci_fetch_array($stid, OCI_ASSOC+OCI_RETURN_NULLS)) { print "\n"; foreach ($row as $item) {print "\n"; } print "\n";}print "
<table border="1">
<td>" . ($item !== null ? htmlentities($item, ENT_QUOTES) : " ") . "</td>



When I try to test it – off course nothing is displayed on the page and I have error in the /var/log/httpd/error_log saying

[Fri Dec 20 09:32:35 2013] [error] [client] PHP Warning:  oci_connect(): ORA-12546: TNS:permission denied in /var/www/html/index.php on line 3
[Fri Dec 20 09:32:35 2013] [error] [client] PHP Fatal error:  ORA-12546: TNS:permission denied in /var/www/html/index.php on line 6

Further examination of the log /var/log/audit/audit.log reveals these errors

type=AVC msg=audit(1387556756.066:17251): avc:  denied  { name_connect } for  pid=30906 comm="httpd" dest=1521 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:oracle_port_t:s0 tclass=tcp_socket

OK, so let’s examine what we can do about it.

getsebool -a|grep -i http|grep -i net
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off

Let’s try to set httpd_can_network_connect to on

setsebool -P httpd_can_network_connect on

And reload our page.
It works!!!

Section 4. References

Posted in Linux, Oracle, php | Tagged , , | 4 Comments

sendmail dsn=5.6.0, stat=Data format error

Just faced this issue today and here are some tips on troubleshooting.
My environment (RedHat 5.9 , sendmail-8.13.8-8.1.el5_7)
While trying to send mail from command line

mail -s "Subject" user@domain < textfile

and monitoring /var/log/maillog I could see that message was rejected with below error

dsn=5.6.0, stat=Data format error

Not a very user friendly error description, so I’ve decided to trace communication with tcpdump between sendmail running on the host from which I’m sending e-mail and relay.
To my surprise this error was coming due to relay not accepting domain name of my server

I had few other servers that were sending e-mails fine to the same relay host, but these servers were running postfix, so I could not compare configuration to sendmail.mc or sendmail.cf files.
Tracing postfix to relay host communication with tcpdump revealed what domain name relay was accepting.
My next step was to masquerade domain name coming from sendmail and make it look same as the one coming from postfix

Here are a few articles that talk exactly about this

How to rewrite sender address
Masquerading and Relaying
Sendmail masquerade outgoing email address

So including these lines in my /etc/mail/sendmail.mc file


Generating new sendmail.cf

make -C /etc/mail

and restarting sendmail

service sendmail restart

Fixed the issue for me.

Posted in Linux, Networking, sendmail | 1 Comment

HP Procurve to Cisco cheat sheet

cisco syntax / terminology HP Procurve syntax / terminology
copy running-config startup-config write memory
access port untagged port
trunk port tagged port
port channel trunk
port channel trunk

SOURCE http://www.sisis.de/sisis-tech/howTo/systems/HP_ProCurve_Switch_2824.html
The switch maintains two configuration files, the running-config and the startup-config.
The startup-config is saved in the flash-memory of the switch. It is the saved configuration und this one is used while booting the switch.
After boot-process is done, the startup-config is copied to the volatile memory und used as running-config. All configuration changes are made in the volatile memory. So you have a possibilty to try changes in the running-config. If you are satisfied that the changes are satisfactory, use the command

write memory

to make the changes permanent. If you wouldn’t do so, the switch will not use the changed options when booting next time.
The following commands are useful to handle the two configuration files:

show config Display a listing of the startup-configuration file of the switch.
show running-config Display a listing of the running-configuration file of the switch.
show config status Compares the current running-configuration file to the startup-config
and determines whether there are updates or no.

To reset the switch to the Factory-Default configuration use the following command:

erase startup-config

If you want to make a backup of the configuration file to an other host you must have a tftp-Server running on this host. The following command saves the running-config to the host to the file “ProCurveConf”

copy running-config tftp ProCurveConf

To use a saved configuration from a remote host, try the following command:

copy tftp running_config ProCurveConf

The last command replaces the running-config with a saved config named “ProCurveConf” on host

Some more information on differences in terminology and supported technology

HP Procurve CLI commands

Posted in Uncategorized | Leave a comment

showmount rpc mount export: RPC: Unable to receive; errno = No route to host

I’ve been setting up nfs server and nfs client on RedHat 6 and ran into a problem with the firewall.
When showmount command was executed from the client – it was returning following error.

[root@gw ~]# showmount -e
rpc mount export: RPC: Unable to receive; errno = No route to host

For a quick test I’ve deactivated firewall rules on nfs server and sure enough – showmount started to work as expected.

[root@gw ~]# showmount -e
Export list for

Then I ran tcpdump trace on the nfs server and ran command showmount from the client.

[root@west sysconfig]# tcpdump -nn host
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:30:20.552571 IP > UDP, length 56
17:30:20.554127 IP > UDP, length 28
17:30:20.554982 IP > Flags [S], seq 391790854, win 5840, options [mss 1460,sackOK,TS val 24287328 ecr 0,nop,wscale 6], length 0
17:30:20.555086 IP > Flags [S.], seq 3599606635, ack 391790855, win 5792, options [mss 1460,sackOK,TS val 10617644 ecr 24287328,nop,wscale 6], length 0
17:30:20.555408 IP > Flags [.], ack 1, win 92, options [nop,nop,TS val 24287328 ecr 10617644], length 0
17:30:20.556825 IP > Flags [P.], seq 1:81, ack 1, win 92, options [nop,nop,TS val 24287330 ecr 10617644], length 80
17:30:20.556849 IP > Flags [.], ack 81, win 91, options [nop,nop,TS val 10617646 ecr 24287330], length 0
17:30:20.561901 IP > Flags [P.], seq 1:77, ack 81, win 91, options [nop,nop,TS val 10617651 ecr 24287330], length 76
17:30:20.564942 IP > Flags [.], ack 77, win 92, options [nop,nop,TS val 24287335 ecr 10617651], length 0
17:30:20.564973 IP > Flags [F.], seq 81, ack 77, win 92, options [nop,nop,TS val 24287336 ecr 10617651], length 0
17:30:20.566009 IP > Flags [F.], seq 77, ack 82, win 91, options [nop,nop,TS val 10617654 ecr 24287336], length 0
17:30:20.566342 IP > Flags [.], ack 78, win 92, options [nop,nop,TS val 24287338 ecr 10617654], length 0

Then activated iptables again and ran another trace

[root@west sysconfig]# tcpdump -nn host
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:28:19.106875 IP > UDP, length 56
17:28:19.108146 IP > UDP, length 28
17:28:19.109633 IP > Flags [S], seq 2783177657, win 5840, options [mss 1460,sackOK,TS val 24168915 ecr 0,nop,wscale 6], length 0
17:28:19.109670 IP > ICMP host unreachable - admin prohibited, length 68
17:28:19.110616 IP > UDP, length 56
17:28:19.111011 IP > UDP, length 28
17:28:19.112546 IP > UDP, length 76
17:28:19.112576 IP > ICMP host unreachable - admin prohibited, length 112

Line 6 in the above trace is particularly interesting.
We can see that client ( is trying to establish TCP connection towards NFS server (, it sends TCP SYN towards port 38796 and there is no Syn-Ack in reply.
If compare this “bad” trace to a successful trace with iptables switched off – we can see on lines 6,7,8 a 3 way TCP handshake.
So this is where communication breaks.

Who is listening on port 38796?

[root@west /]# lsof -i4TCP:38796
rpc.mount 2912 root    7u  IPv4  16008      0t0  TCP *:38796 (LISTEN)

So apparently showmount needs access to rpcbind and also to rpc.mount
Port on which rpc.mount listens can be configured in the /etc/sysconfig/nfs

# Port rpc.mountd should listen on.

Actually it makes sense to configure this port to some known number, otherwise during next reboot rpc.mount can as well be listening on a totally different port.

Also we can see rpcmount port via

[root@west /]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100021    1   udp  38286  nlockmgr
    100021    3   udp  38286  nlockmgr
    100021    4   udp  38286  nlockmgr
    100021    1   tcp  56953  nlockmgr
    100021    3   tcp  56953  nlockmgr
    100021    4   tcp  56953  nlockmgr
    100005    1   udp  53364  mountd
    100005    1   tcp  38796  mountd
    100005    2   udp  53364  mountd
    100005    2   tcp  38796  mountd
    100005    3   udp  53364  mountd
    100005    3   tcp  38796  mountd

Long story short:
I set rpc.mount port to 892
added below rule to iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 892 -j ACCEPT

And rebooted nfs server

My showmount command started to work.
Byproduct of this is that now I can browse and automount shares via autofs from the client.
Previously it was impossible due to showmount returning error.

PS: After I did all this I found that it’s actually documented on the
RedHat documentation portal.

Posted in Linux, Uncategorized | Tagged | 3 Comments

Making webex work on 64bit Fedora Core 18

Webex on 64 bit linux

I’ve seen in many places people having issues with the webex on 64bit systems.
Some reported that installing 32 bit version of firefox, 32 bit version of java and running webex under this 32 bit version environment works for them.

In my case it didn’t work, or rather most important features of webex didn’t work.
I could not share my screen or see other’s screen.
When I was sharing my screen – other people just saw a green background.
And when others were sharing – I saw nothing at all, except webex main window.
I had a fairly fresh installation of Fedora Core 18 64 bit so I imagine that most of people who will install Fedora Core 18 will have roughly the same set of packages.

Here is how I made webex work and not just in 32 bit mode, but in 64 bit as well!
So as a first step let’s make sure that we have strace package

yum install strace

Now we can trace processes, but webex is a multithreaded application, so to trace it properly we need to include option -ff. With this option it will track new opened threads and will create a separate trace file for each of these threads.
Without this option all we will get in the output will be something similar to this

Process 12610 attached
futex(0x7f100e64c9d0, FUTEX_WAIT, 12613, NULL

Which will sit there for the duration of the webex session.

In the below command I’m tracing webex and outputting trace information to files webex.$pid
where $pid is each individual thread process id.

strace -ff -t -p `ps -ef|grep -v grep|grep java|awk '{ print $2 }'` -o webex

Above command assumes that we don’t have any other java stuff running on the system at the time of tracing, otherwise we might need to adjust grep statement.

So let’s start our webex session, in terminal run tracing command and wait till we have webex main window open.
Now let’s close this webex window.
And it’s time to examine our trace files.

Let’s search for open system call

grep open webex.????

We will see lots of output.
This output means that different threads are searching for shared libraries on the system (or some other files as well).
Normally library search process goes like this and when library is finally found on the system it returs some file descriptor like on line 3

webex.8406:14:58:18 open("/usr/lib/firefox/libXt.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/firefox/plugins/libXt.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/lib/libXt.so.6", O_RDONLY|O_CLOEXEC) = 3

Examining output I noticed that one of the libraries was not found and it seems like it also produced error message on line 8

webex.8406:14:58:18 open("/usr/lib/tls/i686/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/tls/sse2/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/tls/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/i686/sse2/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/i686/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/sse2/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 open("/usr/lib/libpangox-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
webex.8406:14:58:18 writev(2, [{"/home/dmitry/.webex/1124/atasjni", 32}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libpangox-1.0.so.0", 18}, {": ", 2}, {"cannot open shared object file", 30}, {": ", 2}, {"No such file or directory", 25}, {"\n", 1}], 10) = 150

Luckily this error message was at the very bottom of the output, so I didn’t have to go throgh all 1047 lines returned to me by grep command.

My next step was to find out what package this shared library belongs to

yum provides "*/libpangox-1.0.so.0"

And then installing it.

yum install pangox-compat-0.0.2-1.fc18.x86_64

Above I’m installing 64 bit version since my system is 64 bit, but if you are on a 32 bit OS or using 32 bit workaround – then just install 32 bit version of it.

yum install pangox-compat-0.0.2-1.fc18.i686

After installing above package my webex started to work like it was intended to.
I can share my screen and I can see while others sharing.
And it’s all on 64 bit – my firefox is 64bit and my java is 64 bit.

Posted in Linux, RedHat | Tagged , , | 12 Comments

Installing systemtap on CentOS 6

In this article I will post series of commands that I used for installing systemtap on CentOS 6.3

yum --disablerepo="*" --enablerepo="centosplus" install kernel.x86_64
yum install kernel-debuginfo.x86_64


[root@lab1 ~]# uname -r
2.6.32-279.22.1.el6.centos.plus.x86_64    ### After reboot we are on a new kernel from the centosplus repo
[root@lab1 ~]# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Checking "/lib/modules/2.6.32-279.22.1.el6.centos.plus.x86_64/build/.config" failed with error: No such file or directory
Incorrect version or missing kernel-devel package, use: yum install kernel-devel-2.6.32-279.22.1.el6.centos.plus.x86_64 

yum --disablerepo="*" --enablerepo="centosplus" install kernel-devel.x86_64

[root@lab1 yum.repos.d]# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 82 library script(s) using 194432virt/23108res/3032shr kb, in 150usr/140sys/330real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 419080virt/120152res/8200shr kb, in 1600usr/870sys/2719real ms.
Pass 3: translated to C into "/tmp/stapfsss5b/stap_fa03e96bd5bec496c0cb8ac61c9c099c_1590_src.c" using 409312virt/116044res/6532shr kb, in 20usr/0sys/16real ms.
Pass 4: compiled C into "stap_fa03e96bd5bec496c0cb8ac61c9c099c_1590.ko" in 11210usr/3570sys/15241real ms.
Pass 5: starting run.
read performed    ### Success
Pass 5: run completed in 50usr/90sys/436real ms.
Posted in RedHat | Tagged | 2 Comments

Troubleshooting Application issues in the load balanced clusters with Windows 2008 logon banner

I often work with Citrix farms running on top of Windows 2008.
From time to time there are users complaining about inconsistent behavior of their published applications.
From the users perspective – they don’t know which server in the farm they are hitting when they access published app, so it becomes difficult to narrow down sometime which particular server in a farm is misbehaving.

One of the ways to allow users to see which server in the cluster they are hitting is – adding server identification line in the security logon banner.

LegalNoticeText = "You are about to access Server 1 in the farm."

And if you don’t have logon banner enabled – you can enable it via GPO – details are here. Even though this link is not for Windows 2008 – same hierarchy applies. Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies/Security Options -> Interactive Logon.

Posted in Windows | Tagged , | Leave a comment

Online data relocation in a clustered environment under RedHat 4

In this article I will talk about online data relocation in a clustered environment.
Under clustered environment I mean Red Hat Cluster Suite running on RHEL 4.9.
This article was inspired by one of our clients requesting to migrate data from one of the physical volumes to another physical volume without unmounting file system. At first look I thought this was going to be easy, but clustered environment plus really outdated version of RedHat made this task a bit more complex than I originaly thought.

Production system that I needed to migrate is running
kernel 2.6.9-101.ELsmp
lvm2 with lvm2-cluster
All of the file systems reside on top of lvm2
Some of them are ext3 and some are gfs
Physical volumes look like this

# pvs
  PV                VG       Fmt  Attr PSize   PFree
  /dev/mapper/lun00 vgarc    lvm2 a-     1.05T      0
  /dev/mapper/lun01 vgredo   lvm2 a-   198.00G      0
  /dev/mapper/lun02 vgglobal lvm2 a-   850.00G 351.00G
  /dev/mapper/lun03 vgdb     lvm2 a-   990.00G      0
  /dev/mapper/lun05 vgbackup lvm2 a-     5.01T      0
  /dev/mapper/lun30 vgglobal lvm2 a-   150.00G      0

Notice how lun02 and lun30 are both in the vgglobal volume group?
So this is what the request was for.
Client wanted to move physical extents from lun30 to lun02 and then destroy lun30.

In theory this task can be as simple as

pvmove -i 5 /dev/mapper/lun30 /dev/mapper/lun02
vgreduce vgglobal /dev/mapper/lun30

But according to RedHat 4 Cluster Logical Volume Manager documentation online migration is unsupported in a cluster

Because the pvmove command uses mirroring, it is not cluster-aware and needs exclusive access to a volume. For information on activating logical volumes on individual nodes in a cluster, see Section 4.8, “Activating Logical Volumes on Individual Nodes in a Cluster”.

However RedHat 5 Logical Volume Manager Administration
has a note that online migration is possible

In order to perform a pvmove operation in a cluster, you should ensure that the cmirror and cmirror-kmod packages are installed and that the cmirror service is running. The cmirror-kmod package that must be installed depends on the kernel that is running. For example, if the running kernel is kernel-largesmp, it is necessary to have cmirror-kmod-largesmp for the corresponding kernel version.

My next move was to head to CentOS Vault and check if there is anything similar to cmirror and cmirror-kmod in the 4.x branch. And there was something promising, 2 cmirror packages and a series of cmirror-kernel-* packages for different kernel flavors (regular kernel, smp, largesmp, XenU). The latest cmirror package http://vault.centos.org/4.9/csgfs/x86_64/RPMS/cmirror-1.0.2-1.el4_8.x86_64.rpm is kernel independent and all it provides is the init.d script to load and unload module

# rpm -qpl cmirror-1.0.2-1.el4_8.x86_64.rpm

But cmirror-kernel* packages containing the actual dm-cmirror.ko module are kernel dependent so I had to find one that was built for my kernel – 2.6.9-101.ELsmp

yum provides "/lib/modules/2.6.9-101.ELsmp/kernel/cluster/dm-cmirror.ko"

yelds nothing. May be there is one on the RedHat portal, but in the CentOS vault there is none matching my particular kernel.
Since I didn’t feel like chasing people within my company who have RedHat portal access I decided to explore possibility to build this module from sources.
Latest source rpm for this module can be found here http://vault.centos.org/4.9/csgfs/SRPMS/cmirror-kernel-2.6.9-43.19.el4.src.rpm
So let’s install it

 rpm -ivh http://vault.centos.org/4.9/csgfs/SRPMS/cmirror-kernel-2.6.9-43.19.el4.src.rpm

And before building it let’s modify specs file a bit, or else it will ask you to have many kernel dependencies for regule, largesmp and xen versions.
So in the file /usr/src/redhat/SPECS/cmirror-kernel.spec let’s modify these lines

%define buildlargesmp 1
%define buildxen 1
%define kernel_version 2.6.9-103.EL

to these

%define buildlargesmp 0
%define buildxen 0
%define kernel_version 2.6.9-101.EL

Ok, now it’s time to build rpm

rpmbuild -bb /usr/src/redhat/SPECS/cmirror-kernel.spec

After build is finished let’s install it

rpm -ivh /usr/src/redhat/RPMS/x86_64/cmirror-kernel-smp-2.6.9-43.19.x86_64.rpm

And now it’s time to load module and test pvmove.

Since it wasn’t wise to do all above steps and testing on the actual production system I ended up creating Red Hat Cluster with 2 nodes under virtual box (well, actually I created a CentOS cluster and matched kernel and package versions to production system). Presented 2 LUNs via iSCSI from the main host and these became my 2 PVs.
Below are results of the simulation

[root@node-1 ~]# clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  node-1                                   Online, Local, rgmanager
  node-2                                   Online, rgmanager

[root@node-1 ~]# uname -a
Linux node-1.lab 2.6.9-101.ELsmp #1 SMP Thu Jul 21 17:46:19 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@node-1 ~]# iscsi-ls -l
SFNet iSCSI Driver Version ...4:0.1.11-7(14-Apr-2008)
TARGET NAME             : iqn.2013-02.com.example.ubuntu:lun1
TARGET ALIAS            :
HOST ID                 : 2
BUS ID                  : 0
TARGET ID               : 0
TARGET ADDRESS          :,1
SESSION STATUS          : ESTABLISHED AT Sat Feb 16 09:32:12 CST 2013
SESSION ID              : ISID 00023d000001 TSIH 300

LUN ID : 0
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI SCSI revision: 05

LUN ID : 1
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 05
  page83 type1: IET 00010001
2:0:0:1: expected length 24, got length 20
  page80: 0a
  Device: /dev/sdb
LUN ID : 2
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 05
  page83 type1: IET 00010002
2:0:0:2: expected length 24, got length 20
  page80: 0a
  Device: /dev/sdc

[root@node-1 ~]# pvs
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a-     7.88G      0
  /dev/sdb   vgglobal   lvm2 a-   808.00M 484.00M             ### ISCSI LUN 1
  /dev/sdc   vgglobal   lvm2 a-   140.00M  12.00M             ### ISCSI LUN 2

[root@node-1 ~]# lvs
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  LogVol00 VolGroup00 -wi-ao   6.91G
  LogVol01 VolGroup00 -wi-ao 992.00M
  lvrepnb  vgglobal   -wi-ao 128.00M                           ### We will be moving physical extents of this logical volume
  mygfs    vgglobal   -wi-ao 324.00M

[root@node-1 ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/mapper/vgglobal-mygfs on /mygfs type gfs (rw)               ### This is gfs file system mounted on both nodes simultaneously
/dev/mapper/vgglobal-lvrepnb on /mypackagefs type ext3 (rw)      ### This is ext3 file system that we will attempt to move

Now let’s try to attempt pvmove without cmirror-kernel module.

[root@node-1 ~]# pvmove -i 5 -n vgglobal/lvrepnb /dev/sdc /dev/sdb
FATAL: Could not open '/lib/modules/2.6.9-101.ELsmp/kernel/cluster/dm-cmirror.ko': No such file or directory
  /sbin/modprobe failed: 1
  Error locking on node node-1: Volume is busy on another node
  Failed to activate lvrepnb

Now let’s install cmirror-kernel package that we built from source before and cmirror package.
Make sure to install on all nodes in a cluster!

rpm -ivh http://vault.centos.org/4.9/csgfs/x86_64/RPMS/cmirror-1.0.2-1.el4_8.x86_64.rpm
rpm -ivh /usr/src/redhat/RPMS/x86_64/cmirror-kernel-smp-2.6.9-43.19.x86_64.rpm
service cmirror start

And let’s attempt pvmove once again

[root@node-1 x86_64]# pvmove -i 5 -n vgglobal/lvrepnb /dev/sdc /dev/sdb
  /dev/sdc: Moved: 12.5%
  /dev/sdc: Moved: 25.0%
  /dev/sdc: Moved: 34.4%
  /dev/sdc: Moved: 46.9%
  /dev/sdc: Moved: 56.2%
  /dev/sdc: Moved: 68.8%
  /dev/sdc: Moved: 78.1%
  /dev/sdc: Moved: 90.6%
  /dev/sdc: Moved: 100.0%
[root@node-1 x86_64]# pvs
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a-     7.88G      0
  /dev/sdb   vgglobal   lvm2 a-   808.00M 356.00M
  /dev/sdc   vgglobal   lvm2 a-   140.00M 140.00M

During the pvmove I unmounted vgglobal/lvrepnb file system that was being moved and mounted it on node-2 and then mounted it back on node-1. Consistency of the files was not compromised after move.

Posted in RedHat | Tagged , | Leave a comment

Linux notes

Various uncategorized Linux notes

Shell, consoles, environment

man searches for files under $MANPATH
Files under $MANPATH can be displayed on the screen with
nroff -man filename

shell checks for new e-mails every 5 minutes

How long shell has been running

turn off mail call

Display console number

Switch to left or right console

Alt + Print Screen
Switch to the previous console

chvt N
Switch to console N

Resets terminal

Reads information from file /var/log/wtmp

apropos keyword
Search for keyword in the documentation

apropos is an alias for man -k

whatis keyword
One line description of a program keyword

info files are located under /usr/info directory
If info file is located somewhere else it’s possible to use
info -f filename.info

To pass special characters as a string – add $ in front

echo Hello$'\n\n'

Special characters

\b backspace
\e escape
\n newline
\t horizontal tab
\v vertical tab
\NNN character whose ASCII code is NNN in octal (base 8)

Display .gz files on the fly without gunzip

zless filename.gz

EMACS style key bindings

Key Sequence Action
Ctrl + K Kill or cut all the text
Ctrl + U Kill everything on a command line to the left of the cursor
Ctrl + Y Paste the text that was killed
Ctrl + A Cursor to the beginning of line
Ctrl + E Cursor to the end of line
Alt + F Move cursor forward one word
Alt + B Move cursor backward one word

Undoing mistakes on command line

Key Sequence Action
Ctrl + T Transpose 2 characters before cursor
Alt + T Transpose 2 words before cursor

Changing prompt

Variable PS1 is responsible for prompt
\d current date
\h hostname
\n newline
\t current system time in 24 hour format
\@ current system time in 12 hour format
\w current working directory
\u user name
\! history number of this command

Posted in Linux | Tagged | Leave a comment