install perl DBD::Oracle (Lesson learned: CPAN and yum don't mix)

From the many tales of woe on the web about installing perl DBD::Oracle, from "invalid lvalue in assignment" to mysterious make errors, and the pages of intricate instructions doubtfully translated from the French, I assumed that it was a long and difficult process and it was natural that I was having problems installing on 64-bit CentOS. WRONG! It can actually be easy for lazy and dumb people like me.

First, just use yum and hand-compiling. Don't add CPAN to the mix.

1. Add the rpmforge repo and the EPEL repo (see links for instructions) so that you can install perl-DBD and perl-DBI via yum.
2. Install perl-DBD and perl-DBI via yum.
3. Download and install the OCI client "basic" and "sdk/devel" packages from Oracle. Note that you might need an older version if you're connecting to an older version of Oracle. Note also that Oracle makes you log in to download this. Note also that you need both the SDK and the Basic package. I recommend getting the rpms - install with a simple rpm -Uvh .
4. Oracle puts the libraries in a wacky place, e.g. /usr/lib/oracle/11.2/client64/lib if you're using the 64-bit version of 11.2. Therefore, create a new file, e.g. oci.conf, in /etc/ld.so.conf.d/, with the library location in it, and then run (as root) ldconfig -v to add it.
5. Download the DBD::Oracle source from cpan and extract it someplace. 
6. Set some environment variables:
   

   export ORACLE_HOME=/usr/lib/oracle/11.2/client64
   export PATH=$PATH:$ORACLE_HOME/bin
   export LD_LIBRARY_PATH=$ORACLE_HOME/lib

7. Run "perl Makefile.PL -V 11.2.0" in the directory where the OCI client was extracted. Change the version number to whatever the correct version is. This avoids the "I could not determine Oracle client version so I'll just default to version 8.0.0.0" issue.
8. Run "make install".
9. You should be done!

The latest versions of the Oracle Instant Client default to making incredibly verbose logs in the user's home directories - how to disable default oradiag_user instant client logging.

perl cpan "recursive dependency detected"

Using the CPAN shell for the first time on a new server, after install Bundle::CPAN" and then "reload cpan", on the next install I got:

Recursive dependency detected:
    Bundle::CPAN
 => ExtUtils::MakeMaker
 => M/MS/MSCHWERN/ExtUtils-MakeMaker-6.56.tar.gz
 => File::Spec
 => S/SM/SMUELLER/PathTools-3.33.tar.gz
 => File::Path
 => D/DL/DLAND/File-Path-2.08.tar.gz
 => File::Spec.
Cannot continue.

in the perl cpan installer. Alas! I thought. Will I have to install the old-fashioned way? Indeed no! This may just be a sign that you need to "install Bundle::CPAN" and then "reload cpan".

But I just did that, damn it! Apparently, "reload cpan" isn't always completely effective after "install Bundle::CPAN". If you encounter this error, try quitting cpan and reloading.

Proof yet again that almost all tech problems can be solved by restarting the (whatever).

Btw, if you're not using the CPAN shell, use perl -MCPAN -e 'install( q{Bundle::CPAN} )' to install/update the cpan bundle.

xen guest I/O issue: task blocked for more than 120 seconds

(Update to this post here, with more info for G7 blades.)

A lot of people on CentOS/RHEL who run Xen (and apparently also people on Debian/Ubuntu) who have HP or Fujitsu hardware RAIDs are experiencing this issue. Under sustained, heavy I/O on the guest/domU, the guest load average will rise (beyond 20 even on a single-CPU VM) and messages like the following will appear in syslog:

INFO: task syslogd:1500 blocked for more than 120 seconds. 
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
syslogd D 0000000000000110 0 1500 1 1503 1491 (NOTLB) 
 ffff8800b0739d88 0000000000000286 ffff8800b8922970 ffff8800b8922970 
 0000000000000009 ffff8800bb2dd0c0 ffff8800baa55080 0000000000002b40 
 ffff8800bb2dd2a8 0000000000000000 
Call Trace: 
 [] :jbd:log_wait_commit+0xa3/0xf5 
 [] autoremove_wake_function+0x0/0x2e 
 [] :jbd:journal_stop+0x1cf/0x1ff 
 [] __writeback_single_inode+0x1d9/0x318 
 [] do_readv_writev+0x26e/0x291 
 [] sync_inode+0x24/0x33 
 [] :ext3:ext3_sync_file+0xcc/0xf8 
 [] do_fsync+0x52/0xa4 
 [] __do_fsync+0x23/0x36 
 [] tracesys+0xab/0xb6 

Two excellent bugs to start out with are:
http://bugs.centos.org/view.php?id=4515
https://bugzilla.redhat.com/show_bug.cgi?id=605444

This problem has apparently cropped up with all -194 kernels and persists into the -238 kernels. I find it continues even with CentOS 5.6 and 2.6.18-238.9.1.el5xen. I've found that it affects all versions of Xen from 3.0.3 to 3.4.3 to 4.1.

There is a firmware update for HP that you should definitely apply, but it doesn't appear to fix the problem completely. (Supposedly, the Fujitsu firmware update does fix the issue.) Particularly, some users got an aggravated version of the problem when a firmware issue caused the onboard battery not to charge. Other users were able to help the issue by disabling irqbalanced, though of course you may not want to do this if your vm uses multiple processors.

One extra source of ambiguity with these forum posts is that it's not clear what other people are doing to cause this issue. Post firmware upgrade, this issue can only reliably be triggered with sustained, heavy I/O. I used dd if=/dev/zero of=./test1024M bs=1024k count=1024 conv=fsync to ensure problems during testing.
- On a dom0, this test would complete in seconds with an average speed of 72-ish mb/second.
- On an affected domU, this test would cause the above-referenced load and dmesg messages, cause the entire system to be unresponsive, and have an average speed around 300 bytes/second.

The two things that made a material difference in our case, with HP Proliant BL460 G5s running CentOS 5.5 and 5.6 with CentOS guest vms, were, after the firmware update:
 - Switching the I/O scheduler on both the dom0 and the domU to noop
 - Capping the max raid speed on both the dom0 and the domU.

You can temporarily switch the I/O scheduler with something like:
echo noop > /sys/block/[your-block-device-name]/queue/schedule
... to change the default on reboot, add "elevator=noop" to the kernel line in /etc/grub.conf .

To set the max raid speed, you  can use
echo "50000" > /proc/sys/dev/raid/speed_limit_max
The default for raid/speed_limit_max is 200K; this sets it to 50K. I had expected that this would only matter on the dom0, since the domU doesn't technically have a raid, but putting it in both places eliminated some intermittent issues.

After changing the scheduler, I found that the domU 1GB copy from /dev/zero would average over 40MB/sec, but would still sometimes freak out, crawl at 2MB/sec, and fill up dmesg with complaints. The load would sometimes climb near 10. After setting the raid max speed, I found that the domU 1GB copy would always be between 47 and 49 MB/sec, the load would never climb above 1.5, and the VM stayed responsive. So it's a band-aid solution, but very effective, and should tide us over until there is a kernel or firmware solution.

*** *** ***

Update 20 June 2011:

Looking over the latest comments (see 28+) on https://bugzilla.redhat.com/show_bug.cgi?id=615543 , it seems that people running certain HP Smart Arrays may completely solve this issue by upgrading their driver AND firmware, though people on other models are not quite so lucky and still need to use the noop scheduler.

I also want to note that if you're in the middle of a slowdown due to this issue and can get any response at all, you'll probably notice that the amount of "Dirty" memory in /proc/meminfo is very high, and running "sync" can sometimes force your system out of the freeze a little quicker than it would find its way out on its own.
awk '/Dirty/ {printf( "%d\n", $2 )}' /proc/meminfo
will give you just the integer by itself, if you want a script to monitor this value.

executing java from php in rhel/centos - not enough space for code cache

The apache user could run java programs just fine from the command line, but even the simplest java command in php would fail:

<?php
system('java -version');
?>

with the error 

Error occurred during initialization of VM
Could not reserve enough space for code cache. 

We already had the setsebool -P httpd_ssi_exec=1 option on, so that Apache had permission to execute whatever. Strangely, however, this "memory" issue is also due, somehow to selinux. Disabling selinux temporarily (e.g. with echo 0 >/selinux/enforce) or setting setsebool -P httpd_disable_trans=1 fixes the problem. I'd like less of a sledgehammer solution, but messing with other selinux settings does not seem to have any effect.


Edit: As a commenter below just indicated, you can use setsebool -P httpd_execmem=1 to disable this setting. If that doesn't work, well, back to the sledgehammer.

hosts.conf on os x

Use sudo vi /etc/hosts on OS X to set custom DNS resolutions, e.g. to test an Apache configuration prior to hostmaster changes. The syntax is just the familiar [ip] [tab] [hostname].

xen on centos on hp blades: slow networking

Had Xen running on CentOS on some HP 460c blades. CentOS by itself on the blades had great network performance, but the blades running with the Xen kernel had terrible performance, both in the domU and the dom0. It turns out that the issue is the particular networking driver that the CentOS/RHEL/Fedora family has for the blades' Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe cards. You can see something similar on this RHEL bug.

The lazy workaround is to add "options bnx2x disable_tpa=1" to /etc/modprobe.conf on the dom0 and then restart the dom0. Like so:

# cat /etc/modprobe.conf

alias eth0 bnx2x

alias eth1 bnx2x

options bnx2x disable_tpa=1

alias scsi_hostadapter cciss

alias scsi_hostadapter1 usb-storage

# 

HP advises that you can manually reload the driver, but this did not work for me.

command-line mysql connection works, but not php

I was able to connect to a remote MySQL server from the command line on one of my servers, but that server, running php, was not able to make the connection, just returning a frustrating generic "cannot connect" message. The problem was of course selinux, and the solution:

# setsebool -P httpd_can_network_connect=1

Hat tip.