Only Four Left…

I feel like a n00b for even posting this, but if part of my goal here is to catalog my mistakes, well, here's one I probably won't make again.

I've been involved in a project to migrate an E-Business Suite database tier from 32-bit to 64-bit RDBMS. Thus far it's been pretty straightforward. One thing had been bugging me, though. Our original test platform was 64-bit SUSE Linux Enterprise Server 9 (SLES9), but a decision was made to upgrade to SLES10 instead. Not a big deal in this case, except for one thing: the settings I'd used in SLES9 to allocate hugepages weren't doing a darn thing in SLES10:

sles10db:~ # grep -i nr_huge /etc/sysctl.conf
vm.nr_hugepages = 8196

sles10db:~ # grep memlock /etc/security/limits.conf
oracle soft memlock 17000000
oracle hard memlock 17000000

Okay so far...

sles10db:~ # grep -i huge /proc/meminfo
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

sles10db:~ # cat /proc/sys/vm/nr_hugepages
0

Whahuh? Why wasn't my nr_hugepages setting getting picked up from /etc/sysctl.conf? Wrong question, of course. If other settings in that file were being picked up (they were), then clearly something else was clobbering the value I had set. Sure enough, after not much poking around, I found this:

sles10db:~ # grep -i nr_huge /etc/sysconfig/oracle
# /proc/sys/vm/nr_hugepages indicates the current number of configured hugetlb
NR_HUGE_PAGES=0

How embarrassing is that? It's even documented in a very official, check-here-first sort of place. Since /etc/init.d/oracle is run long after /etc/init.d/boot.sysctl, the values set in /etc/sysconfig/oracle clearly will clobber anything set in /etc/sysctl.conf. My only defense is that NR_HUGE_PAGES was not included in /etc/sysconfig/oracle on our SLES9 test system, which is why I was setting it in /etc/sysctl.conf in the first place. Of course, we might've been using an old version of orarun.rpm on that server, so it's still a pretty weak excuse. At any rate, changing the setting in /etc/sysconfig/oracle and running rcoracle restart did the job:

sles10db:~ # grep -i nr_huge /etc/sysconfig/oracle
# /proc/sys/vm/nr_hugepages indicates the current number of configured hugetlb
NR_HUGE_PAGES=8196

sles10db:~ # grep -i huge /proc/meminfo
HugePages_Total:  4645
HugePages_Free:   4645
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Hope this helps someone out there; otherwise, I'll just sit here, alone in my hopefully-temporary idiocy.