rac2 troubles

I had MANY problems with rac2 and spent hours installing, reinstalling grid. Here are some of the issues I encountered.

After installing the OS, and configuring the storage and asm, make sure you can see the disks after a clean reboot on all nodes. Especially on your rac2.

[root@rac2 crs]# oracleasm listdisks
CRSVOL
DATAVOL
FRAVOL

When grid has been installed, after a reboot, make sure you can see the asm disks (above). And also make sure the ocrcheck is valid.

[root@rac2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2252
Available space (kbytes) : 259868
ID : 2022892366
Device/File Name : +CRSDG
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@rac2 bin]#



Other issues are as follows.

symptom:
while running root.sh during installation, we get
/u01/app/11.2.0/grid/lib/libclntsh.so.11.1: cannot restore segment prot after reloc: Permission denied

fix:
deconfig, and run root.sh again
su - root (do this even if you are already root)
/u01/app/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force
chcon -t textrel_shlib_t /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
/u01/app/11.2.0/grid/root.sh


symptom:
after a reboot of rac2,
[root@rac2 alert]# cat /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_4644.trc

...
----- Error Stack Dump -----
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper

fix:
vi /etc/sysctl.conf
modify: kernel.sem = 250 100 -> kernel.sem = 250 32000 100
sysctl -p
reboot