I had a lot of project with Oracle RAC on HP-UX based on Intel Itanuim platform.
We started with version 10.1.0.4, but quickly moved to 10.2.0.2 when it became available.
There were clusters with 2, 4 and 8 nodes and there weren’t stability problems with 2 or 4 nodes cluster, but those one – with 8 nodes … was the real nightmare…
At some point in time there were about 28 databases and ASM… was hanging… every 2 days!!!
To temporary resolve this hanging issue customer had to shutdown all ASM instances at once and than start them – so it required FULL CUSTER OUTAGE…
All this ASM stability issues disappeared after installing CRS Bundle #2 over 10.2.0.4, but recently I faced with one My Oracle Support NOTE that reminded me that store. It was
1105825.1 HP Itanium – ORA-240 or process on ASM & Database hang
with quite famous symptoms:
- Processes on the database report error ORA-240.
- The process holding the CF enqueue waits on KSV Master Wait
in this NOTE Oracle stated that the cause was:
“This is a problem caused for some HPUX patches as described on this HP document”
so lets look into mentioned HP document:
“It seems this problem can also be caused by the installation of HP patches PHKL_38762 or PHKL_39145. QXCR1000940361 tracked the problem. This has been fixed now, and the problem is not present anymore when installing the superseded patch PHKL_40208“
but the more important part of this HP document is:
“If any patches related to Oracle RAC are included in a patch bundle (whether it is HP-UX specific patching that RAC depends upon or vice versa), RAC requires that the binaries be relinked prior to starting them.”
So, this is THE REMINDER when MANUAL RELINK may be required and this step is often forgotten…
Let’s look into NOTE:131321.1 How to Relink Oracle Database Software on UNIX:
Relinking Oracle manually is suggested under the following circumstances (even though the OS vendor may not require it):
- An OS upgrade has occurred.
- A change has been made to the OS system libraries. This can occur during the application of an OS patch.
- A new install failed during the relinking phase.
- Individual Oracle executables core dump during initial startup.
- An individual Oracle patch has been applied (however, explicit relink instructions are usually either included in the README or integrated into the patch install script)
- don’t forget to relink oracle binaries after upgrading OS or after installing patches to the OS system libraries
- often OS maintenance is a responsibility of another person/team, so task for Oracle binaries relinking have to be additionally scheduled