crsd.bin hangs on HP-UX results ASM hang and DB hang with ORA-240

I had a lot of project with Oracle RAC on HP-UX based on Intel Itanuim platform.

We started with version 10.1.0.4, but quickly moved to 10.2.0.2 when it became available.

There were clusters with 2, 4 and 8 nodes and there weren’t stability problems with 2 or 4 nodes cluster, but those one – with 8 nodes … was the real nightmare…

At some point in time there were about 28 databases and ASM… was hanging…  every 2 days!!!

To temporary resolve this hanging issue customer had to shutdown all ASM instances at once and than start them – so it required FULL CUSTER OUTAGE

All this ASM stability issues disappeared after installing CRS Bundle #2 over 10.2.0.4, but recently I faced with one My Oracle Support NOTE that reminded me that store. It was

1105825.1 HP Itanium – ORA-240 or process on ASM & Database hang

with quite famous symptoms:

  • Processes on the database report error ORA-240.
  • The process holding the CF enqueue waits on KSV Master Wait

in this NOTE Oracle stated that the cause was:

“This is a problem caused for some HPUX patches as described  on this HP document”

so lets look into mentioned HP document:

“It seems this problem can also be caused by the installation of HP patches PHKL_38762 or PHKL_39145. QXCR1000940361 tracked the problem. This has been fixed now, and the problem is not present anymore when installing the superseded patch PHKL_40208

but the more important part of this HP document is:

“If any patches related to Oracle RAC are included in a patch bundle (whether it is HP-UX specific patching that RAC depends upon or vice versa), RAC requires that the binaries be relinked prior to starting them.”

.

So, this is THE REMINDER when MANUAL RELINK may be required and this step is often forgotten…

Let’s look into NOTE:131321.1 How to Relink Oracle Database Software on UNIX:

Relinking Oracle manually is suggested under the following circumstances (even though the OS vendor may not require it):

  • An OS upgrade has occurred.
  • A change has been made to the OS system libraries.  This can occur during the application of an OS patch.
  • A new install failed during the relinking phase.
  • Individual Oracle executables core dump during initial startup.
  • An individual Oracle patch has been applied (however, explicit relink  instructions are usually either included in the README or integrated into the patch install script)

.

Conclusion:

  • don’t forget to relink oracle binaries after upgrading OS or after installing patches to the OS system libraries
  • often OS maintenance is a responsibility of another person/team, so task for Oracle binaries relinking have to be additionally scheduled

PS:

for more reasons for ORA-240 read previous posts: here and there

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s