crsd.bin hangs on HP-UX results ASM hang and DB hang with ORA-240

I had a lot of project with Oracle RAC on HP-UX based on Intel Itanuim platform.

We started with version, but quickly moved to when it became available.

There were clusters with 2, 4 and 8 nodes and there weren’t stability problems with 2 or 4 nodes cluster, but those one – with 8 nodes … was the real nightmare…

At some point in time there were about 28 databases and ASM… was hanging…  every 2 days!!!

To temporary resolve this hanging issue customer had to shutdown all ASM instances at once and than start them – so it required FULL CUSTER OUTAGE

All this ASM stability issues disappeared after installing CRS Bundle #2 over, but recently I faced with one My Oracle Support NOTE that reminded me that store. It was

1105825.1 HP Itanium – ORA-240 or process on ASM & Database hang

with quite famous symptoms:

  • Processes on the database report error ORA-240.
  • The process holding the CF enqueue waits on KSV Master Wait

in this NOTE Oracle stated that the cause was:

“This is a problem caused for some HPUX patches as described  on this HP document”

so lets look into mentioned HP document:

“It seems this problem can also be caused by the installation of HP patches PHKL_38762 or PHKL_39145. QXCR1000940361 tracked the problem. This has been fixed now, and the problem is not present anymore when installing the superseded patch PHKL_40208

but the more important part of this HP document is:

“If any patches related to Oracle RAC are included in a patch bundle (whether it is HP-UX specific patching that RAC depends upon or vice versa), RAC requires that the binaries be relinked prior to starting them.”


So, this is THE REMINDER when MANUAL RELINK may be required and this step is often forgotten…

Let’s look into NOTE:131321.1 How to Relink Oracle Database Software on UNIX:

Relinking Oracle manually is suggested under the following circumstances (even though the OS vendor may not require it):

  • An OS upgrade has occurred.
  • A change has been made to the OS system libraries.  This can occur during the application of an OS patch.
  • A new install failed during the relinking phase.
  • Individual Oracle executables core dump during initial startup.
  • An individual Oracle patch has been applied (however, explicit relink  instructions are usually either included in the README or integrated into the patch install script)



  • don’t forget to relink oracle binaries after upgrading OS or after installing patches to the OS system libraries
  • often OS maintenance is a responsibility of another person/team, so task for Oracle binaries relinking have to be additionally scheduled


for more reasons for ORA-240 read previous posts: here and there


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s