Another reason for ORA-240 with ASM & RAC in place

I was looking which search engine queries forwarded people to my blog and was quite surprised that most of them were like this one:

ORA-00240: control file enqueue held for

People doing searches by this phrase almost every day…

So I decided to point to another reason for ORA-240 with ASM & RAC.

The Issue is connected with fact that Unix/Linux processes started through inittab(like CRSD & CSSD) don’t inherit ulimit -n settings specified in profile/login scripts.

Let’s look at some details of the problem – just short excerpts:

NOTE: 858279.1 ASM and Database Instance hang when exceeding around 1800 sessions

Cause:

cssd was stuck due to reached max open files limit at 1024.

Regardless of ulimit value set it to unlimit for oracle and root user.Max open file setting in limits.conf doesn’t take affect for ocssd.

CSSD started via /etc/inittab, then login scripts aren’t executed.

so limits aren’t set.We need to explicitly set them in the init scripts.

Another issue that has the common root with previous one:

NOTE: 416784.1 WARNING:Oracle instance running on a system with low open file descriptor limit

Cause:

The ‘ulimit -n unlimited’ is not taking effect for processes started via inittab

On linux using something higher to 65536 or unlimited set the actual limit to 1024.

The same problem can happen on 11.1.0.6 and 11.1.0.7

This ulimit problem can result in database or asm hangs (see Note 858279.1)

BUG:8537153 ASM & DB INSTANCE HUNG DUE TO ORA-240

(Base BUG:8429716 DATABASE HANGS AT 1900 USER LOAD)

ORA-16146: standby destination control file enqueue unavailable

ORA-00240: control file enqueue held for more than 120 seconds

Conclusion:

  • following installation pre-requirements not always enough!
  • it’s good practice to check init.crsd & init.cssd for ulimit -n after initial installation and after each CRS patch(set) application

My personal advise for CRS & ASM versions:

  • be at version 11.1.0.7 with latest(for now it’s #1) CRS Bundle Patch applied for both products
  • be at least at version 10.2.0.4 with at least CRS Bundle Patch #3 applied for both products
  • database may be at any version not greater then ASM

Later I’ll point to another issues that may be cause of ORA-00240 errors.

For previous post on this topic read ASM Performance with HUGE number of files.

Advertisements

4 thoughts on “Another reason for ORA-240 with ASM & RAC in place

  1. There is a bug in 11.2.0.1 that can cause the ORA-240 error:
    Bug 8621625: TB: ORA-00240: CONTROL FILE ENQUEUE HELD FOR MORE THAN 120 SECONDS

    It is fixed in 11.2.0.2.

  2. Hi Tom,

    Thanks for pointing on this undocumented issue!

    Actually, there are a lot of reasons for ORA-240 and not all of them are related to BUGs in Oracle software
    like this one: 1105825.1 HP Itanium – ORA-240 or process on ASM & Database hang

    Oleksander

  3. Hi
    Today faced off with this error in my production RAC 😦 10.2.0.5 (DB and CRS version). By the end of this year we will be migrating to 11.2.0.3.

    Regards,
    Pavel.

  4. Pingback: crsd.bin hangs on HP-UX results ASM hang and DB hang with ORA-240 « Oleksandr Denysenko's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s