ONS doesn’t start after applying 10.2.0.5

I have applied 10.2.0.5 patchset to one of my test clusters and after running $CRS_HOME/install/root102.sh I was quite surprised than ONS demon wasn’t started.

ОК.  It’s time to look deeper.

$ srvctl start nodeapps -n `hostname`
CRS-1006: No more members to consider
racnode1:ora.racnode1.ons:Remote port for local node in local config does not match that from OCR.
racnode1:ora.racnode1.ons:Number of configuration nodes retrieved: 1
racnode1:ora.racnode1.ons:0: {node = racnode1.domain, port = 4948}

racnode1:ora.racnode1.ons:ons is not running …
CRS-0215: Could not start resource ‘ora.racnode1.ons’.

So we see port 4948 was configured as ONS port in OCR. OCRDUMP displays the same information.

Quite interesting message “Remote port for local node in local config does not match that from OCR” was produced by $CRS_HOME/opmn/bin/ons and previous version of this file didn’t contain this message -> some new behavior/check was introduced in 10.2.0.5 .

Let’s look in local ONS configuration file $CRS_HOME/opmn/conf/ons.config

localport=6100
remoteport=6200
loglevel=3
useocr=on

So we have 2 different ports configured for ONS in local config file(6200) and in OCR(4948).

The main cause of described issue is not the fact that local and OCR configuration mismatched, but the presence in local configuration file additional parameter useocr=on. So ONS have to use configuration from OCR and claims that it mismatches with local configuration…

The funny thing is that everything was OK before patching – mismatch wasn’t an issue…

I still don’t understand why not fully use only OCR configuration settings for cluster-wide ONS (CRS_HOME) and use local configuration only for debug purpose ?

Conclution:

  1. Patchset applications have to be done in test environment first to minimize possible issues for production system
  2. Described situation may be quite frequent because of:
    • I have seen a lot of configuration with mismatched settings in OCR, ons.config and with useocr=on in place
    • 6739989: INCORRECT ONS REMOTEPORT IS ADDED TO OCR WHEN WE ADD NEW NODE TO CLUSTER
    • 5948848: ONS.CONFIG SETTINGS RESET AFTER APPLICATION OF THE 10.2.0.3 PATCHSET
  3. for more information about ONS Deamon read next My Oracle Support document

759895.1 The ONS Daemon Explained In RAC/CRS environment

Advertisements

3 thoughts on “ONS doesn’t start after applying 10.2.0.5

  1. Hi,

    We applied Patchset 5 on our Oracle 10g R2 RAC on HP-UX and got this issue. Is there a fix for this issue?

    Rgds,
    CSK

    • Hi CSK,

      In my post I have provided all required information, but without directly giving solution.

      In short You have to change your ONS configuration in some way: local or OCR.
      I have changed local configuration and it works perfectly.

      Oleksandr

  2. Changing the local ons.conf setting worked for me. Had to find the solution from here and not from Oracle support though!
    Thanks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s