IBM PowerHA(HACMP) shutdowns oracle 11.2 because of IPv6 and DNS is not accessable

Hi.

Just a quick post with some additions to situation covered in

Investigation of the 11g Oracle*Net connection delay when DNS server is unreachable.

Just a short recap:

  • customer experienced long timeout(slow connection) doing tnsping to 11.2 database running on AIX when DNS server was unreachable;

” tnsping ‘ (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dns-hostname)(PORT = 1521))\ (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XXX)))’

TNS Ping Utility for IBM/AIX RISC System/6000: Version 11.2.0.3.0 – Production

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dns-hostname)(PORT = 1521))   (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XXX)))

OK (150010 msec)

  • issue was because of IPv6 support was included in Oracle 11g and even if connection hostname was resolved by /etc/hosts to IPv4 client tried to resolve it to IPv6 address too.
  • issue was fixed by replacing local, bind by local4, bind4  in the /etc/netsvc.conf.

So why I’m doing this recap ?

It’s because of mentioned problem was only a part of issue faced during IBM PowerHA(HACMP) cluster with oracle resources testing.

In short:

  1. two nodes cluster IBM PowerHA(HACMP)
  2. oracle database software was installed on both nodes
  3. single-instance(no RAC) database was created
  4. IBM PowerHA(HACMP) is used as a fail-over solution
  5. oracle listener and instance were configured as cluster resources
  6. everything was working as expected, including services fail-over to another node in a case of node failure
  7. during one of the test DNS server became inaccessible
  8. IBM PowerHA(HACMP) tried to check oracle listener, but wasn’t able to do it because of tnsping responded only after 150 seconds, but cluster resource timeout was lower than 150 seconds
  9. IBM PowerHA(HACMP) stopped all cluster resource including oracle database, listener, shared disk resource, etc and failed over to second node
  10. at the second node situation was the same, so IBM PowerHA(HACMP) wasn’t able to check cluster resources after startup and decided to stop them all too.
  11. no fail-over to the first node
  12. cluster was completely down because of DNS issue when dns host name wasn’t resolved to IPv6 locally

That’s complete small story about the whole IBM PowerHA(HACMP) cluster unavailability because of IPv6 support included in Oracle 11g and DNS issue.

Oleksandr

PS: I remember at least one more issue similar to this – when cluster manager tried to check cluster resource and wasn’t able to do it because of some issue. It decided that resource was dead and was just killing production oracle database.

Advertisements

6 thoughts on “IBM PowerHA(HACMP) shutdowns oracle 11.2 because of IPv6 and DNS is not accessable

  1. Hi,

    I would appreciate if you could share any documents related to Oracle 11gR2 installation on AIX HACMP, listener and instance configuration on cluster resources. Please mail me on muddu83@gmail.com

    Regards
    Mohammed

  2. Thanks, Oleksandr! Yes, we would be just having the db resource instance. Sorry, but I did not understand what did you mean by ‘use free of charge option’.

    Mohammed

    • Mohammed,

      You don’t have to pay for Oracle Grid Infrastructure as long as You have Oracle RDBMS licences.
      As I understand, You have to pay for using HACMP(PowerHA)

      Oleksandr

      • Hi Oleksandr,

        Could you please guide me how do I install and configure Single Instance Oracle DB on powerHA? Do I have to create separate instances on both the AIX nodes?

  3. Mohammed,
    Have You read mentioned “Configuring Oracle in an HACMP cluster” ?
    there are topics on configuring Oracle Instance
    there many others documents on Internet dedicated to this topic.
    Start with http://www.ibm.com or try to find somebody near You how will be able
    to implement desired configuration.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s