crsd.bin hangs on HP-UX results ASM hang and DB hang with ORA-240

I had a lot of project with Oracle RAC on HP-UX based on Intel Itanuim platform.

We started with version 10.1.0.4, but quickly moved to 10.2.0.2 when it became available.

There were clusters with 2, 4 and 8 nodes and there weren’t stability problems with 2 or 4 nodes cluster, but those one – with 8 nodes … was the real nightmare…

At some point in time there were about 28 databases and ASM… was hanging…  every 2 days!!!

To temporary resolve this hanging issue customer had to shutdown all ASM instances at once and than start them – so it required FULL CUSTER OUTAGEContinue reading

Failed to read volume information for XXX, err 1005

Hi.

There are some discussions that mentioned error doesn’t allow to install Oracle RAC on Windows platform.

I have faced the same issue last week so I decided to make short post about it.

Excerpt from C:\Windows\Temp\exectask.log:

01/31/11 12:59:16 : (1896) : QueryDosDeviceAll: buffer was filled with 8922 bytes
...
01/31/11 12:59:16 : (1896) : Failed to read volume information for \Device\Harddisk2\Partition1, err 1005
01/31/11 12:59:16 : (1896) : The count = 8
01/31/11 12:59:16 : (1896) : The signature is 0x526f5d4d
01/31/11 12:59:16 : (1896) : Failed to read volume information for \Device\Harddisk3\Partition1, err 1005
01/31/11 12:59:16 : (1896) : The count = 8
01/31/11 12:59:16 : (1896) : The signature is 0xb893c528
01/31/11 12:59:16 : (1896) : Failed to read volume information for \Device\Harddisk4\Partition1, err 1005
01/31/11 12:59:16 : (1896) : The count = 8

Continue reading

ONS doesn’t start after applying 10.2.0.5

I have applied 10.2.0.5 patchset to one of my test clusters and after running $CRS_HOME/install/root102.sh I was quite surprised than ONS demon wasn’t started.

ОК.  It’s time to look deeper.

$ srvctl start nodeapps -n `hostname`
CRS-1006: No more members to consider
racnode1:ora.racnode1.ons:Remote port for local node in local config does not match that from OCR.
racnode1:ora.racnode1.ons:Number of configuration nodes retrieved: 1
racnode1:ora.racnode1.ons:0: {node = racnode1.domain, port = 4948}

racnode1:ora.racnode1.ons:ons is not running …
CRS-0215: Could not start resource ‘ora.racnode1.ons’.

So we see port 4948 was configured as ONS port in OCR. OCRDUMP displays the same information. Continue reading

Oracle Database PatchSet 10.2.0.5 is Available

Hi.

Quite good news for those who are going to stay with Oracle RDBMS release 10.2 for some more time.

Oracle RDBMS PatchSet version 10.2.0.5 is available now!

For now it’s available only for Linux x86-64, but we are expecting availability for most platforms during May 2010.

Patch number is 8202632.

Happy patching! 😉

PS.

It’s interesting fact that PatchSet 10.2.0.5 is already available(Apr 30, 2010), but no references for it in NOTE:316900.1 ALERT: Oracle 10g Release 2 (10.2) Support Status and Alerts…

PS2

NOTE:1088172.1 List of fixes included in 10.2.0.5

NOTE:1087991.1 Known issues and alerts affecting 10.2.0.5

Archiving HP-UX old /var/adm/syslog/OLDsyslog.log

It’s hard to troubleshoot constantly rebooting Oracle RAC nodes…

Especially if old system log are constantly overwritten.

To cope with this issue on HP-UX I wrote simple script that at system startup time archives old /var/adm/syslog/OLDsyslog.log file with compression. It makes possible to track back in time when issue appeared first.

To install this solution please create script /tmp/OLDsyslog.log.archiver with next contents and execute it as root:

. /tmp/OLDsyslog.log.archiver install Continue reading

Application VIP and RAC node LISTENER on 0.0.0.0

One interesting thread was started a half year ago on Russian forum SQL.RU.

It is worth to be mentioned here in English.

In general, topic-starter asked about possibility to add additional VIP into cluster configuration to overcome one of consolidation issue – moving database to cluster brings necessarily to modify connection information on many clients machines.

The right way is to use centralized names resolution – like OID,

but topic-starter decided to add to cluster another application VIP with IP-address of server where database was hosted before consolidation.

Nothing unusual, but to be able to made connection to this VIP we need new LISTENER which depends on this VIP and is able to run on any node of the cluster to overcome possible of node down. Once more – nothing unusual, just do job.

What was really interesting in this thread is this post by Alex Roudnev:

“Just modify cluster listeners to listen on address 0.0.0.0 and they will automatically listen on every VIP when it move to another node” Continue reading

DB doesn’t start after upgrade to 10.2.0.4 (ORA-07445 [krslvna()+5073])

It was quite interesting night.

I was patching six 10.2.0.3 RAC databases to 10.2.0.4 with some patches over it and I was quite surprised that 3 database of 6 didn’t want to start…

Quick solution was found after a glance look on this part of alert.log

LGWR: STARTING ARCH PROCESSES COMPLETE
LNSb started with pid=31, OS id=28175
Error 12514 received logging on to the standby
Tue Dec  8 23:41:16 2009
Errors in file /ora1/oracle/admin/DB/bdump/db3_lgwr_28013.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Tue Dec  8 23:41:16 2009
LGWR: Error 12514 verifying archivelog destination LOG_ARCHIVE_DEST_2
LGWR: Continuing...
Tue Dec  8 23:41:16 2009
Errors in file /ora1/oracle/admin/DB/bdump/db3_lgwr_28013.trc:
ORA-07445: exception encountered: core dump [krslvna()+5073] [SIGSEGV] [Invalid permissions for mapped object] [0x000000254] [] []
Tue Dec  8 23:41:17 2009
Trace dumping is performing id=[cdmp_20091208234117]
Tue Dec  8 23:41:19 2009
Errors in file /ora1/oracle/admin/DB/bdump/db3_pmon_27978.trc:
ORA-00470: LGWR process terminated with error
Tue Dec  8 23:41:19 2009
PMON: terminating instance due to error 470
Tue Dec  8 23:41:21 2009
Shutting down instance (abort)

Continue reading