DB doesn’t start after upgrade to 10.2.0.4 (ORA-07445 [krslvna()+5073])

It was quite interesting night.

I was patching six 10.2.0.3 RAC databases to 10.2.0.4 with some patches over it and I was quite surprised that 3 database of 6 didn’t want to start…

Quick solution was found after a glance look on this part of alert.log

LGWR: STARTING ARCH PROCESSES COMPLETE
LNSb started with pid=31, OS id=28175
Error 12514 received logging on to the standby
Tue Dec  8 23:41:16 2009
Errors in file /ora1/oracle/admin/DB/bdump/db3_lgwr_28013.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Tue Dec  8 23:41:16 2009
LGWR: Error 12514 verifying archivelog destination LOG_ARCHIVE_DEST_2
LGWR: Continuing...
Tue Dec  8 23:41:16 2009
Errors in file /ora1/oracle/admin/DB/bdump/db3_lgwr_28013.trc:
ORA-07445: exception encountered: core dump [krslvna()+5073] [SIGSEGV] [Invalid permissions for mapped object] [0x000000254] [] []
Tue Dec  8 23:41:17 2009
Trace dumping is performing id=[cdmp_20091208234117]
Tue Dec  8 23:41:19 2009
Errors in file /ora1/oracle/admin/DB/bdump/db3_pmon_27978.trc:
ORA-00470: LGWR process terminated with error
Tue Dec  8 23:41:19 2009
PMON: terminating instance due to error 470
Tue Dec  8 23:41:21 2009
Shutting down instance (abort)


So quick solution was to disable archiving to standby site by executing:

STARTUP NOMOUNT;
ALTER SYSTEM SET log_archive_dest_state_2=DEFER;
SHUTDOWN IMMEDIATE;
STARTUP

After finishing scheduled work I decided to look for what it was.

And it was: BUG:6490140 – LGWR may crash the instance

The LGWR process can terminate unexpectedly crashing the instance.

Conditions to face this BUG:

  • standby with MAXIMUM PERFORMANCE: LGWR ASYNC for sending changes
  • down standby when starting primary or
  • shutdown standby when primary is sending changes
  • database is on HP-UX Itanium, but it may be generic
  • it looks like that probability if this BUG in RAC is higher
  • some more facts that may be reasons for BUG to (NOT) appear
  • it’s not consistent

Some more interesting BUGs which are result of mentioned BUG:6490140:

  • BUG:7260814 CANNOT STARTUP PRIMARY INSTANCES DUE TO ORA-07445[KRSLVNA] WHEN STANDBY INSTANCE
  • BUG:7156967 LGWR ON PRIMARY DIES WITH ORA-7445 [KRSLVNA] IF STANDBY INSTANCE IS SHUTDOWN
  • BUG:8761245 ORA-7445 [KRSLVNA()+5073] IN LGWR WITH ASYNC TRANSPORT WHEN STANDBY IS SHUTDOWN

Conclusion:

  • If You are using 10.2.0.3-4 with standby in MAXIMUM PERFORMANCE MODE(LGWR ASYNC) that it will be a good idea to install patch containing fix for BUG:6490140
  • disabling redo shipment to standby may be used as a temporary workaround(before startup primary or shutdown standby)
  • this bug was fixed in: 10.2.0.4.1, 10.2.0.5(not released as of 14-DEC-2009), 11.1.0.6
  • As for me it’s one more reason to apply PATCH: 8833280 10.2.0.4.2 Patch Set Update (PSU 2)
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s