Scenario.
=======================
Following several ORA-600 coredumps, scheduled backup process were in failed/stuck status, which resulted in DB being hand for any DML operation.
The reson for ORA-600 is out of scope for this post.
Simple SHUTDOWN IMMEDIATE command could not be completed, because Oracle could not either commit or rollback transactions.
The solution in this case is to issue SHUTDOWN ABORT command.
Upon STARTUP, there is an ORA-10873: file 1 needs to be either taken out of backup mode or media recovered
This issue is addressed by forcing backup completion by:
ALTER DATABASE END BACKUP;
From trace file igt_ora_10009.trc
=============================================================Dump continued from file: /software/oracle/diag/rdbms/igt/igt/trace/igt_ora_10009.trc
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4153], [16], [], [], [], [], [], [], [], [], [], []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programm
========= Dump for incident 88824 (ORA 603) ========
*** 2016-03-27 01:00:17.944
----- Error Stack Dump -----
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4153], [16], [], [], [], [], [], [], [], [], [], []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kcocdm()+37] [SIGSEGV] [ADDR:0x0] [PC:0x7B9B88F] [SI_KERNEL(general_protection)] []
----- SQL Statement (None) -----
Current SQL information unavailable - no SGA.
Dump continued from file: /software/oracle/diag/rdbms/igt/igt/trace/igt_ora_10009.trc
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4153], [16], [], [], [], [], [], [], [], [], [], []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programm
========= Dump for incident 88824 (ORA 603) ========
*** 2016-03-27 01:00:17.944
----- Error Stack Dump -----
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4153], [16], [], [], [], [], [], [], [], [], [], []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kcocdm()+37] [SIGSEGV] [ADDR:0x0] [PC:0x7B9B88F] [SI_KERNEL(general_protection)] []
----- SQL Statement (None) -----
Current SQL information unavailable - no SGA.
SQL> SHUTDOWN IMMEDIATE;
The database just hangs.
SQL> ALTER SYSTEM CHECKPOINT;
The database just hangs.
SQL> SHUTDOWN ABORT;
ORACLE instance shut down.
SQL> STARTUP RESTRICT;
ORACLE instance started.
Total System Global Area 8551575552 bytes
Fixed Size 2161400 bytes
Variable Size 4093641992 bytes
Database Buffers 4429185024 bytes
Redo Buffers 26587136 bytes
Database mounted.
ORA-10873: file 1 needs to be either taken out of backup mode or media
recovered
ORA-01110: data file 1: '/oracle_db/db1/db_igt/ora_system_01.dbf'
SQL> STARTUP NOMOUNT;
ORACLE instance started.
SQL> ALTER DATABASE OPEN;
ALTER DATABASE OPEN
*
ERROR at line 1:
ORA-10873: file 1 needs to be either taken out of backup mode or media
recovered
ORA-01110: data file 1: '/oracle_db/db1/db_igt/ora_system_01.dbf'
SQL> SELECT * FROM V$BACKUP;
FILE# STATUS CHANGE# TIME
---------- ------------------ ---------- ---------
1 ACTIVE 4119977880 27-MAR-16
2 ACTIVE 4119977880 27-MAR-16
3 ACTIVE 4119977880 27-MAR-16
4 ACTIVE 4119977880 27-MAR-16
5 ACTIVE 4119977880 27-MAR-16
6 ACTIVE 4119977880 27-MAR-16
7 ACTIVE 4119977880 27-MAR-16
8 ACTIVE 4119977880 27-MAR-16
9 ACTIVE 4119977880 27-MAR-16
10 ACTIVE 4119977880 27-MAR-16
11 ACTIVE 4119977880 27-MAR-16
12 ACTIVE 4119977880 27-MAR-16
13 ACTIVE 4119977880 27-MAR-16
14 ACTIVE 4119977880 27-MAR-16
15 ACTIVE 4119977880 27-MAR-16
16 ACTIVE 4119977880 27-MAR-16
17 ACTIVE 4119977880 27-MAR-16
18 ACTIVE 4119977880 27-MAR-16
19 ACTIVE 4119977880 27-MAR-16
20 ACTIVE 4119977880 27-MAR-16
21 ACTIVE 4119977880 27-MAR-16
21 rows selected.
SQL> ALTER DATABASE END BACKUP;
Database altered.
SQL> SELECT * FROM V$BACKUP;
FILE# STATUS CHANGE# TIME
---------- ------------------ ---------- ---------
1 NOT ACTIVE 4119977880 27-MAR-16
2 NOT ACTIVE 4119977880 27-MAR-16
3 NOT ACTIVE 4119977880 27-MAR-16
4 NOT ACTIVE 4119977880 27-MAR-16
5 NOT ACTIVE 4119977880 27-MAR-16
6 NOT ACTIVE 4119977880 27-MAR-16
7 NOT ACTIVE 4119977880 27-MAR-16
8 NOT ACTIVE 4119977880 27-MAR-16
9 NOT ACTIVE 4119977880 27-MAR-16
10 NOT ACTIVE 4119977880 27-MAR-16
11 NOT ACTIVE 4119977880 27-MAR-16
12 NOT ACTIVE 4119977880 27-MAR-16
13 NOT ACTIVE 4119977880 27-MAR-16
14 NOT ACTIVE 4119977880 27-MAR-16
15 NOT ACTIVE 4119977880 27-MAR-16
16 NOT ACTIVE 4119977880 27-MAR-16
17 NOT ACTIVE 4119977880 27-MAR-16
18 NOT ACTIVE 4119977880 27-MAR-16
19 NOT ACTIVE 4119977880 27-MAR-16
20 NOT ACTIVE 4119977880 27-MAR-16
21 NOT ACTIVE 4119977880 27-MAR-16
21 rows selected.
SQL>
SQL> ALTER DATABASE OPEN;
Database altered.
No comments:
Post a Comment