Pages

Monday, June 18, 2018

Restoring database after corruption without RMAN Backup. RECOVER DATABASE ALLOW 1 CORRUPTION;

===============================
General
===============================
After electrical issues on site,  the server was abruptly rebooted, and when it came up, Oracle could not be started due to a corruption in one of datafiles.
There was no backups either, as the storage of the backup was now offline, due to a corruption.

===============================
Evidences
===============================
Errors in alert.log file /software/oracle/diag/rdbms/igt/igt/trace/igt_ora_14392.trc:
ORA-01172: recovery of thread 1 stuck at block 110573 of file 5
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN...

===============================
Bad attempt to bring Oracle up
===============================
sqlplus / as sysdba

SQL> SHUTDOWN IMMEDIATE;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.

SQL> STARTUP NOMOUNT;
ORACLE instance started.

Total System Global Area 1068937216 bytes
Fixed Size                  2166536 bytes
Variable Size             700449016 bytes
Database Buffers          360710144 bytes
Redo Buffers                5611520 bytes

SQL> ALTER DATABASE MOUNT;
Database altered.

RECOVER DATABASE;
SQL> RECOVER DATABASE;
ORA-00283: recovery session canceled due to errors
ORA-10877: error signaled in parallel recovery slave PR02
ORA-00600: internal error code, arguments: [3020], [5], [110573], [21082093],
[], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 5, block# 110573)
ORA-10564: tablespace IGT_INDEX
ORA-01110: data file 5: '/oracle_db/db1/

===============================
Solution
===============================
>SHUTDOWN IMMEDIATE;
ORA-01109: database not open

>STARTUP MOUNT;
ORACLE instance started.

Total System Global Area 1068937216 bytes
Fixed Size                  2166536 bytes
Variable Size             700449016 bytes
Database Buffers          360710144 bytes
Redo Buffers                5611520 bytes

>RECOVER DATABASE ALLOW 1 CORRUPTION;
Database open

>SHUTDOWN IMMEDIATE;

>STARTUP;

SELECT tablespace_name, segment_type, owner, segment_name
FROM DBA_SEGMENTS
WHERE file_id = 5
  AND block BETWEEN  110572 and 110574;

 Then drop and recreate the bad index.

Thursday, June 7, 2018

Mount entry in /etc/fstab by example

=============================
General
=============================
Oracle was failing to start because mount point for archive destination was missing

=============================
Error in alert.log
=============================
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance igt - Archival Error
ORA-16038: log 1 sequence# 160 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 1 thread 1: '/oracle_db/db1/db_igt/ora_redo_01_a.rdo'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance igt - Archival Error
ORA-16014: log 1 sequence# 160 not archived, no available destinations
ORA-00312: online log 1 thread 1: '/oracle_db/db1/db_igt/ora_redo_01_a.rdo'
Thu Jun 07 12:19:32 2018

ARC1: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle_db/db2/db_igt/archarch0001_160_827818122.arc' (error 19502) (igt)

=============================
Example how to mount
=============================
oracle@my_server:~>% df -hP
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/Volume00-LogVol00  2.0G  761M  1.1G  41% /
tmpfs                          4.0G  717M  3.4G  18% /dev/shm
/dev/vda1                      991M  623M  317M  67% /boot
/dev/mapper/Volume00-LogVol01  9.8G  2.1G  7.3G  22% /usr
/dev/mapper/Volume00-LogVol02   12G  312M   11G   3% /var
/dev/mapper/Volume00-LogVol03  2.0G  5.0M  1.9G   1% /tmp
/dev/mapper/Volume00-LogVol05  988M  1.5M  936M   1% /home
/dev/mapper/Volume00-Backup     27M  351K   25M   2% /backup
/dev/mapper/Volume00-LogVol04  5.8G  537M  5.0G  10% /opt
/dev/mapper/Volume00-LogVol10  4.4G  715M  3.4G  18% /kits
/dev/mapper/Volume00-LogVol12  3.9G  995M  2.7G  27% /software
/dev/mapper/Volume00-LogVol13   27M  351K   25M   2% /oracle_db
/dev/mapper/Volume00-LogVol14   16G  5.5G  9.4G  37% /software/oracle
/dev/mapper/Volume00-LogVol15  160G  4.2G  148G   3% /oracle_db/db1

oracle@my_server:~>% cat /etc/fstab
tmpfs                   /dev/shm         tmpfs   size=4g,nodev,nosuid,noexec             0 0
devpts                  /dev/pts         devpts  gid=5,mode=620  0 0
UUID=6168975d-c5ec-4a74-9a87-cfa073d510f7  
                            /boot        ext4   defaults      1 2
/dev/Volume00/LogVol00      /            ext4   defaults      1 1
/dev/Volume00/LogVol01      /usr         ext4   defaults      0 0
/dev/Volume00/LogVol02      /var         ext4   defaults      0 0
/dev/Volume00/LogVol03      /tmp         ext4   defaults      0 0
/dev/Volume00/LogVol05      /home        ext4   defaults      0 0
/dev/Volume00/LogVol06      swap         swap   defaults      0 0
/dev/OraVG01/OraLV01        /backup/ora_online ext4 defaults  0 0
/dev/OraVG01/OraLV02        /backup/ora_exp ext4  defaults    0 0
/dev/OraVG02/OraVol03       /oracle_db/db2  ext4  defaults    0 0
/dev/Volume00/Backup        /backup      ext4   defaults      0 0
/dev/Volume00/LogVol04      /opt         ext4   defaults      0 0
/dev/Volume00/LogVol10      /kits        ext4   defaults      0 0
/dev/Volume00/LogVol12      /software    ext4   defaults      0 0
/dev/Volume00/LogVol13      /oracle_db   ext4   defaults      0 0
/dev/Volume00/LogVol14      /software/oracle ext4  defaults   0 0
/dev/Volume00/LogVol15      /oracle_db/db1   ext4  defaults   0 0

root@my_server:~>% mount /dev/OraVG02/OraVol03

root@my_server:~>% df -hP
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/Volume00-LogVol00  2.0G  761M  1.1G  41% /
tmpfs                          4.0G  717M  3.4G  18% /dev/shm
/dev/vda1                      991M  623M  317M  67% /boot
/dev/mapper/Volume00-LogVol01  9.8G  2.1G  7.3G  22% /usr
/dev/mapper/Volume00-LogVol02   12G  312M   11G   3% /var
/dev/mapper/Volume00-LogVol03  2.0G  5.0M  1.9G   1% /tmp
/dev/mapper/Volume00-LogVol05  988M  1.5M  936M   1% /home
/dev/mapper/Volume00-Backup     27M  351K   25M   2% /backup
/dev/mapper/Volume00-LogVol04  5.8G  537M  5.0G  10% /opt
/dev/mapper/Volume00-LogVol10  4.4G  715M  3.4G  18% /kits
/dev/mapper/Volume00-LogVol12  3.9G  995M  2.7G  27% /software
/dev/mapper/Volume00-LogVol13   27M  351K   25M   2% /oracle_db
/dev/mapper/Volume00-LogVol14   16G  5.5G  9.4G  37% /software/oracle
/dev/mapper/Volume00-LogVol15  160G  4.2G  148G   3% /oracle_db/db1
/dev/mapper/OraVG02-OraVol03   345G  249M  327G   1% /oracle_db/db2



After restarting Oracle, issue was resolved.