Pages

Sunday, March 4, 2018

ORA-08103: object no longer exists. Investigation and resolution.

=======================================
General
=======================================
Large Oracle trace file (~600 Mb) are generated.
What could be the cause?

=======================================
Evidences
=======================================
In alert.log there is only this non informative error message:
Stopping background process CJQ0

Checking the trace files timestamp, they all were generated at apprximately same time each day, at ~23:51.

In all trace files, there is same error message: "dump suspect buffer"
But the trace file has also session detalis.

-rw-r----- 1 oracle dba 650,767,781 Feb  3 23:52
/software/oracle/diag/rdbms/igt/igt/trace/igt_j000_16517.trc

Trace file /software/oracle/diag/rdbms/igt/igt/trace/igt_j000_16517.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining,
Oracle Database Vault and Real Application Testing options
ORACLE_HOME = /software/oracle/111
System name:    Linux
Node name:      arg-cti-2-aps-1
Release:        2.6.18-274.el5
Version:        #1 SMP Fri Jul 8 17:36:59 EDT 2011
Machine:        x86_64
Instance name: igt
Redo thread mounted by this instance: 1
Oracle process number: 175
Unix process pid: 16517, image: oracle@arg-cti-2-aps-1 (J000)


*** 2018-02-03 23:50:24.551
*** SESSION ID:(164.47150) 2018-02-03 23:50:24.551
*** CLIENT ID:() 2018-02-03 23:50:24.551
*** SERVICE NAME:(SYS$USERS) 2018-02-03 23:50:24.551
*** MODULE NAME:() 2018-02-03 23:50:24.551
*** ACTION NAME:() 2018-02-03 23:50:24.551

kcbzib: dump suspect buffer
buffer tsn: 4 rdba: 0x01007af2 (4/31474)
scn: 0x0dac.ef6f7cc4 seq: 0x01 flg: 0x04 tail: 0x7cc42301
frmt: 0x02 chkval: 0xe4b3 type: 0x23=PAGETABLE SEGMENT HEADER
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x0000000082AA8000 to 0x0000000082AAA000

=======================================
Investigation
=======================================
Checking DBA_HIST tables by session_id.

SELECT  DBA_HIST_ACTIVE_SESS_HISTORY.* 
FROM DBA_HIST_ACTIVE_SESS_HISTORY 
WHERE session_id = '164' AND session_serial# = 47150;

user_id = 49
sql_id  cnsp07td49muk

SELECT * FROM SBA_HIST_SQLTEXT 
 WHERE sql_id = 'cnsp07td49muk';

SELECT * FROM DBA_USERS WHERE user_id = 49;
--MY_USER

SELECT * FROM DBA_JOBS WHERE schema_user = 'MY_USER'

Indeed there is a job, which is scheduled to run at 23:50 Daily.

Checking that job code, there is the SQL which sql_id is referenced by 'cnsp07td49muk';

Checking the application log table for that job, there this error message:

Error during procedure: ORA-08103: object no longer exists

=======================================
Resolution
=======================================
Following the error message ORA-08103: object no longer exists, additional jobs which where running at same time were investigated.

Apparently, there was another job, performing a purge of old data from same table, by ALTER TABLE XXX TRUNCATE PARTITION YYY;

This was the cause for error ORA-08103: object no longer exists error.

Issue was resolved, by rescheduling the failing job, so it would not run at he same time with the purge data job.

No comments:

Post a Comment