Workaround for 10110863 Trace file with "kcbzib: dump suspect buffer" without any error.

================================
General
================================
From time to time Oracle generate a huge memory dump trace file.
The size can be up to several Gb in size.
The File is generated for no apparent reason, no errors are reported in alert.log
Sometimes the file is generated during expdp process.
Sometimes the file is generated during Oracle statistics gathering process.

Oracle version is 11.1.0.7
Seems that the installation is facing a known issue:

10110863 Trace file with "kcbzib: dump suspect buffer" without any error.

================================
Sample of error trace file
================================

Trace file /software/oracle/diag/rdbms/orainst/orainst/trace/orainst_j001_14123.trc

Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production

With the Partitioning, Oracle Label Security, OLAP, Data Mining,

Oracle Database Vault and Real Application Testing options

ORACLE_HOME = /software/oracle/111

System name:    Linux

Node name:      my_server

Release:        2.6.18-194.el5

Version:        #1 SMP Tue Mar 16 21:52:39 EDT 2010

Machine:        x86_64

Instance name: igt

Redo thread mounted by this instance: 1

Oracle process number: 49

Unix process pid: 14123, image: oracle@my_server (J001)

*** 2016-10-30 01:00:06.705

*** SESSION ID:(237.17778) 2016-10-30 01:00:06.705

*** CLIENT ID:() 2016-10-30 01:00:06.705

*** SERVICE NAME:(SYS$USERS) 2016-10-30 01:00:06.705

*** MODULE NAME:(DBMS_SCHEDULER) 2016-10-30 01:00:06.705

*** ACTION NAME:(ORA$AT_OS_OPT_SY_30756) 2016-10-30 01:00:06.705

kcbzib: dump suspect buffer

buffer tsn: 9 rdba: 0x02400e72 (9/3698)

scn: 0x0004.7078a3ba seq: 0x01 flg: 0x04 tail: 0xa3ba2301

frmt: 0x02 chkval: 0x5dbf type: 0x23=PAGETABLE SEGMENT HEADER

Hex dump of block: st=0, typ_found=1

Dump of memory from 0x00000000917B2000 to 0x00000000917B4000

================================
Oracle Tech note Doc ID 10110863.8
================================

Bug 10110863 - Trace file with "kcbzib: dump suspect buffer" without any error (Doc ID 10110863.8)

Description

Trace file with "kcbzib: dump suspect buffer" dump is produced

along with large buffer dump, without any user visible errors.

Rediscovery Notes:

1. Trace files with "kcbzib: dump suspect buffer" dump,

   along with large buffer dump, without any user visible errors.

2. ORA-8103, ORA-10632 & ORA-1410 raised by check functions internally

   and trapped before returning to user.

   This can be verified to catch error with events like following

 event="8103 errorstack(1)"

 event="10632 errorstack(1)"

Workaround

None but just deleting trace file with this dump is sufficient.

================================
Code
================================

This script searches for trc files with string "kcbzib: dump suspect buffer" and delete these files.

Each deleted file is logged to  a log file.

There is a crontab task that activates this script on a daily basis.

#!/bin/bash
SEARCH_PATH=/software/oracle/diag/rdbms/igt/igt/trace
FILES="$(ls $SEARCH_PATH/igt_*.trc | xargs grep -l "kcbzib: dump suspect buffer" )"
LOG_FILE=/software/oracle/oracle/scripts/delete_memory_dump_trc.log
RUN_DATE="$(date +"%Y%m%d"_"%H%M%S")"

touch $LOG_FILE
echo "Starting delete_memory_dump_trc.sh at $RUN_DATE" >> $LOG_FILE
for f in $FILES
do
  echo  "handling file $f" >> $LOG_FILE
  rm $f
  echo "File was Deleted" >> $LOG_FILE
done
echo "Finished delete_memory_dump_trc.sh" >> $LOG_FILE
echo "==================================" >> $LOG_FILE
echo " " >> $LOG_FILE

crontab entry
0 4 * * * /software/oracle/oracle/scripts/delete_memory_dump_files.sh

================================
Sample of output logfile
================================

Starting delete_memory_dump_trc.sh at 20161030_124203
handling file /software/oracle/diag/rdbms/orainst/orainst/trace/orainst_j001_14123.trc
File was Deleted
Finished delete_memory_dump_trc.sh
==================================

======================================
Simple version to delete old trace files from Oracle
======================================
#!/bin/bash
export ORA_INST=orainst
export DAYS_TO_KEEP=10

#Delete files older than $DAYS_TO_KEEP
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" -mtime +${DAYS_TO_KEEP} -exec rm {} \;
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trm" -mtime +${DAYS_TO_KEEP} -exec rm {} \;

#Delete files greater than 1Gb
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" -size +1G -exec rm {} \;

#Delete files with "kcbzib: dump suspect buffer"
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" | xargs grep -l "kcbzib: dump suspect buffer" | xargs rm -f