General
================================
From time to time Oracle generate a huge memory dump trace file.
The size can be up to several Gb in size.
The File is generated for no apparent reason, no errors are reported in alert.log
Sometimes the file is generated during expdp process.
Sometimes the file is generated during Oracle statistics gathering process.
Oracle version is 11.1.0.7
Seems that the installation is facing a known issue:
10110863 Trace file with "kcbzib: dump suspect buffer" without any error.
================================
Sample of error trace file
================================
Trace file /software/oracle/diag/rdbms/orainst/orainst/trace/orainst_j001_14123.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining,
Oracle Database Vault and Real Application Testing options
ORACLE_HOME = /software/oracle/111
System name: Linux
Node name: my_server
Release: 2.6.18-194.el5
Version: #1 SMP Tue Mar 16 21:52:39 EDT 2010
Machine: x86_64
Instance name: igt
Redo thread mounted by this instance: 1
Oracle process number: 49
Unix process pid: 14123, image: oracle@my_server (J001)
*** 2016-10-30 01:00:06.705
*** SESSION ID:(237.17778) 2016-10-30 01:00:06.705
*** CLIENT ID:() 2016-10-30 01:00:06.705
*** SERVICE NAME:(SYS$USERS) 2016-10-30 01:00:06.705
*** MODULE NAME:(DBMS_SCHEDULER) 2016-10-30 01:00:06.705
*** ACTION NAME:(ORA$AT_OS_OPT_SY_30756) 2016-10-30 01:00:06.705
kcbzib: dump suspect buffer
buffer tsn: 9 rdba: 0x02400e72 (9/3698)
scn: 0x0004.7078a3ba seq: 0x01 flg: 0x04 tail: 0xa3ba2301
frmt: 0x02 chkval: 0x5dbf type: 0x23=PAGETABLE SEGMENT HEADER
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x00000000917B2000 to 0x00000000917B4000
================================
Oracle Tech note Doc ID 10110863.8
================================
Bug 10110863 - Trace file with "kcbzib: dump suspect buffer" without any error (Doc ID 10110863.8)
Description
Trace file with "kcbzib: dump suspect buffer" dump is produced
along with large buffer dump, without any user visible errors.
Rediscovery Notes:
1. Trace files with "kcbzib: dump suspect buffer" dump,
along with large buffer dump, without any user visible errors.
2. ORA-8103, ORA-10632 & ORA-1410 raised by check functions internally
and trapped before returning to user.
This can be verified to catch error with events like following
event="8103 errorstack(1)"
event="10632 errorstack(1)"
Workaround
None but just deleting trace file with this dump is sufficient.
Code
================================
This script searches for trc files with string "kcbzib: dump suspect buffer" and delete these files.
Each deleted file is logged to a log file.
There is a crontab task that activates this script on a daily basis.
SEARCH_PATH=/software/oracle/diag/rdbms/igt/igt/trace
FILES="$(ls $SEARCH_PATH/igt_*.trc | xargs grep -l "kcbzib: dump suspect buffer" )"
LOG_FILE=/software/oracle/oracle/scripts/delete_memory_dump_trc.log
RUN_DATE="$(date +"%Y%m%d"_"%H%M%S")"
touch $LOG_FILE
echo "Starting delete_memory_dump_trc.sh at $RUN_DATE" >> $LOG_FILE
for f in $FILES
do
echo "handling file $f" >> $LOG_FILE
rm $f
echo "File was Deleted" >> $LOG_FILE
done
echo "Finished delete_memory_dump_trc.sh" >> $LOG_FILE
echo "==================================" >> $LOG_FILE
echo " " >> $LOG_FILE
crontab entry
0 4 * * * /software/oracle/oracle/scripts/delete_memory_dump_files.sh
================================
Sample of output logfile
================================
Starting delete_memory_dump_trc.sh at 20161030_124203
handling file /software/oracle/diag/rdbms/orainst/orainst/trace/orainst_j001_14123.trc
File was Deleted
Finished delete_memory_dump_trc.sh
==================================
======================================
Simple version to delete old trace files from Oracle
======================================
#!/bin/bash
export ORA_INST=orainst
export DAYS_TO_KEEP=10
#Delete files older than $DAYS_TO_KEEP
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" -mtime +${DAYS_TO_KEEP} -exec rm {} \;
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trm" -mtime +${DAYS_TO_KEEP} -exec rm {} \;
#Delete files greater than 1Gb
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" -size +1G -exec rm {} \;
#Delete files with "kcbzib: dump suspect buffer"
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" | xargs grep -l "kcbzib: dump suspect buffer" | xargs rm -f
