Pages

Thursday, August 17, 2017

Many trc files with message "KGX cleanup..." filling up disk space to 100%

=========================
General
=========================
Oracle version is 11.1.0.7
In alert.log there are no error messages.
But the trace folder is filled up with many trc files, having same text at the end of trc file:

KGX cleanup...
KGX Atomic Operation Log 0x2570f7e70
 Mutex 0x25ba10558(207, 0) idn 2b2d96fb6306 oper EXCL
 Library Cache uid 207 efd 21 whr 102 slp 0
 oper=0 pt1=(nil) pt2=(nil) pt3=(nil)
 pt4=(nil) pt5=(nil) ub4=0


=========================
Evidences
=========================

ls /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace/*.trc | xargs grep -l "KGX cleanup" | wc -l
9207

These file are generated frequently, with no apparent pattern:
-rw-r----- 1 oracle dba 2564096 Aug 17 15:45 igt_ora_7505.trc
-rw-r----- 1 oracle dba 1286144 Aug 17 15:50 igt_ora_4501.trc
-rw-r----- 1 oracle dba 1282048 Aug 17 15:50 igt_ora_5670.trc
-rw-r----- 1 oracle dba 1282048 Aug 17 15:52 igt_ora_7665.trc
-rw-r----- 1 oracle dba 1282048 Aug 17 16:01 igt_ora_9603.trc
-rw-r----- 1 oracle dba 2564096 Aug 17 16:05 igt_ora_465.trc
-rw-r----- 1 oracle dba 1280197 Aug 17 16:15 igt_ora_9351.trc
-rw-r----- 1 oracle dba 1280226 Aug 17 16:16 igt_ora_5427.trc
-rw-r----- 1 oracle dba 1280196 Aug 17 16:16 igt_ora_5794.trc
-rw-r----- 1 oracle dba 1280196 Aug 17 16:17 igt_ora_9574.trc
-rw-r----- 1 oracle dba 2560452 Aug 17 16:17 igt_ora_9579.trc
-rw-r----- 1 oracle dba 2560424 Aug 17 16:18 igt_ora_9848.trc
-rw-r----- 1 oracle dba 1280226 Aug 17 16:23 igt_ora_6918.trc
-rw-r----- 1 oracle dba 2560392 Aug 17 16:25 igt_ora_9280.trc
-rw-r----- 1 oracle dba 2560423 Aug 17 16:26 igt_ora_9282.trc

=========================
Oracle Documentation
=========================
Per oracle technote:

 Bug 6638558 - KGX cleanup trace files produced (Doc ID 6638558.8)

bug 6638558  KGX cleanup trace files produced
Range of versions believed to be affected Versions BELOW 11.2
This issue is fixed in 11.2.0.1 (Base Release)

KGX cleanup trace files containing shared examine mutexes
may be generated when a parse-related error occurs.
Trace files may be generated containing text of the form:
  KGX cleanup...
  KGX Atomic Operation Log c00000010de2cf90
   Mutex c0000000c0c17290(1006, 0) idn 0 oper EXAM
   Cursor Parent uid 1006 efd 5 whr 26 slp 0
   oper=DEFAULT pt1=c00000009b720b00 pt2=c00000009b720ce8 pt3=c00000009b721208
   pt4=0000000000000000 u41=0 stt=0
  *** 2007-11-14 16:19:19.645
  KGX cleanup...
  KGX Atomic Operation Log c00000010de2cf90
   Mutex c0000000c0c17290(1006, 0) idn 0 oper EXAM
   Cursor Parent uid 1006 efd 5 whr 26 slp 0
   oper=DEFAULT pt1=c000000072c5fe30 pt2=c000000072c5fe98 pt3=0000000000000000
   pt4=0000000000000000 u41=0 stt=0
Workaround
  Correct the cause of any parse-related error.
 or
  Ignore the trace entries as these are due to the code unnecessarily 
  logging its cleanup operation.


=========================
Solution
=========================
Create a crontab task to execute script, that will delete *.trc files with text "KGX cleanup"

The script to delete trace files:

less delete_trace_files.sh 
#!/bin/bash

export ORA_INST=igt
BASENAME=`basename $0`
LOG_FILE=`echo $BASENAME | sed s/.sh/.log/`
HOME_DIR=/software/oracle/oracle/scripts
echo Log File: ${LOG_FILE}
cd $HOME_DIR

./write_log.sh $0
echo "About to Delete these Files:" >> $LOG_FILE

ls /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace/*.trc | xargs grep -l "KGX cleanup" | xargs ls -ltr >> ./${LOG_FILE}
ls /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace/*.trc | xargs grep -l "KGX cleanup" | xargs rm -f

less write_log.sh 
#!/bin/bash

PROGRAM_NAME=$1
WORKDIR=/software/oracle/oracle/scripts
cd ${WORKDIR}
RUN_DATE=`date +"%Y%m%d_%H%M%S"`
DELIMITER="============================================="
BASENAME=`basename $PROGRAM_NAME`
LOG_FILE=`echo $BASENAME | sed s/.sh/.log/`

touch $LOG_FILE
echo $DELIMITER >> $LOG_FILE
echo "Running $PROGRAM_NAME at $RUN_DATE" >>  $LOG_FILE
echo $DELIMITER >> $LOG_FILE


exit 0


The crontab entry
2 * * * * /software/oracle/oracle/scripts/delete_kgx_cleanup_trace_files.sh

No comments:

Post a Comment