Linux file processing examples

==================================
Index
==================================

1. Loop on files

2. Sort file by multiple fields
3. Delete files by name and text inside files

==================================
1. Loop on files
==================================
#!/bin/bash
export WORK_DIR=//cygdrive/d/Functionality/my_dir
export FILES=$WORK_DIR/Xfer_Files/*.old
export OUTPUT_FILE=WORK_DIR/Xfer_Files/output_file.txt

rm $OUTPUT_FILE
touch $OUTPUT_FILE

for f in $FILES
do
  echo "Processing $f file..."
  # take action on each file. $f store current file name
  cat $f >>$OUTPUT_FILE
done

==================================

2. Sort file by multiple fields

==================================

Need to sort by date+time fields

Input File:

D;0950002684082;101;00161418707101;61421459499;505013474093338;20150801;08:17:58;00:00:33;0;

D;0950002684055;321;00161411000321;61412990538;505021412990538;20150801;05:10:39;00:01:05;0;

D;0950002684088;901;001447802010074;447733334274;234106713428743;20150801;08:41:33;00:00:02;0;

D;0950002684101;0097455559807;00197455559807;97455066688;427012920203344;20150801;11:30:52;00:00:11;0;

D;0950002684043;0324668081;00182324668081;821031899259;450088290030694;20150801;01:18:11;00:01:11;0;

D;0950002684103;0097455558907;00197455558907;97455066688;427012920203344;20150801;11:32:12;00:01:21;0;

command

sort -t ';' -k7,1 -k8,2   output_file.txt

-t ';' : - Use fields separator ';' instead of default space.

-k7,1    - Field No 7 - use as first field in sort.

-k8,2    - Field No 8 - use as second field in sort.

Output File:

D;0950002684043;0324668081;00182324668081;821031899259;450088290030694;20150801;01:18:11;00:01:11;0;

D;0950002684055;321;00161411000321;61412990538;505021412990538;20150801;05:10:39;00:01:05;0;

D;0950002684082;101;00161418707101;61421459499;505013474093338;20150801;08:17:58;00:00:33;0;

D;0950002684088;901;001447802010074;447733334274;234106713428743;20150801;08:41:33;00:00:02;0;

D;0950002684101;0097455559807;00197455559807;97455066688;427012920203344;20150801;11:30:52;00:00:11;0;

D;0950002684103;0097455558907;00197455558907;97455066688;427012920203344;20150801;11:32:12;00:01:21;0;

==================================

3. Delete files by name and text inside files

==================================
Example A.

General - Oracle generates large memory dump files which fill up the disk.
These file can be identified by a string "kcbzib: dump suspect buffer"

This issue seems to be a known Oracle issue: 10110863 Trace file with "kcbzib: dump suspect buffer" without any error.

In this case, the files were generated randomly by nightly expdp process, and generated trc files name was orainst_dw0*.trc

#!/bin/bash

SEARCH_PATH=/software/oracle/diag/rdbms/orainst/orainst/trace

FILES="$(ls $SEARCH_PATH/orainst_dw0*.trc | xargs grep -l "kcbzib: dump suspect buffer" )"

LOG_FILE=/software/oracle/oracle/scripts/delete_memory_dump_trc.log

RUN_DATE="$(date +"%Y%m%d"_"%H%M%S")"

touch $LOG_FILE

echo "Starting delete_memory_dump_trc.sh at $RUN_DATE" >> $LOG_FILE

for f in $FILES

do

  echo  "handling file $f" >> $LOG_FILE

  rm $f

  echo "File was Deleted" >> $LOG_FILE

done

echo "Finished delete_memory_dump_trc.sh" >> $LOG_FILE

echo "==================================" >> $LOG_FILE

echo " " >> $LOG_FILE

Example B.

#!/bin/bash

#Delete deadlock trace files

LOG_FILE=/software/oracle/oracle/scripts/delete_deadlock_trace.log

WORK_DIR=/software/oracle/diag/rdbms/igt/igt/trace

RUN_DATE=`date +"%Y%m%d_%H%M%S"` 

touch $LOG_FILE

echo "===================================" >> $LOG_FILE

echo "Run Date: $RUN_DATE" >>  $LOG_FILE

echo "===================================" >>  $LOG_FILE

FILES=`find /software/oracle/diag/rdbms/igt/igt/trace/*.trc | xargs grep -l ORA-00060`

for file in $FILES 

do

  echo "Found deadlock ORA-00060 in File $file " >>  $LOG_FILE

  ls -ltr $file >> $LOG_FILE

  echo "deleting File..... $file " >>  $LOG_FILE

  rm -f $file 

  echo "Done" >> $LOG_FILE

  echo >> $LOG_FILE  

done

==================================

4. Loop on files and scp files one by one

==================================

#!/bin/bash

WORK_DIR=/software/oracle/oracle/scripts
FILE_SUCCESS=Export_Outbound_Roamers_*_SUCCESS
FILES=${WORK_DIR}/${FILE_SUCCESS}
REPORT_FILE=''

cd ${WORK_DIR}

for f in $FILES
do
  echo "Processing $f file..."
  REPORT_FILE=`echo $f | sed s/_SUCCESS//`
  echo "mv $REPORT_FILE to target_server"
  scp -r ${REPORT_FILE} target_server:/starhome/data_export/target/path/

  rm -f ${REPORT_FILE}
  rm -f $f

done