Pages

Showing posts with label GG. Show all posts
Showing posts with label GG. Show all posts

Sunday, June 27, 2021

REPLICAT is stuck at same point

 ===========================
General
===========================
Replicat process is stuck.
It has big lag, and it is not coming down
What to check?

===========================
Steps - check REPLICAT
===========================
When running the SEND REPLICAT REP_I_01 STATUS
It seems that the replicat is stuck on the same file.

When checking for dirdat/in files, this is the last file received from the other node.
Need to check on DATAPUMP EXTRACT on the other node


===========================
Steps - check DATAPUMP
===========================
Checking on Datapump Extract
The extract status is RUNNING but it is stuck.
When running SEND EXTRACT DPM_I_01 STATUS
It is stuck on the same trunsaction

===========================
Steps - Fix DATAPUMP issue
===========================

GGSCI (qanfv-1-dbs-1b) 68> STOP EXTRACT DPM_S_01

Sending STOP request to EXTRACT DPM_S_01 ...
STOP request pending end-of-transaction (1,034 records so far).

GGSCI (qanfv-1-dbs-1b) 69> kill EXTRACT DPM_S_01

Sending KILL request to MANAGER ...
Killed process (31058) for EXTRACT DPM_S_01

GGSCI (qanfv-1-dbs-1b) 90> INFO DPM_S_01

EXTRACT    DPM_S_01  Last Started 2021-06-27 12:22   Status ABENDED
Checkpoint Lag       22:57:23 (updated 00:00:01 ago)
Log Read Checkpoint  File /software/ogg/191/dirdat/01/out/es000021373
                     2021-06-26 13:25:39.000000  RBA 450922877

2021-06-27 12:23:02  ERROR   OGG-01091  Unable to open file "/software/ogg/191/dirdat/01/out/es000021373" (error 2, No such file or directory).

****************************************************************
DATAPUMP Extract is expecting for next file, 21373, but this file does not exists.
Extract should be skipped to next available file 21603


GGSCI (qanfv-1-dbs-1b) 93> ALTER EXTRACT DPM_S_01 extseqno 21603
EXTRACT altered.


GGSCI (qanfv-1-dbs-1b) 94> START  EXTRACT DPM_S_01

Sending START request to MANAGER ...
EXTRACT DPM_S_01 starting


Now, Datapump is sending records to the other node, and the REPLICAT is processing data, and the Lag is reducing.

Thursday, February 18, 2021

GG Restart from crontab

=========================
General
=========================

 Restart GG processes from crontab

=========================
Code
=========================
crontab entry
1 06 1 * * /software/oracle/oracle/scripts/gg_restart/gg_restart_all_groups.sh

gg_restart_all_groups.sh
#!/bin/bash

WORK_HOME=/software/oracle/oracle/scripts/gg_restart
LOG_FILE=gg_restart.log

cd $WORK_HOME

./gg_restart_log.sh "${WORK_HOME}" "${LOG_FILE}" "START"

./gg_restart_group.sh EXT_S "ext ext_s*" "${WORK_HOME}" "${LOG_FILE}"
./gg_restart_group.sh EXT_P "ext ext_p*" "${WORK_HOME}" "${LOG_FILE}"
./gg_restart_group.sh EXT_I "ext ext_i*" "${WORK_HOME}" "${LOG_FILE}"

./gg_restart_group.sh DPM_S "ext dpm_s*" "${WORK_HOME}" "${LOG_FILE}"
./gg_restart_group.sh DPM_P "ext dpm_p*" "${WORK_HOME}" "${LOG_FILE}"
./gg_restart_group.sh DPM_I "ext dpm_i*" "${WORK_HOME}" "${LOG_FILE}"

./gg_restart_group.sh REP_S "rep rep_s*" "${WORK_HOME}" "${LOG_FILE}"
./gg_restart_group.sh REP_P "rep rep_p*" "${WORK_HOME}" "${LOG_FILE}"
./gg_restart_group.sh REP_I "rep rep_i*" "${WORK_HOME}" "${LOG_FILE}"

./gg_restart_log.sh "${WORK_HOME}" "${LOG_FILE}" "FINISH" 


gg_restart_log.sh

#!/bin/bash

WORK_HOME=$1
LOG_FILE_NAME=$2
VERB=$3

cd $WORK_HOME

LOG_FILE=${WORK_HOME}/${LOG_FILE_NAME}
RUN_DATE=`date "+%Y%m%d"_"%H%M%S"`

touch $LOG_FILE

if [[ $VERB == "START" ]]; then

 echo "" >>$LOG_FILE
 echo ========================================== >>$LOG_FILE
 echo Start at $RUN_DATE >>$LOG_FILE
 echo ========================================== >>$LOG_FILE
 echo "" >>$LOG_FILE
 echo "Before Restart: " >>$LOG_FILE
 echo "" >>$LOG_FILE 
 free -m >> $LOG_FILE
 echo "" >> $LOG_FILE
 sar -r >> $LOG_FILE
  echo "" >>$LOG_FILE 
fi

if [[ $VERB == "FINISH" ]]; then
  echo "" >>$LOG_FILE
  echo "---------------------------" >>$LOG_FILE
  echo "After Restart: " >>$LOG_FILE
  echo "---------------------------" >>$LOG_FILE  
  echo "" >>$LOG_FILE  
  free -m >> $LOG_FILE
  echo "" >> $LOG_FILE
  sar -r >> $LOG_FILE
  echo "" >>$LOG_FILE    
  echo ========================================== >>$LOG_FILE
  echo Finished at $RUN_DATE >>$LOG_FILE
  echo ========================================== >>$LOG_FILE
  echo "" >>$LOG_FILE
fi

exit 0 

gg_restart_group.sh
#!/bin/bash

GROUP_NAME=$1
RESTART_GROUP=$2
WORK_DIR=$3
LOG_FILE_NAME=$4

#GROUP_NAME=EXT_S
#RESTART_GROUP="ext ext_s*"

LOG_FILE=${WORK_DIR}/${LOG_FILE_NAME}


clear
echo "Restarting ${GROUP_NAME} ${RESTART_GROUP}" >> ${LOG_FILE}
echo "Log File is: ${LOG_FILE}"
exit 0

. /etc/profile > /dev/null 2> /dev/null
TEMP_LOG_FILE="/tmp/restart_temp.tmp"

RUN_DATE=`date "+%Y%m%d"_"%H%M%S"`

# Setting Oracle GoldenGate environments
if [ "$OGGSH" = "" ]; then
   OGGSH=$INFRA_ROOT/oggsh
fi

if [ "$ORACLE_SID" = "" ]; then
   INSTANCE=igt
else
   INSTANCE=$ORACLE_SID
fi
. $OGGSH/gg_login.sh $INSTANCE

touch $LOG_FILE
touch $TEMP_LOG_FILE

echo "" >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo Start Restart for ${GROUP_NAME} group ${RESTART_GROUP} at $RUN_DATE >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo "" >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo Initial State >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo "" >>$LOG_FILE

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOD
info all
exit
EOD
cat  $TEMP_LOG_FILE | grep ${GROUP_NAME} >>$LOG_FILE

echo about to do  "stop ext $RESTART_GROUP" >>$LOG_FILE

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOF
stop ${RESTART_GROUP}
exit
EOF
cat  $TEMP_LOG_FILE >>$LOG_FILE

echo "" >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo After Stop >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo "" >>$LOG_FILE

sleep 10

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOD
info all
exit
EOD
cat  $TEMP_LOG_FILE | grep ${GROUP_NAME}  >>$LOG_FILE


echo "Starting  ${RESTART_GROUP}" >> $LOG_FILE

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOF
start ${RESTART_GROUP}
exit
EOF
cat $TEMP_LOG_FILE >>$LOG_FILE

echo "" >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo After Start >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo "" >>$LOG_FILE

sleep 10

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOD
info all
exit
EOD
cat  $TEMP_LOG_FILE | grep ${GROUP_NAME} >>$LOG_FILE


NOT_RUNNING=`grep STOPPED $TEMP_LOG_FILE |grep ${GROUP_NAME} | wc -l`

if [ $NOT_RUNNING -gt 0 ]; then

echo "Some Processes were not started. Attempting restart" >>$LOG_FILE

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOF
start ${RESTART_GROUP}
exit
EOF
cat $TEMP_LOG_FILE >>$LOG_FILE

fi

#============================================================
# Second restart
#============================================================
sleep 10

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOD
info all
exit
EOD
cat  $TEMP_LOG_FILE | grep ${GROUP_NAME} >>$LOG_FILE

NOT_RUNNING=`grep STOPPED $TEMP_LOG_FILE | wc -l`
if [ $NOT_RUNNING -gt 0 ]; then

echo "Some Processes were not started. Attempting restart" >>$LOG_FILE

$GG_HOME/ggsci > $TEMP_LOG_FILE <<EOF
start ${RESTART_GROUP}
exit
EOF
cat $TEMP_LOG_FILE >>$LOG_FILE

sleep 10

$GG_HOME/ggsci >$TEMP_LOG_FILE <<EOD
info all
exit
EOD
cat  $TEMP_LOG_FILE | grep ${GROUP_NAME} >>$LOG_FILE

fi

echo "" >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo Finished Restart for $GROUP_NAME $RESTART_GROUP at $RUN_DATE >>$LOG_FILE
echo ========================================== >>$LOG_FILE
echo "" >>$LOG_FILE
 




Monday, January 18, 2021

Memory Usage Monitoring script. For SWAP using processes, Golden Gate, Top 20 oracle processes.

crontab

 0,15,30,45 * * * * /software/oracle/oracle/scripts/memory_usage/gg_mem_usage_wrapper.sh

gg_mem_usage_wrapper.sh
#!/bin/bash
WORK_DIR=/software/oracle/oracle/scripts/memory_usage
LOG_FILE=gg_memory.log
RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`

cd ${WORK_DIR}
touch ${LOG_FILE}
echo " " >> ${LOG_FILE}
echo "=================" >> ${LOG_FILE}
echo ${RUN_DATE} >> ${LOG_FILE}
./gg_mem_usage.sh >> ${LOG_FILE}
echo " " >> ${LOG_FILE}
free -m >> ${LOG_FILE}
echo " " >> ${LOG_FILE}

echo "Top 20 Processes" >> ${LOG_FILE}
ps -eo rss,pid,euser,lstart,args:100 --sort %mem | grep -v grep | sort -n | tail -20 | awk '{printf $1/1024 "MB"; $1=""; print }' >> ${LOG_FILE}
echo " " >> ${LOG_FILE}

echo "Top SWAP Using Processes" >> ${LOG_FILE}
echo " " >> ${LOG_FILE}
echo "SWAP Usage from free -m" >> ${LOG_FILE}
free -m | grep Swap >> ${LOG_FILE}
echo " " >> ${LOG_FILE}
echo "SWAP Usage from /proc" >> ${LOG_FILE}
find /proc -maxdepth 2 -path "/proc/[0-9]*/status" -readable -exec awk -v FS=":" '{process[$1]=$2;sub(/^[ \t]+/,"",process[$1]);} END {if(process["VmSwap"] && process["VmSwap"] != "0 kB") printf "%10s %-30s %20s\n",process["Pid"],process["Name"],process["VmSwap"]}' '{}' \; | awk '{print $(NF-1),$0}' | sort -hr | head | cut -d " " -f2- >> ${LOG_FILE}
echo " " >> ${LOG_FILE}

exit

gg_mem_usage.sh
#!/bin/bash
###############################
# determine the OS type
###############################
OSNAME=`uname`
case "$OSNAME" in
  "SunOS")
    echo "OSNAME = $OSNAME"
    ;;
  "Linux")
    echo "OSNAME = $OSNAME"
    ;;
  "*")
    echo "This script has not been verified on $OSNAME"
    exit 1
    ;;
esac
###############################
# set the temp file
###############################
TMPFILE=/tmp/pmem.tmp
if [ -f $TMPFILE ]
then
  rm -f $TMPFILE
fi
################################
# loop over the gg process types
################################
PROCESSES="extract replicat"
for PROCESS in $PROCESSES
do
  FLAG=""
  FLAG=`ps -ef | grep $PROCESS`
  if [ -z "FLAG" ]
  then
    echo "No $PROCESS processes found"
  else
    echo
    echo "#####################################"
    echo "# Individual $PROCESS Process Usage #"
    echo "#####################################"
    case "$OSNAME" in
      "Linux")
        ps -C $PROCESS -O rss > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{print $2/1024, "MB", $12}' | sort -k 2
        ;;
      "SunOS")
        ps -efo vsz,uid,pid,ppid,pcpu,args | grep -v grep | grep $PROCESS > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{print $1/1024, "MB", $8}' | sort -k 2
        ;;
      "*")
        echo "This script has not been verified on $OSNAME"
        exit 1
        ;;
    esac
    rm -f $TMPFILE
    echo
    echo "#####################################"
    echo "#   Total $PROCESS Process Usage    #"
    echo "#####################################"
    case "$OSNAME" in
      "Linux")
        ps -C $PROCESS -O rss > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{count ++; sum=sum+$2; } END \
          { print "Number of processes      =",count; \
          print "AVG Memory usage/process =",sum/1024/count, "MB"; \
          print "Total memory usage       =", sum/1024,  " MB"}'
        ;;
      "SunOS")
        ps -efo vsz,uid,pid,ppid,pcpu,comm | grep -v grep | grep $PROCESS > $TMPFILE
        cat $TMPFILE | awk '{count ++; sum=sum+$1; } END \
          { print "Number of processes      =",count; \
          print "AVG Memory usage/process =",sum/1024/count, "MB"; \
          print "Total memory usage       =", sum/1024,  " MB"}'
        ;;
      "*")
        echo "This script has not been verified on $OSNAME"
        exit 1
        ;;
    esac
    rm -f $TMPFILE
  fi
done
exit

=================
Sample output
=================
20210118_181502
OSNAME = Linux

#####################################
# Individual extract Process Usage #
#####################################
38.7109 MB DPM_I_01
42.125 MB DPM_P_01
41.8672 MB DPM_S_01
70.8867 MB EXT_I_01
78.9297 MB EXT_P_01
361.016 MB EXT_S_01

#####################################
#   Total extract Process Usage    #
#####################################
Number of processes      = 6
AVG Memory usage/process = 105.589 MB
Total memory usage       = 633.535  MB

#####################################
# Individual replicat Process Usage #
#####################################
32.4219 MB REP_I_01
37.2383 MB REP_P_01
34.8789 MB REP_S_01

#####################################
#   Total replicat Process Usage    #
#####################################
Number of processes      = 3
AVG Memory usage/process = 34.8464 MB
Total memory usage       = 104.539  MB

             total       used       free     shared    buffers     cached
Mem:         64175      62568       1607       9373        384      57889
-/+ buffers/cache:       4294      59880
Swap:        10239        122      10117

Top 20 Processes
361.016MB 30432 oracle Sat Jan 16 20:46:19 2021 /software/ogg/1212/extract PARAMFILE /software/ogg/1212/dirprm/ext_s_01.prm REPORTFILE /software/ogg/1212/dirrpt/EXT_S_01.rpt PROCESSID EXT_S_01 USESUBDIRS
371.129MB 30853 oracle Sat Jan 16 20:46:24 2021 ora_ms03_igt
392.086MB 30851 oracle Sat Jan 16 20:46:24 2021 ora_ms02_igt
488.648MB 28170 oracle Sat Jan 16 20:46:05 2021 ora_smon_igt
512.262MB 28454 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
607.949MB 28162 oracle Sat Jan 16 20:46:05 2021 ora_mman_igt
682.445MB 28392 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
724.746MB 29669 oracle Sat Jan 16 20:46:13 2021 oracleigt (LOCAL=NO)
972.762MB 30606 oracle Sat Jan 16 20:46:19 2021 oracleigt (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
1046.21MB 29663 oracle Sat Jan 16 20:46:13 2021 oracleigt (LOCAL=NO)
1173.61MB 32712 oracle Sat Jan 16 20:46:47 2021 oracleigt (LOCAL=NO)
1208.2MB 20575 oracle Sun Jan 17 09:59:59 2021 oracleigt (LOCAL=NO)
1278.24MB 28830 oracle Sat Jan 16 20:46:11 2021 oracleigt (LOCAL=NO)
1934.22MB 30150 oracle Sat Jan 16 20:46:16 2021 oracleigt (LOCAL=NO)
2697.98MB 28386 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
3556.99MB 28390 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
5853.8MB 28388 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
5938.82MB 28443 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
6263.12MB 28298 oracle Sat Jan 16 20:46:10 2021 oracleigt (LOCAL=NO)
6995.96MB 28164 oracle Sat Jan 16 20:46:05 2021 ora_dbw0_igt

Top SWAP Using Processes

SWAP Usage from free -m
Swap:        10239        122      10117

SWAP Usage from /proc
     30649 splunkd                                    49264 kB
      8260 vxconfigd                                  24152 kB


Thursday, May 24, 2018

Golden Gate Error: error while loading shared libraries

===================
General
===================
For unknown reason, ggsci is throwing an error:

/software/ogg/1212/ggsci: error while loading shared libraries: libnnz11.so
/software/ogg/1212/ggsci: error while loading shared libraries: libclntsh.so.11.1

===================
Details
===================
libnnz11.so and libclntsh.so.11.1 are libraries that exist under $ORACLE_HOME
Checking $LD_LIBRARY_PATH and $ORACLE_HOME, the definition is correct.
So the reason for the error was not clear.

>% env | grep LIB
LD_LIBRARY_PATH=/usr/lib:/usr/lib:/software/oracle/112/lib:/usr/lib:/software/oracle/112/lib32

>% env | grep ORACLE_HOME
ORACLE_HOME=/software/oracle/112


===================
Solution
===================
Create soft link from Golden Gate home directory to the 2 libraries in question
Once soft links were created, issue was resolved.
/software/ogg/1212>% ln -s $ORACLE_HOME/lib/libnnz11.so libnnz11.so 
/software/ogg/1212>% ln -s $ORACLE_HOME/lib/libclntsh.so.11.1 libclntsh.so.11.1

Monday, May 21, 2018

Golden Gate During dblogin error OGG-01117 and error OGG-01668 are coming

=====================
General
=====================
During dblogin error OGG-01117 and error OGG-01668 are coming

=====================
Example
=====================
oracle@my_server:/software/ogg/1212>% ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.1.2.0.0 17185003 OGGCORE_12.1.2.0.0_PLATFORMS_130924.1316_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Sep 25 2013 00:31:13
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.
GGSCI (my_server) 1> dblogin userid ogg  password xxx

Source Context :
  SourceModule            : [ggapp.util.pcs]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150472/oggcore/OpenSys/src/gglib/ggapp/pcsutl.c]
  SourceFunction          : [AbendHandler]
  SourceLine              : [938]
  ThreadBacktrace         : [21] elements
                          : [/software/ogg/1212/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f4acb86ddee]]
                          : [/software/ogg/1212/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x31c) [0x7f4acb86ac2c]]
                          : [/software/ogg/1212/libgglog.so(_MSG_ERR_SIGNAL_RECEIVED(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x3b) [0x7f4acb84cb37]]
                          : [./ggsci() [0x4e4543]]
                          : [/lib64/libpthread.so.0() [0x399900f710]]
                          : [/lib64/libpthread.so.0(pthread_mutex_lock+0) [0x39990093a0]]
                          : [./ggsci(ggs::gglib::MultiThreading::Mutex::Lock()+0x9) [0x516159]]
                          : [./ggsci(CContextItem::operator char const*() const+0x15) [0x4c1a2b]]
                          : [./ggsci(DBOCI_init_connection_logon(ggs::gglib::ggapp::CLoginName const&, ggs::gglib::ggapp::CDBObjName<(DBObjType)12> const&, char const*, int, int, int, char*)+0x1e5) [0x524215]]
                          : [./ggsci() [0x534baf]]
                          : [./ggsci(gl_db_login(char const*, char const*, ggs::gglib::ggapp::CDBObjName<(DBObjType)11>&, ggs::gglib::ggapp::CDBObjName<(DBObjType)12>&, short, char (&) [2048])+0x5f) [0x534edf]]
                          : [./ggsci() [0x4888fb]]
                          : [./ggsci(GGSCIDB_get_command(char const*, char const*, char const*, short, short, char (&) [2048])+0x11b) [0x48628b]]
                          : [./ggsci(do_cmd(char*, unsigned long, char*, unsigned long)+0x908) [0x4bbe58]]
                          : [./ggsci() [0x4c0193]]
                          : [./ggsci(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x4f) [0x513c2f]]
                          : [./ggsci(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x104) [0x513e34]]
                          : [./ggsci(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x8b) [0x513f2b]]
                          : [./ggsci(main+0x3f) [0x49d33f]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3998c1ed1d]]
                          : [./ggsci() [0x47df19]]

2018-05-21 07:20:22  ERROR   OGG-01117  Received signal: Segmentation violation (11).

2018-05-21 07:20:22  ERROR   OGG-01668  PROCESS ABENDING.
Segmentation fault (core dumped)


=====================
Solution
=====================

Set the Oracle Environment variables.
In this example, it is done via script oraigt

oracle@my_server:~>% env | grep ORA
ORASH=/etc/sh/orash
BAS_ORACLE_LIST=igt


oracle@my_server:~>% oraigt
oracle@my_server:~>% env | grep ORA
ORASH=/etc/sh/orash
BAS_ORACLE_LIST=igt
ORA_VER=1120
ORACLE_SID=igt
ORACLE_BASE=/software/oracle
ORACLE_ENV_DEFINED=yes
ORA_NLS33=/software/oracle/112/ocommon/nls/admin/data
ORACLE_HOME=/software/oracle/112


oracle@my_server:/software/ogg/1212>% ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.1.2.0.0 17185003 OGGCORE_12.1.2.0.0_PLATFORMS_130924.1316_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Sep 25 2013 00:31:13
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.

GGSCI (my_server) 1> dblogin userid ogg  password xxx
Successfully logged into database.

Oracle Golden Gate Extract is failing with: ERROR OGG-00868 Error code 1291, error message: ORA-01291: missing logfile.

=======================
General
=======================
The archive folder was 100% full.
As an SOS operation, all archive files were deleted.
Now, several Extract process are in Abended state.
When checking  with info extract EXT_I_01 DETAIL, the error is:

2018-05-21 06:33:43  ERROR   OGG-00868  Error code 1291, error message: ORA-01291: missing logfile
 (Missing Log File /oracle_db/db2/db_igt/arch/arch0001_136774_827755274.arc. Read Position SCN -976006635).

2018-05-21 06:33:43  ERROR   OGG-01668  PROCESS ABENDING.

How to resolve this situation?

=======================
Evidences
=======================
via ggsci

info all 
EXTRACT     ABENDED     EXT_I_01    00:06:36      118:28:15   
EXTRACT     ABENDED     EXT_I_02    00:06:31      118:28:14   
EXTRACT     ABENDED     EXT_I_04    00:06:28      118:28:20   
EXTRACT     ABENDED     EXT_I_05    00:00:06      263:13:17   
EXTRACT     ABENDED     EXT_I_06    00:06:29      118:28:20   
EXTRACT     ABENDED     EXT_I_07    00:06:28      118:28:18   
EXTRACT     ABENDED     EXT_I_09    00:06:30      118:28:22   
EXTRACT     ABENDED     EXT_I_10    00:06:27      118:28:20   
EXTRACT     ABENDED     EXT_I_11    00:06:26      118:28:21   
EXTRACT     ABENDED     EXT_I_13    00:06:36      118:28:15   
EXTRACT     ABENDED     EXT_I_14    00:06:33      118:28:19   
EXTRACT     ABENDED     EXT_I_15    00:06:31      118:28:19   
EXTRACT     ABENDED     EXT_I_17    00:06:30      118:28:20   
EXTRACT     ABENDED     EXT_P_01    00:06:32      118:28:20   
EXTRACT     ABENDED     EXT_P_02    00:06:32      118:28:15   
EXTRACT     ABENDED     EXT_P_04    00:06:30      118:28:21   
EXTRACT     ABENDED     EXT_P_05    00:00:05      263:13:12   
EXTRACT     ABENDED     EXT_P_06    00:06:25      118:28:26   
EXTRACT     ABENDED     EXT_P_07    00:06:28      118:28:18   
EXTRACT     ABENDED     EXT_P_09    00:06:30      118:28:20   
EXTRACT     ABENDED     EXT_P_10    00:06:27      118:28:21   
EXTRACT     ABENDED     EXT_P_11    00:06:30      118:28:22   
EXTRACT     ABENDED     EXT_P_13    00:06:35      118:28:16   
EXTRACT     ABENDED     EXT_P_14    00:06:28      118:28:19   
EXTRACT     ABENDED     EXT_P_15    00:06:28      118:28:18   
EXTRACT     ABENDED     EXT_P_17    00:06:33      118:28:20   
EXTRACT     ABENDED     EXT_S_13    00:06:29      118:28:19   
EXTRACT     ABENDED     EXT_S_14    00:06:32      118:28:21   
EXTRACT     ABENDED     EXT_S_15    00:06:28      118:28:19   
EXTRACT     ABENDED     EXT_S_17    00:06:34      118:28:18  


GGSCI (my_server) 19> VIEW PARAM EXT_I_01 
Will display extact parameter files


GGSCI (my_server) 19> info extract EXT_I_01 DETAIL

EXTRACT    EXT_I_01  Last Started 2018-01-29 06:48   Status ABENDED
Checkpoint Lag       00:06:36 (updated 118:35:46 ago)
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2018-05-16 07:52:30
                     SCN 3.3318960661 (16203862549)

  Target Extract Trails:

  Trail Name                                       Seqno        RBA     Max MB Trail Type

  /software/ogg/1212/dirdat/01/out/ei                 12     458809        100 EXTTRAIL  


Integrated Extract outbound server first scn: Unavailable.Need DBLOGIN.

  Extract Source          Begin             End        
  Not Available           2018-01-17 10:36  2018-05-16 07:52
  Not Available           2018-01-17 10:36  2018-01-29 06:47
  Not Available           2018-01-17 10:36  2018-01-28 06:47
  Not Available           2018-01-17 10:36  2018-01-27 06:47
  Not Available           2018-01-17 10:36  2018-01-26 06:47
  Not Available           2018-01-17 10:36  2018-01-25 06:47
  Not Available           2018-01-17 10:36  2018-01-24 06:47
  Not Available           2018-01-17 10:36  2018-01-23 06:47
  Not Available           2018-01-17 10:36  2018-01-22 06:47
  Not Available           2018-01-17 10:36  2018-01-21 06:47
  Not Available           2018-01-17 10:36  2018-01-20 06:47
  Not Available           2018-01-17 10:36  2018-01-19 06:47
  Not Available           2018-01-17 10:36  2018-01-18 06:47
  Not Available           * Initialized *   2018-01-17 10:36



Current directory    /software/ogg/1212
Report file          /software/ogg/1212/dirrpt/EXT_I_01.rpt
Parameter file       /software/ogg/1212/dirprm/ext_i_01.prm
Checkpoint file      /software/ogg/1212/dirchk/EXT_I_01.cpe
Process file         /software/ogg/1212/dirpcs/EXT_I_01.pce
Error log            /software/ogg/1212/ggserr.log


 less  /software/ogg/1212/dirrpt/EXT_I_01.rpt
Source Context :
  SourceModule            : [er.redo.oraxo]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150472/oggcore/OpenSys/src/app/er/redo/oracle/redooraix.c]
  SourceFunction          : [REDOORAIX_abend_if_missing_logfile]
  SourceLine              : [8206]
  ThreadBacktrace         : [15] elements
                          : [/software/ogg/1212/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7fd7c02e5dee]]
                          : [/software/ogg/1212/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x31c) [0x7fd7c02e2c2c]]
                          : [/software/ogg/1212/libgglog.so(_MSG_ERR_DB_GENERIC_FAILURE(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7fd7c02c81d1]]
                          : [/software/ogg/1212/extract(REDOORAIX_abend_if_missing_logfile(char*)+0x1b9) [0x7b9a57]]
                          : [/software/ogg/1212/extract(REDOORAIX_attach(ggs::gglib::ggunicode::UString&, int, char (&) [2048])+0x274) [0x7ae574]]
                          : [/software/ogg/1212/extract(REDOORAIX_validate_config()+0x4e9) [0x7b21f9]]
                          : [/software/ogg/1212/extract(REDO_validate_config(int, unsigned int*, int*)+0xfab) [0x71f07b]]
                          : [/software/ogg/1212/extract(redo_log_setup()+0x36) [0x5ad3c6]]
                          : [/software/ogg/1212/extract(extract_main(int, char**)+0x29e) [0x5b096e]]
                          : [/software/ogg/1212/extract(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x4f) [0x68743f]]
                          : [/software/ogg/1212/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x104) [0x687694]]
                          : [/software/ogg/1212/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x8b) [0x68785b]]
                          : [/software/ogg/1212/extract(main+0x3f) [0x5b047f]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3998c1ed1d]]
                          : [/software/ogg/1212/extract() [0x51dc59]]

2018-05-21 06:33:43  ERROR   OGG-00868  Error code 1291, error message: ORA-01291: missing logfile
 (Missing Log File /oracle_db/db2/db_igt/arch/arch0001_136774_827755274.arc. Read Position SCN -976006635).

2018-05-21 06:33:43  ERROR   OGG-01668  PROCESS ABENDING.

======================================================
Solution
======================================================
The solution would be to unregister and register the extract process, causing it to reset to currect SCN value.

GGSCI (my_server) 2> DBLOGIN USERID OGG  PASSWORD <password>
Successfully logged into database.

GGSCI (my_server) 15> STOP EXTRACT EXT_I_02
EXTRACT EXT_I_02 is already stopped.

If extract cannot be stopped do:
ALTER EXTRACT EXT_P_02, BEGIN NOW

GGSCI (my_server) 16> UNREGISTER EXTRACT EXT_I_02 DATABASE   
Successfully unregistered EXTRACT EXT_I_02 from database.
This might take several minutes

GGSCI (my_server) 17> REGISTER EXTRACT EXT_I_02 DATABASE
Extract EXT_I_02 successfully registered with database at SCN 16308342873.
This might take several minutes

GGSCI (my_server) 18> START EXTRACT EXT_I_02

Sending START request to MANAGER ...
EXTRACT EXT_I_02 starting


GGSCI (my_server) 13> INFO EXTRACT EXT_I_02

EXTRACT    EXT_I_01  Last Started 2018-05-21 08:20   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:00 ago)
Process ID           36729
Log Read Checkpoint  Oracle Integrated Redo Logs
                     First Record       
                     SCN 3.3405189667 (16290091555)


===============================
Additional Information
===============================
Checking the SCN with INFO EXTRACT XXX SHOWCH command one can see that the SCN in extract is pointing to a deleted archive file

Get data from Database
oracle@my_server:/oracle_db/db2/db_igt/arch>% ls -ltr | head -10
total 59342928
-rw-r----- 1 oracle dba 372660224 May 20 15:44 arch0001_137783_827755274.arc
-rw-r----- 1 oracle dba 371138560 May 20 15:44 arch0001_137784_827755274.arc
-rw-r----- 1 oracle dba 393000960 May 20 15:44 arch0001_137785_827755274.arc
-rw-r----- 1 oracle dba 372824064 May 20 15:45 arch0001_137786_827755274.arc
-rw-r----- 1 oracle dba 384171008 May 20 15:46 arch0001_137787_827755274.arc
-rw-r----- 1 oracle dba 491667968 May 20 15:49 arch0001_137788_827755274.arc
-rw-r----- 1 oracle dba 489092608 May 20 15:51 arch0001_137789_827755274.arc
-rw-r----- 1 oracle dba 377072128 May 20 15:53 arch0001_137790_827755274.arc
-rw-r----- 1 oracle dba 379522048 May 20 15:55 arch0001_137791_827755274.arc

SELECT recid, 
       name, 
       first_change#, 
       next_change#,  
       TO_CHAR(first_time, 'YYYYMMDD hh24:mi:ss') AS first_time 
  FROM V$ARCHIVED_LOG 
 WHERE name LIKE '%arch0001_137783_827755274%';

recid  name
------- --------------------------------------------------------
137782 /oracle_db/db2/db_igt/arch/arch0001_137783_827755274.arc

first_change# next_change#  first_time
------------- ------------ -----------------
16290091555 16290177738 20180520 13:23:04


SELECT dbid, current_scn  FROM V$DATABASE;
dbid current_scn
----------- -----------
1066039690 16307173674


Get data from Extract

GGSCI (my_server) 27> INFO EXTRACT EXT_I_05 SHOWCH

EXTRACT    EXT_I_05  Last Started 2018-05-16 07:53   Status ABENDED
Checkpoint Lag       00:00:06 (updated 265:36:17 ago)
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2018-05-10 07:13:58
                     SCN 3.3213072676 (16097974564)


Current Checkpoint Detail:

Read Checkpoint #1

  Oracle Integrated Redo Log

  Startup Checkpoint (starting position in the data source):
    Timestamp: 2015-08-09 08:42:51.000000
    SCN: Not available

  Recovery Checkpoint (position of oldest unprocessed transaction in the data source):
    Timestamp: 2018-05-10 07:13:58.000000
    SCN: 3.3213072615 (16097974503)

  Current Checkpoint (position of last record read in the data source):
    Timestamp: 2018-05-10 07:13:58.000000
    SCN: 3.3213072676 (16097974564)

  BR Previous Recovery Checkpoint:
    Timestamp: 2018-01-29 06:48:22.685278
    SCN: Not available

  BR Begin Recovery Checkpoint:
    Timestamp: 2018-05-10 05:21:30.000000
    SCN: 3.3212132747 (16097034635)

  BR End Recovery Checkpoint:
    Timestamp: 2018-05-10 05:21:30.000000
    SCN: 3.3212132747 (16097034635)

Write Checkpoint #1

  GGS Log Trail

  Current Checkpoint (current write position):
    Sequence #: 363
    RBA: 1495
    Timestamp: 2018-05-10 07:14:04.547894
    Extract Trail: /software/ogg/1212/dirdat/05/out/ei
    Trail Type: EXTTRAIL

Header:
  Version = 2
  Record Source = A
  Type = 13
  # Input Checkpoints = 1
  # Output Checkpoints = 1

File Information:
  Block Size = 2048
  Max Blocks = 100
  Record Length = 2048
  Current Offset = 0

Configuration:
  Data Source = 3
  Transaction Integrity = 1
  Task Type = 0

Status:
  Start Time = 2018-05-16 07:53:24
  Last Update Time = 2018-05-10 07:14:04
  Stop Status = A
  Last Result = 0
  

So in archive logs, the first available SCN is 16290091555 starting at 20180520 13:23:04
But the extract is expecting to process SCN 16097974503 from 2018-05-10 07:13:58.

The latest available archive file start with first_change# 16290091555 but extract expect to process SCN starting from 16097974503.

Since the archive files were deleted, the only option is to reset the extract, by register and register it again to the database.

Once that was done, the SCN in Golden Gate extract was in sync with SCN in database.