Golden Gate Extract is in Abended state because of OGG-01496 Failed to open target trail file

==========================
General
==========================
The issue: One of the Oracle GG (Golden Gate) EXTRACT  processes is in ABENDED status

>%cd /software/ogg/1212
/software/ogg/1212/>% ./ggsci
info all

EXTRACT     RUNNING     EXT_S_04    00:00:06      00:00:06    
EXTRACT     RUNNING     EXT_S_05    00:00:05      00:00:01    
EXTRACT     RUNNING     EXT_S_06    00:00:06      00:00:06    
EXTRACT     ABENDED     EXT_S_07    00:00:06      144:13:03   
EXTRACT     RUNNING     EXT_S_09    00:00:05      00:00:01    
EXTRACT     RUNNING     EXT_S_10    00:00:06      00:00:06    
EXTRACT     RUNNING     EXT_S_11    00:00:05      00:00:01    
EXTRACT     RUNNING     EXT_S_12    00:00:05      00:00:01    
EXTRACT     RUNNING     EXT_S_13    00:00:06      00:00:06  

==========================
Investigation and resolution
==========================
/software/ogg/1212/>% ./ggsci
VIEW REPORT EXT_S_07
...
...
Database Version:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

Database Language and Character Set:
NLS_LANGUAGE     = "AMERICAN" 
NLS_TERRITORY    = "AMERICA" 
NLS_CHARACTERSET = "AL32UTF8" 

2018-04-22 06:59:58  INFO    OGG-02089  Source redo compatibility version is: 11.2.0.4.

2018-04-22 06:59:59  WARNING OGG-02045  Database does not have streams_pool_size initialization parameter configured.

2018-04-22 07:00:00  INFO    OGG-02068  Integrated capture successfully attached to logmining server OGG$CAP_EXT_S_07 using OGGCapt
ure API.

2018-04-22 07:00:00  INFO    OGG-02086  Integrated Dictionary will be used.

Source Context :
  SourceModule            : [er.recovery]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150472/oggcore/OpenSys/src/app/er/recovery.c]
  SourceFunction          : [RECOVERY_initialize]
  SourceLine              : [1862]
  ThreadBacktrace         : [11] elements
                          : [/software/ogg/1212/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f3ad7f15dee]]
                          : [/software/ogg/1212/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x31
c) [0x7f3ad7f12c2c]]
                          : [/software/ogg/1212/libgglog.so(_MSG_ERR_RECOVERY_OPEN_FAILED(CSourceContext*, char const*, int, CMessa
geFactory::MessageDisposition)+0x3c) [0x7f3ad7eef8fc]]
                          : [/software/ogg/1212/extract(RECOVERY_initialize()+0x3e6) [0x5623f6]]
                          : [/software/ogg/1212/extract(extract_main(int, char**)+0x34b) [0x5b0a1b]]
                          : [/software/ogg/1212/extract(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x4f) [0x68743f]]
                          : [/software/ogg/1212/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::T
hread::ThreadArgs*)+0x104) [0x687694]]
                          : [/software/ogg/1212/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x8b) [0x68785b]]
                          : [/software/ogg/1212/extract(main+0x3f) [0x5b047f]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3998c1ed1d]]
                          : [/software/ogg/1212/extract() [0x51dc59]]

2018-04-22 07:00:00  ERROR   OGG-01496  Failed to open target trail file /software/ogg/1212/dirdat/07/out/es000436, at RBA 35310590
.

2018-04-22 07:00:00  ERROR   OGG-01668  PROCESS ABENDING.

GGSCI (esp-tel-2-dbu-2) 1> ALTER EXT_I_07 ETROLLOVER

2018-04-22 08:00:46  INFO    OGG-01520  Rollover performed.  For each affected output trail of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new trail file;  it will not happen automatically.
EXTRACT altered.

GGSCI (esp-tel-2-dbu-2) 2> INFO EXT_I_07

EXTRACT    EXT_I_07  Initialized   2018-04-22 07:06   Status ABENDED
Checkpoint Lag       144:05:34 (updated 00:01:03 ago)
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2018-04-16 07:55:12
                     SCN 3.2337132669 (15222034557)


GGSCI (esp-tel-2-dbu-2) 3> START EXTRACT EXT_I_07

Sending START request to MANAGER ...
EXTRACT EXT_I_07 starting


GGSCI (esp-tel-2-dbu-2) 4> INFO EXT_I_07

EXTRACT    EXT_I_07  Last Started 2018-04-22 08:02   Status RUNNING
Checkpoint Lag       144:06:58 (updated 00:00:06 ago)
Process ID           12870
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2018-04-16 07:55:12
                     SCN 3.2337132669 (15222034557)



====================================================
What is the cause for "OGG-01496 Failed to open target trail file"?
====================================================
If existing extract output trail file was purged, it would cause the extract process to encounter this error.

By default, Extract Process writes EOF to trail file and the downstream Pump or Replicat process expect to see an EOF before switching to a next file.

The problem, is that Extract Process cannot write EOF to the current file because is was 

deleted or corrupted, and we need manually switch to next sequence file.

The solution is to use ETROLLOVER command and restart the EXTRACT.

====================================================

ETROLLOVER
====================================================
ETROLLOVER causes the extract to increment to the next trail file in the trail sequence when it restarts.

For example, if the current file is ET000002, the current file will be ET000003 when Extract restarts. 

A trail can be incremented from 000001 through 999999, and then the sequence numbering starts over at 000000.