ERROR OGG-02191 Incompatible record

=====================
General
=====================

REPLICAT abends with error
ERROR   OGG-02171  Error reading LCR from data source. 

Need to skip the bad record in file, and restart REPLICAT

=====================
Evidences
=====================

 2020-10-28 15:47:29  INFO    OGG-02243  Opened trail file /software/ogg/191/dirdat/01/in/es000000000 at 2020-10-28 15:47:29.963515.

2020-10-28 15:47:50  ERROR   OGG-02171  Error reading LCR from data source. Status 509, data source type TrailDataSource.

2020-10-28 15:47:50  ERROR   OGG-02191  Incompatible record 101 in /software/ogg/191/dirdat/01/in/es000000000, rba 696,172 when getting trail header.

2020-10-28 15:47:50  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/es000000000, current RBA 696,172, 0 records, m_file_seqno = 0, m_file_rba = 696,172.

=====================
Solution
=====================

Use logdump utility to get next record position.
In general, need to skip the bad record in file, and restart REPLICAT.
First, position the position on the failed record.

The tool will display the next record.

Optionally, get the next+1 record.

Then, set the REPLICAT to start at the next record.

ALTER REP_S_01, EXTSEQNO 0, EXTRBA <THE SAVED RBA>

cd $GG_HOME

oracle@qanfv-1-dbs-1b:/software/ogg/191>% ./logdump

Oracle GoldenGate Log File Dump Utility for Oracle Version 19.1.0.0.1 OGGCORE_19.1.0.0.0_PLATFORMS_190524.2201

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.

Logdump 1 >open

Error: Expected a filename

Logdump 2 >open /software/ogg/191/dirdat/01/in/es000000000

Current LogTrail is /software/ogg/191/dirdat/01/in/es000000000

Logdump 3 >ghdr on

Logdump 4 >detail on

Logdump 5 >pos 696172

Reading forward from RBA 696172

Logdump 6 >scanforheader

___________________________________________________________________

Hdr-Ind    :     E  (x45)     Partition  :     L  (x4c)
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)
RecLength  :   962  (x03c2)   IO Time    : 2020/10/26
0:19:07.005.706
IOType     :   135  (x87)     OrigNode   :   255  (xff)
TransInd   :     .  (x03)     FormatType :     R  (x52)
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)
AuditRBA   :     200132       AuditPos   : 122219168
Continued  :     N  (x00)     RecCount   :     1  (x01)

2020/10/26 10:19:07.005.706 GGSUnifiedPKUpdate   Len   962 RBA 696345
Name: OGG.GG_HEARTBEAT_SEED  (TDR Index: 2)
After  Image:                                             Partition x4c   G  s

 df01 0000 0000 0700 0000 0300 4947 5401 001f 0000 | ............IGT.....
 0032 3032 302d 3130 2d32 363a 3130 3a31 383a 3037 | .2020-10-26:10:18:07
 2e32 3530 3233 3330 3030 0200 0400 ffff 0000 0300 | .250233000..........
 0400 ffff 0000 0400 0400 ffff 0000 0500 0400 ffff | ....................
 0000 0600 1f00 ffff 0000 0000 0000 0000 0000 0000 | ....................
 0000 0000 0000 0000 0000 0000 0000 0000 0007 001f | ....................
 00ff ff00 0000 0000 0000 0000 0000 0000 0000 0000 | ....................
Before Image          Len   483 (x000001e3)
BeforeColumnLen    479 (x000001df)
Column     0 (x0000), Len     7 (x0007)
Column     1 (x0001), Len    31 (x001f)
Column     2 (x0002), Len     4 (x0004)
Column     3 (x0003), Len     4 (x0004)
Column     4 (x0004), Len     4 (x0004)
Column     5 (x0005), Len     4 (x0004)
Column     6 (x0006), Len    31 (x001f)
Column     7 (x0007), Len    31 (x001f)
Column     8 (x0008), Len    31 (x001f)
Column     9 (x0009), Len    31 (x001f)
Column    10 (x000a), Len    12 (x000c)
Column    11 (x000b), Len    14 (x000e)
Column    12 (x000c), Len     4 (x0004)
Column    13 (x000d), Len    31 (x001f)
Column    14 (x000e), Len    31 (x001f)
Column    15 (x000f), Len    31 (x001f)
Column    16 (x0010), Len    31 (x001f)
Column    17 (x0011), Len     4 (x0004)
Column    18 (x0012), Len    14 (x000e)
Column    19 (x0013), Len    14 (x000e)
Column    20 (x0014), Len    31 (x001f)

After Image           Len   479 (x000001df)
Column     0 (x0000), Len     7 (x0007)
Column     1 (x0001), Len    31 (x001f)
Column     2 (x0002), Len     4 (x0004)
Column     3 (x0003), Len     4 (x0004)
Column     4 (x0004), Len     4 (x0004)
Column     5 (x0005), Len     4 (x0004)
Column     6 (x0006), Len    31 (x001f)
Column     7 (x0007), Len    31 (x001f)
Column     8 (x0008), Len    31 (x001f)
Column     9 (x0009), Len    31 (x001f)
Column    10 (x000a), Len    12 (x000c)
Column    11 (x000b), Len    14 (x000e)
Column    12 (x000c), Len     4 (x0004)
Column    13 (x000d), Len    31 (x001f)
Column    14 (x000e), Len    31 (x001f)
Column    15 (x000f), Len    31 (x001f)
Column    16 (x0010), Len    31 (x001f)
Column    17 (x0011), Len     4 (x0004)
Column    18 (x0012), Len    14 (x000e)
Column    19 (x0013), Len    14 (x000e)
Column    20 (x0014), Len    31 (x001f)

Logdump 7 >n
___________________________________________________________________
Hdr-Ind    :     E  (x45)     Partition  :     .  (x0c)
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)
RecLength  :   271  (x010f)   IO Time    : 2020/10/26
0:19:15.005.532
IOType     :     5  (x05)     OrigNode   :   255  (xff)
TransInd   :     .  (x00)     FormatType :     R  (x52)
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)
AuditRBA   :     200132       AuditPos   : 131688464
Continued  :     N  (x00)     RecCount   :     1  (x01)

2020/10/26 10:19:15.005.532 Insert               Len   271 RBA 697424
Name: LAB_QANFV_ALLQQ.GA_W_COUNTERS_HISTORY  (TDR Index: 1)
After  Image:                                             Partition
0c   G  b

 0000 1300 0000 0f00 3130 3330 3130 3438 3130 3030 |
.......103010481000
 3030 3001 0009 0000 0005 0033 3635 3839 0200 0600 |
00........36589....
 0000 0200 3136 0300 0500 0000 0100 3004 0005 0000 | ....16........0.....
 0001 0030 0500 0500 0000 0100 3006 0005 0000 0001 | ...0........0.......
 0030 0700 0500 0000 0100 3008 000a 0000 0000 0000 | .0........0.........
 0000 0000 0009 000a 0000 0000 0000 0000 0000 000a | ....................
 0015 0000 0032 3032 302d 3130 2d32 363a 3130 3a31 | .....2020-10-26:10:1
Column     0 (x0000), Len    19 (x0013)
Column     1 (x0001), Len     9 (x0009)
Column     2 (x0002), Len     6 (x0006)
Column     3 (x0003), Len     5 (x0005)
Column     4 (x0004), Len     5 (x0005)
Column     5 (x0005), Len     5 (x0005)
Column     6 (x0006), Len     5 (x0005)
Column     7 (x0007), Len     5 (x0005)
Column     8 (x0008), Len    10 (x000a)
Column     9 (x0009), Len    10 (x000a)
Column    10 (x000a), Len    21 (x0015)
Column    11 (x000b), Len    10 (x000a)
Column    12 (x000c), Len     7 (x0007)
Column    13 (x000d), Len    10 (x000a)
Column    14 (x000e), Len    21 (x0015)
Column    15 (x000f), Len     4 (x0004)
Column    16 (x0010), Len     4 (x0004)
Column    17 (x0011), Len     4 (x0004)
Column    18 (x0012), Len     4 (x0004)
Column    19 (x0013), Len     4 (x0004)
Column    20 (x0014), Len    10 (x000a)
Column    21 (x0015), Len     5 (x0005)
Logdump 8 >
Logdump 8 >exit

./ggsci

GGSCI (qanfv-1-dbs-1b) 2> ALTER REPLICAT REP_S_01, EXTSEQNO 0 ,EXTRBA 696345

2020-10-28 15:58:14  INFO    OGG-06594  Replicat REP_S_01 has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REP_S_01 with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.

GGSCI (qanfv-1-dbs-1b) 3> START REP_S_01

Sending START request to MANAGER ...

REPLICAT REP_S_01 starting

GGSCI (qanfv-1-dbs-1b) 111>  INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:04
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:00
EXTRACT     RUNNING     DPM_S_01    00:00:05      00:00:08
EXTRACT     RUNNING     EXT_I_01    00:00:03      00:00:06
EXTRACT     RUNNING     EXT_P_01    00:00:03      00:00:03
EXTRACT     RUNNING     EXT_S_01    00:00:02      00:00:08
REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:01
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:01
REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:01

Golden Gate REPLICAT Error: ERROR OGG-01091 Unable to open file "XXX" (error 2, No such file or directory).

=========================
General
=========================
Oracle Golden Gate Replicat Process abends with following error:
ERROR   OGG-01091  Unable to open file "/software/ogg/191/dirdat/01/in/es000004665" (error 2, No such file or directory).

Checking the file system, indeed there is no such file:
There is indeed no such file.
First file being es000000000

ls -ltr   /software/ogg/191/dirdat/01/in/es*/
-rw-r----- 1 oracle dba 499997096 Feb 13 08:26 es000000000
-rw-r----- 1 oracle dba 499996646 Feb 13 08:26 es000000001
-rw-r----- 1 oracle dba 499999334 Feb 13 08:27 es000000002
-rw-r----- 1 oracle dba 499998299 Feb 13 08:27 es000000003
-rw-r----- 1 oracle dba 499998846 Feb 13 08:27 es000000004

=========================
Solution
=========================

Reset Replicat to start at first available file:

EXTSEQNO 0 - point to file es000000000

EXTRBA 0 - point to start position of the file

GGSCI> DBLOGIN USERID USER PASSWORD PASS

GSCI (qanfv-1-dbs-1b) 17> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:09

EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:05

EXTRACT     RUNNING     DPM_S_01    00:00:03      00:00:04

EXTRACT     RUNNING     EXT_I_01    00:00:03      00:00:02

EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:09

EXTRACT     RUNNING     EXT_S_01    00:00:02      00:00:05

REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:04

REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:04

REPLICAT    ABENDED     REP_S_01    00:00:00      07:10:31

GGSCI> ALTER REPLICAT REP_S_01, EXTSEQNO 0 EXTRBA 0

REPLICAT altered.

GGSCI> START REPLICAT REP_S_01

REPLICAT REP_P_01 starting

GGSCI> INFO ALL

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:02

EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:01

EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:04

EXTRACT     RUNNING     EXT_I_01    00:00:03      00:00:04

EXTRACT     RUNNING     EXT_P_01    00:00:00      00:00:03

EXTRACT     RUNNING     EXT_S_01    00:00:02      00:00:04

REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:01

REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:00

REPLICAT    RUNNING     REP_S_01    43:50:16      00:00:00

Since many files were pending replication, there is a backlog, which should be cleared alter some time

REPLICAT    RUNNING     REP_S_01    43:49:57      00:00:00

REPLICAT    RUNNING     REP_S_01    42:12:40      00:00:00

Note: This is not good!!
Because when starting without specific RBA, it will default to RBA from the old file, instead of starting with RBA=0. For example, it will try to start with EXTSEQNO 0 EXTRBA 8132, instead of EXTSEQNO 0 EXTRBA 0

GGSCI> ALTER REPLICAT REP_S_01 EXTSEQNO 0

REPLICAT altered.

GGSCI> INFO ALL

GGSCI> REPLICAT    ABENDED     REP_S_01    00:00:00      07:10:31

=========================
Additional info
=========================

Over weekend GG Replication was broken, and files filled up 100% of /software/ogg/191/dirdat/01/

Since this was a test environment, as a quick solution, existing files were deleted and GG started from scratch.

Golden Gate, Unregister and Register Extract by Example

===========================

Unregister and Register Extract

===========================
DBLOGIN USERID OGG PASSWORD XXXXXX

STOP EXTRACT EXT_I_01

    

DELETE EXTRACT EXT_I_01

REGISTER EXTRACT EXT_I_01 DATABASE

ADD EXTRACT EXT_I_01 INTEGRATED TRANLOG, BEGIN NOW

ADD EXTTRAIL /software/ogg/191/dirdat/01/out/ei EXTRACT EXT_I_01

 

===========================
Unregister and Register Datapump

===========================

DELETE EXTRACT DPM_I_01

ADD EXTRACT DPM_I_01 EXTTRAILSOURCE /software/ogg/191/dirdat/01/out/ei

ADD RMTTRAIL /software/ogg/191/dirdat/01/in/ei EXTRACT DPM_I_01

START EXTRACT DPM_I_01

INFO DPM_I_01

===========================

Unregister and Register Replicat

===========================

STOP REPLICAT REP_I_01

DELETE REPLICAT REP_I_01

ADD REPLICAT REP_I_01,  EXTTRAIL /software/ogg/191/dirdat/01/in/ei

START REPLICAT REP_I_01

===========================

Unregister and Register Group

===========================

DBLOGIN USERID OGG, PASSWORD XXXXXX

STOP EXTRACT EXT_N_01

UNREGISTER EXTRACT EXT_N_01 DATABASE 

DELETE EXTRACT EXT_N_01

REGISTER EXTRACT EXT_N_01 DATABASE

ADD EXTRACT EXT_N_01 INTEGRATED TRANLOG, BEGIN NOW

ADD EXTTRAIL /software/ogg/191/dirdat/01/out/en EXTRACT EXT_N_01

START EXTRACT EXT_N_01

STOP EXTRACT DPM_N_01

DELETE EXTRACT DPM_N_01

ADD EXTRACT DPM_N_01 EXTTRAILSOURCE /software/ogg/191/dirdat/01/out/en

ADD RMTTRAIL /software/ogg/191/dirdat/01/in/en EXTRACT DPM_N_01

START EXTRACT DPM_N_01

STOP REPLICAT REP_N_01

DELETE REPLICAT REP_N_01

ADD REPLICAT REP_N_01, INTEGRATED EXTTRAIL /software/ogg/191/dirdat/01/in/en 

START REPLICAT REP_N_01

Whenever starting a GG processes, it is logged

alert log:

GoldenGate Apply Reader AS01 for OGG$REP_N_01 with pid=158 OS id=17350 stopped

2021-06-28T06:32:25.564814+00:00

GoldenGate APPLY AP05 for OGG$REP_N_01 with pid=157, OS id=17346 stopped

2021-06-28T06:32:33.662778+00:00

GoldenGate Apply: OGG$REP_N_01 APPLY Dropped

2021-06-28T06:32:33.663002+00:00

APPLY OGG$REP_N_01: Apply User: OGG

2021-06-28T06:32:33.663083+00:00

APPLY OGG$REP_N_01: Apply Tag: 00

2021-06-28T06:32:47.704457+00:00

GoldenGate Apply: OGG$REP_N_01 APPLY Created

2021-06-28T06:32:47.752593+00:00

APPLY OGG$REP_N_01: Apply User: OGG

2021-06-28T06:32:47.752693+00:00

APPLY OGG$REP_N_01: Apply Tag: 00

2021-06-28T06:32:48.323311+00:00

GoldenGate APPLY AP05 for OGG$REP_N_01 started with pid=108, OS id=22864

2021-06-28T06:32:48.329577+00:00

APPLY OGG$REP_N_01: Apply User: OGG

2021-06-28T06:32:48.329670+00:00

APPLY OGG$REP_N_01: Apply Tag: 00

2021-06-28T06:32:48.330235+00:00

APPLY OGG$REP_N_01: Parameter Set by User: ALLOW_DUPLICATE_ROWS Value: Y

2021-06-28T06:32:48.330341+00:00

APPLY OGG$REP_N_01: Parameter Set by User: PRESERVE_ENCRYPTION Value: N

2021-06-28T06:32:48.330442+00:00

APPLY OGG$REP_N_01: Parameter Set by User: EAGER_SIZE Value: 15100

2021-06-28T06:32:48.330524+00:00

APPLY OGG$REP_N_01: Parameter Set by User: CDGRANULARITY Value: ROW

2021-06-28T06:32:48.330606+00:00

APPLY OGG$REP_N_01: Parameter Set by User: SUPPRESSTRIGGERS Value: Y

2021-06-28T06:32:48.330687+00:00

APPLY OGG$REP_N_01: Parameter Set by User: _GG_ERROR_HANDLING_MODE Value: ASYNC

2021-06-28T06:32:48.330907+00:00

APPLY OGG$REP_N_01: Parameter Set by User: PARALLELISM Value: 8

2021-06-28T06:32:48.893939+00:00

In ggserr.log

2021-06-28T06:32:26.092+0000  INFO    OGG-00994  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  REPLICAT REP_N_01 stopped normal

ly.

2021-06-28T06:32:26.141+0000  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): STOP REPLIC

AT REP_N_01.

2021-06-28T06:32:26.601+0000  ERROR   OGG-15163  Oracle GoldenGate Command Interpreter for Oracle:  There was a problem sending a messa

ge to REPLICAT REP_N_01 (Connection reset by peer).

2021-06-28T06:32:29.114+0000  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): STOP REPLIC

AT REP_N_01.

2021-06-28T06:32:33.465+0000  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): DELETE REPL

ICAT REP_N_01.

2021-06-28T06:32:34.340+0000  INFO    OGG-02529  Oracle GoldenGate Command Interpreter for Oracle:  Successfully unregistered REPLICAT

REP_N_01 inbound server OGG$REP_N_01 from database.

2021-06-28T06:32:34.343+0000  INFO    OGG-14034  Oracle GoldenGate Command Interpreter for Oracle:  Heartbeat entries with [REP_N_01] d

eleted.

2021-06-28T06:32:39.946+0000  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): ADD REPLICA

T REP_N_01  INTEGRATED EXTTRAIL /software/ogg/191/dirdat/01/in/en.

2021-06-28T06:32:43.378+0000  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): START REPLI

CAT REP_N_01.

2021-06-28T06:32:43.380+0000  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from GGSCI on host 10

.10.51.39:50218 (START REPLICAT REP_N_01 ).

2021-06-28T06:32:43.385+0000  INFO    OGG-00975  Oracle GoldenGate Manager for Oracle, mgr.prm:  REPLICAT REP_N_01 starting.

2021-06-28T06:32:43.804+0000  INFO    OGG-00995  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  REPLICAT REP_N_01 starting.

2021-06-28T06:32:43.804+0000  INFO    OGG-03059  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Operating system character set i

dentified as UTF-8.

2021-06-28T06:32:43.804+0000  INFO    OGG-02695  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  ANSI SQL parameter syntax is use

d for parameter parsing.

2021-06-28T06:32:43.804+0000  INFO    OGG-02095  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Successfully set environment variable ORACLE_SID=igt.

2021-06-28T06:32:43.804+0000  INFO    OGG-02095  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Successfully set environment variable ORACLE_HOME=/software/oracle/122.

2021-06-28T06:32:43.804+0000  INFO    OGG-02095  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Successfully set environment variable NLS_LANG=AMERICAN_AMERICA.AL32UTF8.

2021-06-28T06:32:43.944+0000  INFO    OGG-01360  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  REPLICAT is running in Integrated mode.

2021-06-28T06:32:44.513+0000  INFO    OGG-06451  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Triggers will be suppressed by default.

2021-06-28T06:32:44.524+0000  INFO    OGG-01815  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Virtual Memory Facilities for: COM

                                  anon alloc: mmap(MAP_ANON)  anon free: munmap

                                  file alloc: mmap(MAP_SHARED)  file free: munmap

                                  target directories:

                                  /software/ogg/191/dirtmp.

2021-06-28T06:32:44.535+0000  INFO    OGG-06604  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Database IGT CPU info: CPU Count 16, CPU Core Count 16, CPU Socket Count 2.

2021-06-28T06:32:44.618+0000  INFO    OGG-02545  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Parameter GROUPTRANSOPS is ignored by Integrated Replicat when parallelism is greater than 1.

2021-06-28T06:32:44.618+0000  INFO    OGG-02527  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Integrated Replicat does not populate a trace table.

2021-06-28T06:32:47.777+0000  INFO    OGG-02528  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  REPLICAT REP_N_01 successfully registered with database as inbound server OGG$REP_N_01.

2021-06-28T06:32:48.271+0000  INFO    OGG-00996  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  REPLICAT REP_N_01 started.

2021-06-28T06:32:49.439+0000  INFO    OGG-02530  Oracle GoldenGate Delivery for Oracle, rep_n_01.prm:  Integrated replicat successfully attached to inbound server OGG$REP_N_01.

===================

Starhome Stuff

===================

DBLOGIN USERID OGG PASSWORD rad21mif

I_01

STOP EXTRACT EXT_I_01

SEND EXTRACT EXT_I_01, FORCESTOP

UNREGISTER EXTRACT EXT_I_01 DATABASE 

DELETE EXTRACT EXT_I_01

REGISTER EXTRACT EXT_I_01 DATABASE

ADD EXTRACT EXT_I_01 INTEGRATED TRANLOG, BEGIN NOW

ADD EXTTRAIL /software/ogg/191/dirdat/01/out/ei EXTRACT EXT_I_01

START EXTRACT EXT_I_01

STOP EXTRACT DPM_I_01

SEND EXTRACT DPM_I_01, FORCESTOP

DELETE EXTRACT DPM_I_01

ADD EXTRACT DPM_I_01 EXTTRAILSOURCE /software/ogg/191/dirdat/01/out/ei

ADD RMTTRAIL /software/ogg/191/dirdat/01/in/ei EXTRACT DPM_I_01

START EXTRACT DPM_I_01

INFO DPM_I_01

STOP REPLICAT REP_I_01

SEND REPLICAT REP_I_01, FORCESTOP

DELETE REPLICAT REP_I_01

ADD REPLICAT REP_I_01, EXTTRAIL /software/ogg/191/dirdat/01/in/ei

START REPLICAT REP_I_01

S_01

STOP EXTRACT EXT_S_01
SEND EXTRACT EXT_S_01, FORCESTOP

UNREGISTER EXTRACT EXT_S_01 DATABASE 

DELETE EXTRACT EXT_S_01

REGISTER EXTRACT EXT_S_01 DATABASE

ADD EXTRACT EXT_S_01 INTEGRATED TRANLOG, BEGIN NOW

ADD EXTTRAIL /software/ogg/191/dirdat/01/out/es EXTRACT EXT_S_01

START EXTRACT EXT_S_01

STOP EXTRACT DPM_S_01

SEND EXTRACT DPM_S_01, FORCESTOP

DELETE EXTRACT DPM_S_01

ADD EXTRACT DPM_S_01 EXTTRAILSOURCE /software/ogg/191/dirdat/01/out/es

ADD RMTTRAIL /software/ogg/191/dirdat/01/in/es EXTRACT DPM_S_01

START EXTRACT DPM_S_01

STOP REPLICAT REP_S_01

SEND REPLICAT REP_S_01, FORCESTOP

DELETE REPLICAT REP_S_01

ADD REPLICAT REP_S_01, INTEGRATED EXTTRAIL /software/ogg/191/dirdat/01/in/es

START REPLICAT REP_S_01

P_01

STOP EXTRACT EXT_P_01

SEND EXTRACT EXT_P_01, FORCESTOP

UNREGISTER EXTRACT EXT_P_01 DATABASE 

DELETE EXTRACT EXT_P_01

REGISTER EXTRACT EXT_P_01 DATABASE

ADD EXTRACT EXT_P_01 INTEGRATED TRANLOG, BEGIN NOW

ADD EXTTRAIL /software/ogg/191/dirdat/01/out/ep EXTRACT EXT_P_01

START EXTRACT EXT_P_01

STOP EXTRACT DPM_P_01

SEND EXTRACT DPM_P_01, FORCESTOP

DELETE EXTRACT DPM_P_01

ADD EXTRACT DPM_P_01 EXTTRAILSOURCE /software/ogg/191/dirdat/01/out/ep

ADD RMTTRAIL /software/ogg/191/dirdat/01/in/ep EXTRACT DPM_P_01

START EXTRACT DPM_P_01

STOP REPLICAT REP_P_01

SEND REPLICAT REP_P_01, FORCESTOP

DELETE REPLICAT REP_P_01

ADD REPLICAT REP_P_01, INTEGRATED EXTTRAIL /software/ogg/191/dirdat/01/in/ep

START REPLICAT REP_P_01

===================

Check Status

===================

Golden Gate

==========================
Golden gate Checkpoint
==========================

Example:

GGSCI (my_server) 1> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:03
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:01
EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:04
EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:05
EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:01
EXTRACT     RUNNING     EXT_S_01    00:00:03      00:00:06
REPLICAT    RUNNING     REP_I_01    00:00:00      01:43:12
REPLICAT    RUNNING     REP_P_01    00:00:04      00:00:00
REPLICAT    RUNNING     REP_S_01    00:00:04      00:00:00

Checkpoint Files are under dirchk directory:

oracle@my_server:/software/ogg/191/dirchk>% ls -ltr

-rw-r----- 1 oracle dba  6144 Sep 29 01:00 EXT_I_01.cpb
-rw-r----- 1 oracle dba  6144 Sep 29 01:00 EXT_S_01.cpb
-rw-r----- 1 oracle dba  6144 Sep 29 01:00 EXT_P_01.cpb

-rw-r----- 1 oracle dba 53248 Sep 30 10:54 EXT_S_01.cpe
-rw-r----- 1 oracle dba 69632 Sep 30 10:54 EXT_I_01.cpe
-rw-r----- 1 oracle dba 71680 Sep 30 10:54 EXT_P_01.cpe

-rw-r----- 1 oracle dba 14336 Sep 30 10:54 DPM_S_01.cpe
-rw-r----- 1 oracle dba 14336 Sep 30 10:54 DPM_I_01.cpe
-rw-r----- 1 oracle dba 14336 Sep 30 10:54 DPM_P_01.cpe

-rw-r----- 1 oracle dba 16384 Sep 30 09:12 REP_I_01.cpr
-rw-r----- 1 oracle dba 14336 Sep 30 10:54 REP_P_01.cpr
-rw-r----- 1 oracle dba 16384 Sep 30 10:54 REP_S_01.cpr

Each process in Goldengate has its own checkpoint file which gets updated by default every 10 seconds, if the process is in running status.

Time since checkpoint is the time elapsed since the checkpoint file associated to process was updated.

Lag at checkpoint is the actual lag of the process. It is updated when the checkpoint file was updated.

REPLICAT is running slow

===========================
General
===========================

Replicat process is running slow. 
What to check?

Quick solution:

SEND REPLICAT REP_I_01 NOHANDLECOLLISIONS OWNER.TABLE 

===========================
Commands
===========================

GGSCI (ITMIL7DB00001) 1> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:01
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:02
EXTRACT     RUNNING     DPM_S_01    00:00:04      00:00:08
EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:07
EXTRACT     RUNNING     EXT_P_01    00:00:03      00:00:01
EXTRACT     RUNNING     EXT_S_01    00:00:03      00:00:01
REPLICAT    RUNNING     REP_I_01    83:00:54      03:24:01
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:00
REPLICAT    RUNNING     REP_S_01    00:00:05      00:00:01

GGSCI (ITMIL7DB00001) 12> SEND REPLICAT REP_I_01 STATUS

Sending STATUS request to REPLICAT REP_I_01 ...
  Current status: Processing data
  Sequence #: 1,046
  RBA: 205,827,294
  116,969 records in current transaction.

GGSCI (ITMIL7DB00001) 13> SEND REPLICAT REP_I_01 GETLAG

Sending GETLAG request to REPLICAT REP_I_01 ...
Last record lag 311,350 seconds.

311350/60/60~86:30

GGSCI (ITMIL7DB00001) 10> SEND REPLICAT REP_I_01 REPORT

Sending REPORT request to REPLICAT REP_I_01 ...

Request processed.

less REP_I_01.rpt

*********************************************************************

*                 ** Run Time Statistics **                         *

*********************************************************************

Last record for the last committed transaction is the following:

___________________________________________________________________

Trail name :  /software/ogg/191/dirdat/01/in/ei000001046

Hdr-Ind    :     E  (x45)     Partition  :     .  (x0c)

UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)

RecLength  :   512 (x0200)    IO Time    : 2020-09-05 06:29:53.999626

IOType     :   135  (x87)     OrigNode   :   255  (xff)

TransInd   :     .  (x02)     FormatType :     R  (x52)

SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)

AuditRBA   :     157412       AuditPos   : 52390468

Continued  :     N  (x00)     RecCount   :     1  (x01)

2020-09-05 06:29:53.999626 UnifiedPKUpdate    Len   512 RBA 136585873

TDR Index: 1

___________________________________________________________________

2020-09-08 20:56:06  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000001046, current RBA 205,657,982, 904,894 records, m_file_seqno = 1,046, m_file_rba = 205,658,574.

From Table OWNER.SGA_SUBSCRIBER_SFI to OWNER.SGA_SUBSCRIBER_SFI:

       #                   inserts:       100

       #                   updates:    788107

       #                   deletes:         0

       #                   upserts:         0

       #                  discards:         0

       #         update collisions:    394032

From Table OWNER.IPN_INVALIDATE to OWNER.IPN_INVALIDATE:

       #                   inserts:         4

       #                   updates:         0

       #                   deletes:         0

       #                   upserts:         0

       #                  discards:         0

       #                   ignores:        72

*********************************************************************

**                   Run Time Warnings                             **

*********************************************************************

Check if GG is using swap files, under dirtmp

dirtmp is the default location for storing transaction data when the size exceeds the memory size that is allocated for the cache manager. 

oracle@ITMIL7DB00001:/software/ogg/191/dirtmp>% ls -ltr

total 0

srwxr-x---. 1 oracle dba 0 Aug 31 21:39 DPM_I_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:39 EXT_I_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:40 DPM_P_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:40 EXT_P_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:41 REP_P_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:41 DPM_S_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:42 EXT_S_01.s

srwxr-x---. 1 oracle dba 0 Aug 31 21:42 REP_S_01.s

srwxr-x---. 1 oracle dba 0 Sep  8 11:05 REP_I_01.s

Check memory settings in parameter file

rep_i_01.prm

REPLICAT rep_i_01

setenv (ORACLE_SID="igt")

setenv (ORACLE_HOME="/software/oracle/122")

setenv (NLS_LANG = "AMERICAN_AMERICA.AL32UTF8")

ASSUMETARGETDEFS

USERID ogg, PASSWORD AACAAAAAAAAAAAIAZEDCPHICXGPEQCED ENCRYPTKEY DEFAULT

REPERROR (DEFAULT, IGNORE)

SOURCECHARSET PASSTHRU

CACHEMGR CACHESIZE 1024M

HANDLECOLLISIONS

DISCARDFILE /software/ogg/191/dirdat/01/in/disc01i.txt, append, MEGABYTES 2000

include ./dirprm/mapSFItables_01.inc

===========================
Resolution
===========================

Remove NOHANDLECOLLISIONS from Replicat configuration on both sites.
Checking output of STATS command, the replicat is 90% of the time, is busy parsing collisions.

Before NOHANDLECOLLISIONS:

GGSCI (ITMIL7DB00001) 30> SEND REPLICAT REP_I_01 STATS
Sending STATS request to REPLICAT REP_I_01 ...
Start of Statistics at 2020-09-11 20:49:03.
Replicating from OWNER.SGA_SUBSCRIBER_SFI to OWNER.SGA_SUBSCRIBER_SFI:
*** Total statistics since 2020-09-09 07:57:00 ***
        Total inserts                                   7047.00
        Total updates                              159072656.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                           159079703.00
        Total insert collisions                         1370.00
        Total update collisions                    159071473.00

*** Daily statistics since 2020-09-11 00:00:00 ***
        Total inserts                                   2657.00
        Total updates                               54334620.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                            54337277.00
        Total insert collisions                          365.00
        Total update collisions                     54333908.00

*** Hourly statistics since 2020-09-11 20:00:00 ***
        Total inserts                                    121.00
        Total updates                                2078586.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                             2078707.00
        Total insert collisions                           21.00
        Total update collisions                      2078554.00

*** Latest statistics since 2020-09-09 07:57:00 ***
        Total inserts                                   7047.00
        Total updates                              159072656.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                           159079703.00
        Total insert collisions                         1370.00
        Total update collisions                    159071473.00

Stop Collision Handling for table SGA_SUBSCRIBER_SFI  on both servers, A and B.

SEND REPLICAT REP_I_01 NOHANDLECOLLISIONS OWNER.SGA_SUBSCRIBER_SFI 

After NOHANDLECOLLISIONS

Replicating from OWNER.SGA_SUBSCRIBER_SFI to OWNER.SGA_SUBSCRIBER_SFI:

*** Total statistics since 2020-09-09 07:57:00 ***
        Total inserts                                   8091.00
        Total updates                              159072831.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                               46304442.00
        Total operations                           159080922.00
        Total insert collisions                         1370.00
        Total update collisions                    159071473.00

*** Daily statistics since 2020-09-12 00:00:00 ***
        Total inserts                                    384.00
        Total updates                                     37.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                               20684763.00
        Total operations                                 421.00

*** Hourly statistics since 2020-09-12 02:00:00 ***
        Total inserts                                     48.00
        Total updates                                      2.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                5417534.00
        Total operations                                  50.00

After some time the lag is resolved:

GGSCI (ITMIL7DB00001) 67> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:06
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:03
EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:02
EXTRACT     RUNNING     EXT_I_01    00:00:03      00:00:08
EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:01
EXTRACT     RUNNING     EXT_S_01    00:00:01      00:00:04
REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:05
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:03
REPLICAT    RUNNING     REP_S_01    00:00:02      00:00:00

===========================

Another scenario - no input file

===========================

GGSCI (qanfv-1-dbs-1a) 1> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:03

EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:01

EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:01

EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:02

EXTRACT     RUNNING     EXT_P_01    00:00:01      00:00:01

EXTRACT     RUNNING     EXT_S_01    00:00:01      00:00:05

REPLICAT    RUNNING     REP_I_01    109:56:59     29:07:15

REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:04

REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:01

GGSCI (qanfv-1-dbs-1a) 1> SEND REPLICAT REP_I_01 REPORT

Sending REPORT request to REPLICAT REP_I_01 ...

Request processed.

2020-09-30 09:03:58  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000016, current RBA 453,246,598, 127,887 records, m_file_seqno = 16, m_file_rba = 453,247,190.

Report at 2020-09-30 09:03:58 (activity since 2020-09-30 07:12:38)

No records were replicated.

ls -l /software/ogg/191/dirdat/01/in/ei000000016

ls: cannot access /software/ogg/191/dirdat/01/in/ei000000016: No such file or directory

first available file is:

/software/ogg/191/dirdat/01/in/ei000000243

GGSCI (qanfv-1-dbs-1a) 1> ALTER REPLICAT REP_I_01 extseqno 243 extrba 0

GGSCI (qanfv-1-dbs-1a) 8>  send REP_I_01 STATUS

Sending STATUS request to REPLICAT REP_I_01 ...

  Current status: Processing data

  Sequence #: 16

  RBA: 457,712,054

  135,430 records in current transaction.

STOP request pending end-of-transaction (135,430 records so far).

Need first to stop the REPLICAT

GGSCI (qanfv-1-dbs-1a) 8>  KILL REP_I_01

GGSCI (qanfv-1-dbs-1a) 16> ALTER REPLICAT REP_I_01 extseqno 243 extrba 0

2020-09-30 09:12:00  INFO    OGG-06594  Replicat REP_I_01 has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REP_I_01 with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.

GGSCI (qanfv-1-dbs-1a) 16> START REP_I_01

GGSCI (qanfv-1-dbs-1a) 19> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:02

EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:09

EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:10

EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:00

EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:09

EXTRACT     RUNNING     EXT_S_01    00:00:02      00:00:03

REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:33

REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:06

REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:04

Checking REP_I_01 Report in Intervals

2020-09-30 09:13:55  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000243, current RBA 642,943, 1,081 records, m_file_seqno = 243, m_file_rba = 643,535.

2020-09-30 09:16:34  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000243, current RBA 1,695,519, 2,859 records, m_file_seqno = 243, m_file_rba = 1,696,111.

=========================================

HANDLECOLLISIONS/NOHANDLECOLLISIONS

=========================================

Default behavior is NOHANDLECOLLISIONS

Enabling HANDLECOLLISIONS is done to address mismatch in data on two sites.

Golden Gate handles mismatch on below manner:

INSERT Collision

Insert in source whose key column exist on target is converted to update.
Instead of having duplicate insert, the insert is converted to Update.

UPDATE Collision

Updated in source but row not present in target.

- If the row with the old key is not found in the target, the DML is converted to an insert.
- If a row with the new key exists in the target, Replicat deletes the row that has the old key,    and the row with the new key is updated, effectively replacing the old row.

UPDATE Collision - Ignored

When Replicat encounters a missing-record error during an update that does not affect a key column, the change is discarded.

DELETE Collision
Deleted in source but row not present in target
Ignored

=========================================

Setting HANDLECOLLISIONS/NOHANDLECOLLISIONS value

=========================================

Option A - in REPLICAT parameter file + restart the replicat

Option B - As a parameter to SEND REPLICAT command:

SEND REPLICAT SOME_REPLICAT NOHANDLECOLLISIONS OWNER.TABLE_NAME

With Option B, no need for restart.

Additional Oracle Reference:
HANDLECOLLISIONS | NOHANDLECOLLISIONS

How to use GoldenGate HANDLECOLLISIONS Parameter correctly

=========================================

Plan to revive lagging REPLICAT on a single table

=========================================

1. stop GG

2. stop application

3. drop unique indexes

4. export from site A + import to site B with append mode

5. del duplicates

6. export from site B + import to site A with append mode

7. delete duplicates

8. enable  unique key

9. START REPLICATE XXX BEGIN NOW

10. START EXTRACT XXX BEGIN NOW

11. start application

===========================
Steps to reset Replicat to begin Now
===========================

GGSCI (qanfv-1-dbs-1a) 13> SEND REPLICAT REP_I_01 REPORT

from REP_I_01.rpt

2020-10-05 14:05:00  INFO    OGG-02232  Switching to next trail file /software/ogg/191/dirdat/01/in/ei000000762 at 2020-10-05 14:05:00.576477 due to EOF. with current RBA 499,999,812.

2020-10-05 14:05:21  INFO    OGG-01021  Command received from GGSCI: REPORT.

***********************************************************************

*                   ** Run Time Statistics **                         *

***********************************************************************

2020-10-05 14:05:21  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000762, current RBA 24,913,169, 106,074 records, m_file_seqno = 762, m_file_rba = 24,913,761.

Report at 2020-10-05 14:05:21 (activity since 2020-10-05 14:04:20)

No records were replicated.

GGSCI (qanfv-1-dbs-1a) 13> ALTER REPLICAT REP_I_01, BEGIN NOW

ERROR: REPLICAT REP_I_01 is running and cannot be altered (1,2,No such file or directory).

GGSCI (qanfv-1-dbs-1a) 14> STOP REPLICAT REP_I_01

Sending STOP request to REPLICAT REP_I_01 ...

STOP request pending end-of-transaction (199,394 records so far).

GGSCI (qanfv-1-dbs-1a) 14> KILL REPLICAT REP_I_01

Killed process (31188) for REPLICAT REP_I_01

GGSCI (qanfv-1-dbs-1a) 16> ALTER REPLICAT REP_I_01, BEGIN NOW

2020-10-05 14:10:53  INFO    OGG-06594  Replicat REP_I_01 has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REP_I_01 with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.

GGSCI (qanfv-1-dbs-1a) 17> START REPLICAT REP_I_01

Sending START request to MANAGER ...

REPLICAT REP_I_01 starting

GGSCI (qanfv-1-dbs-1a) 18> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:09

EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:02

EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:09

EXTRACT     RUNNING     EXT_I_01    00:00:00      00:00:00

EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:05

EXTRACT     RUNNING     EXT_S_01    00:00:01      00:00:05

REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:06

REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:03

REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:06

Alternative:

Use in REPLICATE prm file INSERTMISSINGUPDATES

Limitation:
It can be used only if Extract logs all columns, and not just the changed colums + PK columns

In Extract:

To have all columns logged, the Extract should have this setting:

ADD SCHEMATRANDATA <MY_SCHEMA> ALLCOLS

In Replicat:

Edit replicat prm file

INSERTMISSINGUPDATES

MAP MY_SCHEMA.*, target MY_SCHEMA.*;

There is an option to reload parameters files, without restart:
REFRESH REPLICAT <EXTRACT GROUP>

REFRESH EXTRACT <EXTRACT GROUP>