======================
General
======================
The issue:
Data not replicating from server_39 to server_38
Replicat is showing: "No active replication maps."
======================
Details
======================
MANAGER RUNNING
EXTRACT RUNNING DPM_I_01 00:00:00 00:00:09
EXTRACT RUNNING DPM_P_01 00:00:00 00:00:04
EXTRACT RUNNING DPM_S_01 00:00:03 00:00:09
EXTRACT RUNNING EXT_I_01 00:00:01 00:00:09
EXTRACT RUNNING EXT_P_01 00:00:02 00:00:04
EXTRACT RUNNING EXT_S_01 00:00:01 00:00:09
REPLICAT RUNNING REP_I_01 00:00:00 00:00:01
REPLICAT RUNNING REP_P_01 00:00:00 00:00:00
REPLICAT RUNNING REP_S_01 00:00:04 00:00:03
=================================================================
Checking the EXTRACT on server 39, looks OK:
SEND EXTRACT EXT_I_01 STATS
Extracting from MY_USER.IPN_INVALIDATE_TRIGGER to MY_USER.IPN_INVALIDATE_TRIGGER:
*** Hourly statistics since 2020-11-17 11:00:00 ***
Total inserts 1.00
Total updates 0.00
Total deletes 1.00
Total upserts 0.00
Total discards 0.00
Total operations 2.00
But on server 38, the REPLICAT is not getting any input:
GGSCI (server_38) 2> SEND REPLICAT REP_I_01 STATS
Sending STATS request to REPLICAT REP_I_01 ...
No active replication maps.
check input files on 38
Nothing is coming for process for REPICAT REP_I_01
-rw-r----- 1 oracle dba 0 Oct 29 09:48 ei000000000
-rw-r----- 1 oracle dba 20633 Nov 16 14:25 disc01i.txt
-rw-r----- 1 oracle dba 58431 Nov 16 14:25 disc01s.txt
-rw-r----- 1 oracle dba 13947 Nov 16 14:25 disc01p.txt
-rw-r----- 1 oracle dba 21743643 Nov 17 11:47 es000000003
-rw-r----- 1 oracle dba 21219127 Nov 17 11:47 ep000000003
GGSCI (server_38) 5> INFO REP_I_01
REPLICAT REP_I_01 Last Started 2020-11-17 11:55 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:08 ago)
Process ID 15605
Log Read Checkpoint File /software/ogg/191/dirdat/01/in/ei000000001
First Record RBA 0
REPLICAT is looking for flie /software/ogg/191/dirdat/01/in/ei000000001
But this file does not exists.
But this file does not exists.
Checking datapump on 39 server
GGSCI (server_39) 6> SEND EXTRACT DPM_I_01 STATS
Sending STATS request to EXTRACT DPM_I_01 ...
No active extraction maps.
The Extract definition:
ADD EXTRACT DPM_I_01 EXTTRAILSOURCE $GG_HOME/dirdat/01/out/ei
ADD RMTTRAIL $GG_HOME/dirdat/01/in/ei EXTRACT DPM_I_01
START EXTRACT DPM_I_01
Seems that the Datapump on server 39 is not getting input.
Thus nothing is sent to server 38.
DROP + CREATE Datapump - Does not work
To drop + recreate, can create do following steps, but it does not help:
To drop EXTRACT
DBLOGIN USERID OGG, PASSWORD YYY
STOP EXTRACT DPM_I_01
DELETE EXTRACT DPM_I_01
ADD EXTRACT DPM_I_01 EXTTRAILSOURCE /software/ogg/191/dirdat/01/out/ei
ADD RMTTRAIL /software/ogg/191/dirdat/01/in/ei EXTRACT DPM_I_01
GGSCI (server_39 as ogg@orainst) 10> INFO DPM_I_01
EXTRACT DPM_I_01 Last Started 2020-11-17 12:50 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:08 ago)
Process ID 18315
Log Read Checkpoint File /software/ogg/191/dirdat/01/out/ei000000000
First Record RBA 0
======================
Solution
======================
ETROLLOVER to Extract + Update to Datapump
DBLOGIN USERID XXX, PASSWORD YYY
ALTER EXTRACT EXT_I_01, ETROLLOVER
ALTER EXTRACT DPM_I_01, EXTSEQNO 10, EXTRBA 0
GGSCI (server_39 as ogg@orainst) 4> STOP EXTRACT EXT_I_01
Sending STOP request to EXTRACT EXT_I_01 ...
Request processed.
GGSCI (server_39 as ogg@orainst) 6> ALTER EXTRACT EXT_I_01, ETROLLOVER
2020-11-17 14:21:01 INFO OGG-01520 Rollover performed.
For each affected output trail of Version 10 or higher format,
after starting the source extract,
issue ALTER EXTSEQNO for that trail's reader
(either pump EXTRACT or REPLICAT) to move the reader's scan to the new trail file;
it will not happen automatically.
EXTRACT altered.
GGSCI (server_39 as ogg@orainst) 12> INFO EXT_I_01 DETAIL
EXTRACT EXT_I_01 Initialized 2020-11-16 14:26 Status RUNNING
Checkpoint Lag 00:00:03 (updated 00:02:50 ago)
Process ID 27583
Log Read Checkpoint Oracle Integrated Redo Logs
2020-11-17 14:20:37
SCN 2.949429219 (9539363811)
Target Extract Trails:
Trail Name Seqno RBA Max MB Trail Type
/software/ogg/191/dirdat/01/out/ei 2506 0 500 EXTTRAIL
GGSCI (server_39 as ogg@orainst) 14> STOP EXTRACT DPM_I_01
GGSCI (server_39 as ogg@orainst) 15> ALTER EXTRACT DPM_I_01, EXTSEQNO 2506 EXTRBA 0
EXTRACT altered.
GGSCI (server_39 as ogg@orainst) 17> START DPM_I_01
Sending START request to MANAGER ...
EXTRACT DPM_I_01 starting
GGSCI (server_39 as ogg@orainst) 18> INFO DPM_I_01
EXTRACT DPM_I_01 Last Started 2020-11-17 14:24 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:03 ago)
Process ID 27776
Log Read Checkpoint File /software/ogg/191/dirdat/01/out/ei000002506
First Record RBA 0
Now Datapump is advancing.
GGSCI (server_39 as ogg@orainst) 19> INFO DPM_I_01
EXTRACT DPM_I_01 Last Started 2020-11-17 14:24 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:04 ago)
Process ID 27776
Log Read Checkpoint File /software/ogg/191/dirdat/01/out/ei000002506
2020-11-17 14:25:07.000000 RBA 8905
on server 38
Now the file on the remote server is being created: ei000000000
oracle@server_38:/software/ogg/191/dirdat/01/in>% ls -ltr
total 45884
-rw-r----- 1 oracle dba 58431 Nov 16 14:25 disc01s.txt
-rw-r----- 1 oracle dba 13947 Nov 16 14:25 disc01p.txt
-rw-r----- 1 oracle dba 20850 Nov 17 11:55 disc01i.txt
-rw-r----- 1 oracle dba 22915034 Nov 17 14:28 es000000003
-rw-r----- 1 oracle dba 13214 Nov 17 14:28 ei000000000
-rw-r----- 1 oracle dba 23709458 Nov 17 14:28 ep000000003
================================
Next step -Troubleshoot REPLICAT on server 38
================================
GGSCI (server_38) 2> SEND REPLICAT REP_I_01 STATS
Sending STATS request to REPLICAT REP_I_01 ...
No active replication maps.
GGSCI (server_38) 8> INFO REP_I_01
REPLICAT REP_I_01 Last Started 2020-11-17 14:29 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:03 ago)
Process ID 18863
Log Read Checkpoint File /software/ogg/191/dirdat/01/in/ei000000001
First Record RBA 0
But...
The actual file is : /software/ogg/191/dirdat/01/in/ei000000000
Need manually to set REPLICAT to look into the correct file
GGSCI (server_38) 10> STOP REPLICAT REP_I_01
Sending STOP request to REPLICAT REP_I_01 ...
Request processed.
GGSCI (server_38) 15> ALTER REPLICAT REP_I_01, EXTSEQNO 0
2020-11-17 14:35:09 INFO OGG-06594 Replicat REP_I_01 has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REP_I_01 with NOFILTERDUPTRANSACTIONS option.
REPLICAT altered.
GGSCI (server_38) 16> START REPLICAT REP_I_01
Sending START request to MANAGER ...
REPLICAT REP_I_01 starting
GGSCI (server_38) 17> INFO REPLICAT REP_I_01
REPLICAT REP_I_01 Last Started 2020-11-17 14:35 Status RUNNING
Checkpoint Lag 00:00:15 (updated 00:00:01 ago)
Process ID 32390
Log Read Checkpoint File /software/ogg/191/dirdat/01/in/ei000000000
2020-11-17 14:35:07.001959 RBA 20771
Now, REPICAT is reading records.
GGSCI (server_38) 45> SEND REPLICAT REP_I_01 STATS
Sending STATS request to REPLICAT REP_I_01 ...
Start of Statistics at 2020-11-17 14:36:08.
Replicating from OGG.GG_HEARTBEAT_SEED to OGG.GG_HEARTBEAT:
*** Total statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
*** Daily statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
*** Hourly statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
*** Latest statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
Replicating from OGG.GG_HEARTBEAT_SEED to OGG.GG_HEARTBEAT_HISTORY:
*** Total statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
*** Daily statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
*** Hourly statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
*** Latest statistics since 2020-11-17 14:35:21 ***
Total inserts 0.00
Total updates 15.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total operations 15.00
Replicating from MY_USER.IPN_INVALIDATE_TRIGGER to MY_USER.IPN_INVALIDATE_TRIGGER:
Total inserts 1.00
Total updates 0.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total ignores 2.00
Total operations 1.00
*** Daily statistics since 2020-11-17 14:35:21 ***
Total inserts 1.00
Total updates 0.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total ignores 2.00
Total operations 1.00
*** Hourly statistics since 2020-11-17 14:35:21 ***
Total inserts 1.00
Total updates 0.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total ignores 2.00
Total operations 1.00
*** Latest statistics since 2020-11-17 14:35:21 ***
Total inserts 1.00
Total updates 0.00
Total deletes 0.00
Total upserts 0.00
Total discards 0.00
Total ignores 2.00
Total operations 1.00
End of Statistics.
Now Replicat is finally getting input from the other server.
Issue Fixed!
All processes are working OK
GGSCI (server_38) 2> INFO ALL
*** Total statistics since 2020-11-17 14:35:21 ***
