Pages

Sunday, September 13, 2020

REPLICAT is running slow

===========================
General
===========================

Replicat process is running slow. 
What to check?

Quick solution:

SEND REPLICAT REP_I_01 NOHANDLECOLLISIONS OWNER.TABLE 


===========================
Commands
===========================

GGSCI (ITMIL7DB00001) 1> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:01
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:02
EXTRACT     RUNNING     DPM_S_01    00:00:04      00:00:08
EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:07
EXTRACT     RUNNING     EXT_P_01    00:00:03      00:00:01
EXTRACT     RUNNING     EXT_S_01    00:00:03      00:00:01
REPLICAT    RUNNING     REP_I_01    83:00:54      03:24:01
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:00
REPLICAT    RUNNING     REP_S_01    00:00:05      00:00:01


GGSCI (ITMIL7DB00001) 12> SEND REPLICAT REP_I_01 STATUS

Sending STATUS request to REPLICAT REP_I_01 ...
  Current status: Processing data
  Sequence #: 1,046
  RBA: 205,827,294
  116,969 records in current transaction.


GGSCI (ITMIL7DB00001) 13> SEND REPLICAT REP_I_01 GETLAG

Sending GETLAG request to REPLICAT REP_I_01 ...
Last record lag 311,350 seconds.

311350/60/60~86:30


GGSCI (ITMIL7DB00001) 10> SEND REPLICAT REP_I_01 REPORT

Sending REPORT request to REPLICAT REP_I_01 ...
Request processed.

less REP_I_01.rpt
*********************************************************************
*                 ** Run Time Statistics **                         *
*********************************************************************

Last record for the last committed transaction is the following:
___________________________________________________________________
Trail name :  /software/ogg/191/dirdat/01/in/ei000001046
Hdr-Ind    :     E  (x45)     Partition  :     .  (x0c)
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)
RecLength  :   512 (x0200)    IO Time    : 2020-09-05 06:29:53.999626
IOType     :   135  (x87)     OrigNode   :   255  (xff)
TransInd   :     .  (x02)     FormatType :     R  (x52)
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)
AuditRBA   :     157412       AuditPos   : 52390468
Continued  :     N  (x00)     RecCount   :     1  (x01)

2020-09-05 06:29:53.999626 UnifiedPKUpdate    Len   512 RBA 136585873
TDR Index: 1
___________________________________________________________________


2020-09-08 20:56:06  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000001046, current RBA 205,657,982, 904,894 records, m_file_seqno = 1,046, m_file_rba = 205,658,574.

From Table OWNER.SGA_SUBSCRIBER_SFI to OWNER.SGA_SUBSCRIBER_SFI:
       #                   inserts:       100
       #                   updates:    788107
       #                   deletes:         0
       #                   upserts:         0
       #                  discards:         0
       #         update collisions:    394032
From Table OWNER.IPN_INVALIDATE to OWNER.IPN_INVALIDATE:
       #                   inserts:         4
       #                   updates:         0
       #                   deletes:         0
       #                   upserts:         0
       #                  discards:         0
       #                   ignores:        72


*********************************************************************
**                   Run Time Warnings                             **
*********************************************************************

Check if GG is using swap files, under dirtmp
dirtmp is the default location for storing transaction data when the size exceeds the memory size that is allocated for the cache manager. 

oracle@ITMIL7DB00001:/software/ogg/191/dirtmp>% ls -ltr
total 0
srwxr-x---. 1 oracle dba 0 Aug 31 21:39 DPM_I_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:39 EXT_I_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:40 DPM_P_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:40 EXT_P_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:41 REP_P_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:41 DPM_S_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:42 EXT_S_01.s
srwxr-x---. 1 oracle dba 0 Aug 31 21:42 REP_S_01.s
srwxr-x---. 1 oracle dba 0 Sep  8 11:05 REP_I_01.s


Check memory settings in parameter file
rep_i_01.prm
REPLICAT rep_i_01

setenv (ORACLE_SID="igt")
setenv (ORACLE_HOME="/software/oracle/122")
setenv (NLS_LANG = "AMERICAN_AMERICA.AL32UTF8")

ASSUMETARGETDEFS
USERID ogg, PASSWORD AACAAAAAAAAAAAIAZEDCPHICXGPEQCED ENCRYPTKEY DEFAULT

REPERROR (DEFAULT, IGNORE)

SOURCECHARSET PASSTHRU
CACHEMGR CACHESIZE 1024M
HANDLECOLLISIONS
DISCARDFILE /software/ogg/191/dirdat/01/in/disc01i.txt, append, MEGABYTES 2000
include ./dirprm/mapSFItables_01.inc

===========================
Resolution
===========================

Remove NOHANDLECOLLISIONS from Replicat configuration on both sites.
Checking output of STATS command, the replicat is 90% of the time, is busy parsing collisions.

Before NOHANDLECOLLISIONS:

GGSCI (ITMIL7DB00001) 30> SEND REPLICAT REP_I_01 STATS
Sending STATS request to REPLICAT REP_I_01 ...
Start of Statistics at 2020-09-11 20:49:03.
Replicating from OWNER.SGA_SUBSCRIBER_SFI to OWNER.SGA_SUBSCRIBER_SFI:
*** Total statistics since 2020-09-09 07:57:00 ***
        Total inserts                                   7047.00
        Total updates                              159072656.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                           159079703.00
        Total insert collisions                         1370.00
        Total update collisions                    159071473.00

*** Daily statistics since 2020-09-11 00:00:00 ***
        Total inserts                                   2657.00
        Total updates                               54334620.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                            54337277.00
        Total insert collisions                          365.00
        Total update collisions                     54333908.00

*** Hourly statistics since 2020-09-11 20:00:00 ***
        Total inserts                                    121.00
        Total updates                                2078586.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                             2078707.00
        Total insert collisions                           21.00
        Total update collisions                      2078554.00

*** Latest statistics since 2020-09-09 07:57:00 ***
        Total inserts                                   7047.00
        Total updates                              159072656.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                   9701.00
        Total operations                           159079703.00
        Total insert collisions                         1370.00
        Total update collisions                    159071473.00


Stop Collision Handling for table SGA_SUBSCRIBER_SFI  on both servers, A and B.

SEND REPLICAT REP_I_01 NOHANDLECOLLISIONS OWNER.SGA_SUBSCRIBER_SFI 

After NOHANDLECOLLISIONS

Replicating from OWNER.SGA_SUBSCRIBER_SFI to OWNER.SGA_SUBSCRIBER_SFI:

*** Total statistics since 2020-09-09 07:57:00 ***
        Total inserts                                   8091.00
        Total updates                              159072831.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                               46304442.00
        Total operations                           159080922.00
        Total insert collisions                         1370.00
        Total update collisions                    159071473.00

*** Daily statistics since 2020-09-12 00:00:00 ***
        Total inserts                                    384.00
        Total updates                                     37.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                               20684763.00
        Total operations                                 421.00

*** Hourly statistics since 2020-09-12 02:00:00 ***
        Total inserts                                     48.00
        Total updates                                      2.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total ignores                                5417534.00
        Total operations                                  50.00

After some time the lag is resolved:

GGSCI (ITMIL7DB00001) 67> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:06
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:03
EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:02
EXTRACT     RUNNING     EXT_I_01    00:00:03      00:00:08
EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:01
EXTRACT     RUNNING     EXT_S_01    00:00:01      00:00:04
REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:05
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:03
REPLICAT    RUNNING     REP_S_01    00:00:02      00:00:00


===========================
Another scenario - no input file
===========================
GGSCI (qanfv-1-dbs-1a) 1> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:03
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:01
EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:01
EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:02
EXTRACT     RUNNING     EXT_P_01    00:00:01      00:00:01
EXTRACT     RUNNING     EXT_S_01    00:00:01      00:00:05
REPLICAT    RUNNING     REP_I_01    109:56:59     29:07:15
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:04
REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:01

GGSCI (qanfv-1-dbs-1a) 1> SEND REPLICAT REP_I_01 REPORT

Sending REPORT request to REPLICAT REP_I_01 ...
Request processed.


2020-09-30 09:03:58  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000016, current RBA 453,246,598, 127,887 records, m_file_seqno = 16, m_file_rba = 453,247,190.

Report at 2020-09-30 09:03:58 (activity since 2020-09-30 07:12:38)

No records were replicated.


ls -l /software/ogg/191/dirdat/01/in/ei000000016
ls: cannot access /software/ogg/191/dirdat/01/in/ei000000016: No such file or directory


first available file is:
/software/ogg/191/dirdat/01/in/ei000000243


GGSCI (qanfv-1-dbs-1a) 1> ALTER REPLICAT REP_I_01 extseqno 243 extrba 0


GGSCI (qanfv-1-dbs-1a) 8>  send REP_I_01 STATUS

Sending STATUS request to REPLICAT REP_I_01 ...
  Current status: Processing data
  Sequence #: 16
  RBA: 457,712,054
  135,430 records in current transaction.
STOP request pending end-of-transaction (135,430 records so far).

Need first to stop the REPLICAT

GGSCI (qanfv-1-dbs-1a) 8>  KILL REP_I_01

GGSCI (qanfv-1-dbs-1a) 16> ALTER REPLICAT REP_I_01 extseqno 243 extrba 0

2020-09-30 09:12:00  INFO    OGG-06594  Replicat REP_I_01 has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REP_I_01 with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.

GGSCI (qanfv-1-dbs-1a) 16> START REP_I_01

GGSCI (qanfv-1-dbs-1a) 19> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:02
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:09
EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:10
EXTRACT     RUNNING     EXT_I_01    00:00:02      00:00:00
EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:09
EXTRACT     RUNNING     EXT_S_01    00:00:02      00:00:03
REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:33
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:06
REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:04

Checking REP_I_01 Report in Intervals

2020-09-30 09:13:55  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000243, current RBA 642,943, 1,081 records, m_file_seqno = 243, m_file_rba = 643,535.

2020-09-30 09:16:34  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000243, current RBA 1,695,519, 2,859 records, m_file_seqno = 243, m_file_rba = 1,696,111.



=========================================
HANDLECOLLISIONS/NOHANDLECOLLISIONS
=========================================
Default behavior is NOHANDLECOLLISIONS

Enabling HANDLECOLLISIONS is done to address mismatch in data on two sites.
Golden Gate handles mismatch on below manner:

INSERT Collision
Insert in source whose key column exist on target is converted to update.
Instead of having duplicate insert, the insert is converted to Update.

UPDATE Collision
Updated in source but row not present in target.
- If the row with the old key is not found in the target, the DML is converted to an insert.
- If a row with the new key exists in the target, Replicat deletes the row that has the old key,    and the row with the new key is updated, effectively replacing the old row.

UPDATE Collision - Ignored
When Replicat encounters a missing-record error during an update that does not affect a key column, the change is discarded.

DELETE Collision
Deleted in source but row not present in target
Ignored

=========================================
Setting HANDLECOLLISIONS/NOHANDLECOLLISIONS value
=========================================

Option A - in REPLICAT parameter file + restart the replicat

Option B - As a parameter to SEND REPLICAT command:
SEND REPLICAT SOME_REPLICAT NOHANDLECOLLISIONS OWNER.TABLE_NAME

With Option B, no need for restart.

Additional Oracle Reference:
HANDLECOLLISIONS | NOHANDLECOLLISIONS

=========================================
Plan to revive lagging REPLICAT on a single table
=========================================
1. stop GG
2. stop application
3. drop unique indexes
4. export from site A + import to site B with append mode
5. del duplicates
6. export from site B + import to site A with append mode
7. delete duplicates
8. enable  unique key
9. START REPLICATE XXX BEGIN NOW
10. START EXTRACT XXX BEGIN NOW
11. start application



===========================
Steps to reset Replicat to begin Now
===========================
GGSCI (qanfv-1-dbs-1a) 13> SEND REPLICAT REP_I_01 REPORT

from REP_I_01.rpt
2020-10-05 14:05:00  INFO    OGG-02232  Switching to next trail file /software/ogg/191/dirdat/01/in/ei000000762 at 2020-10-05 14:05:00.576477 due to EOF. with current RBA 499,999,812.

2020-10-05 14:05:21  INFO    OGG-01021  Command received from GGSCI: REPORT.

***********************************************************************
*                   ** Run Time Statistics **                         *
***********************************************************************


2020-10-05 14:05:21  INFO    OGG-02333  Reading /software/ogg/191/dirdat/01/in/ei000000762, current RBA 24,913,169, 106,074 records, m_file_seqno = 762, m_file_rba = 24,913,761.

Report at 2020-10-05 14:05:21 (activity since 2020-10-05 14:04:20)

No records were replicated.



GGSCI (qanfv-1-dbs-1a) 13> ALTER REPLICAT REP_I_01, BEGIN NOW
ERROR: REPLICAT REP_I_01 is running and cannot be altered (1,2,No such file or directory).

GGSCI (qanfv-1-dbs-1a) 14> STOP REPLICAT REP_I_01

Sending STOP request to REPLICAT REP_I_01 ...
STOP request pending end-of-transaction (199,394 records so far).


GGSCI (qanfv-1-dbs-1a) 14> KILL REPLICAT REP_I_01
Killed process (31188) for REPLICAT REP_I_01

GGSCI (qanfv-1-dbs-1a) 16> ALTER REPLICAT REP_I_01, BEGIN NOW

2020-10-05 14:10:53  INFO    OGG-06594  Replicat REP_I_01 has been altered. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REP_I_01 with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.


GGSCI (qanfv-1-dbs-1a) 17> START REPLICAT REP_I_01

Sending START request to MANAGER ...
REPLICAT REP_I_01 starting

GGSCI (qanfv-1-dbs-1a) 18> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     DPM_I_01    00:00:00      00:00:09
EXTRACT     RUNNING     DPM_P_01    00:00:00      00:00:02
EXTRACT     RUNNING     DPM_S_01    00:00:00      00:00:09
EXTRACT     RUNNING     EXT_I_01    00:00:00      00:00:00
EXTRACT     RUNNING     EXT_P_01    00:00:02      00:00:05
EXTRACT     RUNNING     EXT_S_01    00:00:01      00:00:05
REPLICAT    RUNNING     REP_I_01    00:00:00      00:00:06
REPLICAT    RUNNING     REP_P_01    00:00:00      00:00:03
REPLICAT    RUNNING     REP_S_01    00:00:00      00:00:06


Alternative:
Use in REPLICATE prm file INSERTMISSINGUPDATES

Limitation:
It can be used only if Extract logs all columns, and not just the changed colums + PK columns

In Extract:
To have all columns logged, the Extract should have this setting:
ADD SCHEMATRANDATA <MY_SCHEMA> ALLCOLS

In Replicat:
Edit replicat prm file
INSERTMISSINGUPDATES
MAP MY_SCHEMA.*, target MY_SCHEMA.*;

There is an option to reload parameters files, without restart:
REFRESH REPLICAT <EXTRACT GROUP>
REFRESH EXTRACT <EXTRACT GROUP>

Wednesday, August 19, 2020

ORA-29972: user does not have privilege to change/ create registration

===============
Issue:
===============

login an dba privileged user and from existing user:

SQL> DROP USER MY_USER CASCADE;

DROP USER MY_USER CASCADE

*

ERROR at line 1:

ORA-29972: user does not have privilege to change/ create registration


===============
Details
===============

In Oracle 12 there is a new feature Change Notification. 
This is a system privilege.
For some reason, you need to revoke this privilege before you can drop the user.

===============
Solution:
===============
SQL> REVOKE CHANGE NOTIFICATION FROM MY_USER;

Revoke succeeded.

SQL> DROP USER MY_USER CASCADE;

User dropped.

Tuesday, July 21, 2020

WARNING: too many parse errors - Oracle 12.2 has a different way to alarm about failing SQL

Oracle 12.2 has a different way to alarm about failing SQL

Rather than fail the SQL with error message, as in an older oracle versions, Oracle is writing a WARNING about parsing a SQL statement to alert.log each time the SQL is executed

And a warning counter is advanced by 100

For example:

WARNING: too many parse errors, count=100 SQL hash=0x4c8ca22e
PARSE ERROR: ospid=13813, error=904 for statement:
2020-07-09T05:11:16.653942+00:00
INSERT INTO ...

WARNING: too many parse errors, count=200 SQL hash=0x4c8ca22e
PARSE ERROR: ospid=13813, error=904 for statement:
2020-07-09T05:12:16.597150+00:00
INSERT INTO ...

The actual error is here: error=904
Which is in this case ORA-00904 - Invalid identifier

This is might be quite confusing, since application is now not seeing any error while executing.

Another Example:
alert.log
2021-02-17T19:34:05.126304+00:00
WARNING: too many parse errors, count=82600 SQL hash=0xf5112286
PARSE ERROR: ospid=14663, error=604 for statement: 
2021-02-17T19:34:05.126406+00:00
MERGE into XXX
Additional information: hd=0x9e5afac0 phd=0x9f458c28 flg=0x110676 cisid=88 sid=88 ciuid=88 uid=88 sqlid=6zjj7hruj28n6
...Current username=PRT_MEOQQ_SPARX
...Application: JDBC Thin Client Action: 

Details re inside trc file:
 
igt_ora_14663.trc
-rw-r----- 1 oracle dba 2955331 Feb 17 19:28 igt_ora_14663.trc
kgscGet-exception: cur=0x7f4cfab81420 cob=0x7f4cfaaab028 err=604
KQRCMT: Write failed with error=1000 po=0x9d8ee970 cid=2
diagnostics : cid=2 hash=0x7bef3983 flag=0x2a
kgscGet-exception: cur=0x7f4cfab83450 cob=0x7f4cfaaa6a98 err=604

KQRCMT: Write failed with error=1000 stands for ORA-01000: maximum open cursors exceeded.

Wednesday, May 27, 2020

Code Example: awk, bash, sqlplus, plsql. Grant Permissions from schema A to schema B from bash. Get List of exported tables

============================
General
============================
Scenario:
There are 2 schemas. CRM and Reports.
Automatically execute Grants to Reports schema from bash script
The Reports schema optionally defined in sql parameters file.
If it is not defined, use a dynamic default name.


============================
Code
============================
sync_reports_permisions.sh
sync_reports_permisions.sql


sync_reports_permisions.sh
#!/bin/bash
REPORTS_USER=`grep REPORTS_USER ../set_vipuser.sql |awk -F= '{print $2}'`
if [[ -z $REPORTS_USER ]]; then
  REPORTS_USER=XXX
fi

sqlplus -S  /nolog @./post_upgrade/sync_reports_permisions.sql ${REPORTS_USER} 
 

sync_reports_permisions.sql
@../set_vipuser.sql
CONNECT &&vipuser/&&vippass@&&connectstr
spool V710_upgrade.log append
SET FEEDBACK OFF
SET SERVEROUTPUT ON
SET VERIFY OFF

PROMPT 
PROMPT =================================================
PROMPT Starting Optional Step of Grant Permissions to Reports Schema


DECLARE
  v_sql    VARCHAR2(4000); 
BEGIN
  v_sql := 'DROP TABLE TEMP_GRANTS_LIST';
  EXECUTE IMMEDIATE v_sql;
EXCEPTION
  WHEN OTHERS THEN
    NULL;
END;
/

DECLARE
  v_sql    VARCHAR2(4000);
BEGIN
  v_sql := 'CREATE TABLE TEMP_GRANTS_LIST(table_name VARCHAR2(30)) TABLESPACE IGT_TABLE';
  EXECUTE IMMEDIATE v_sql;
EXCEPTION
  WHEN OTHERS THEN
    NULL;
END;
/

DECLARE 
  v_sql                VARCHAR2(2000);
  v_input_rep_user     VARCHAR2(30); 
  v_rep_user           VARCHAR2(30);
  v_counter            NUMBER(2);
BEGIN  
  BEGIN    
    SELECT '&&1' INTO v_input_rep_user FROM DUAL;
    IF v_input_rep_user = 'XXX' THEN
      SELECT USER||'_REPORTS' INTO v_rep_user FROM DUAL;
    ELSE
      v_rep_user := v_input_rep_user;
    END IF;

    IF v_input_rep_user = 'XXX' THEN 
      DBMS_OUTPUT.put_line('REPORTS_USER Parameter was NOT provided in set_vipuser.sql');
      DBMS_OUTPUT.put_line('Working with Defaul User: '||v_rep_user);
    ELSE
      DBMS_OUTPUT.put_line('REPORTS_USER was provided in set_vipuser.sql: '||v_rep_user);
    END IF; 

  EXCEPTION
    WHEN OTHERS THEN
      RAISE;
  END;  

  BEGIN
    SELECT COUNT (*) INTO v_counter FROM ALL_USERS WHERE username = v_rep_user;
    IF v_counter = 0 THEN
       
      DBMS_OUTPUT.put_line('REPORTS User '||v_rep_user||' Does Not Exists');
      DBMS_OUTPUT.put_line('Optional Step Of Grant Permissions to Reports Schema is Terminated');
      RETURN;
    END IF;
  END;

  DBMS_OUTPUT.put_line('Granting Permissions to REPORTS Schema '||v_rep_user);

  BEGIN
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_NW_TYPES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_IBR_CELL_REPORT_DATA');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_IBR_CELL_REPORT_DATA');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('DEBUG_GEN_W_SDR');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('DEBUG_GEN_W_SDR');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_PREFERRED_MODE_TYPES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('DB_INSTALLED_VERSIONS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GSM_COUNTRIES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GSM_NETWORKS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GSM_IMSI_PREFIXES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('SGA_OUT_COMM');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('PRIVATE_COMM');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GA_ACCOUNTS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('OVMD_SERVICE_PROPERTY_VALUES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_SERVICE_PROPERTIES_VALUES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('OVMD_PROFILES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GA_PRODUCT_TREE_CONF');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GA_PROPERTY_VALUES_CONF');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_REJECTION_TYPES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IRM_OP_CODE_TYPES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('PROV_PROPERTIES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('PSMS_CAMPAIGNS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('PSMS_CAMPAIGN_CATEGORIES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('PSMS_CAMPAIGN_CATEGORY_TYPES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_DEVICE_CONFIGURATION');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GA_W_COUNTERS_HISTORY');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_PROFILES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_DISTRIBUTION_PROFILES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_ROAMING_DISTRIBUTION');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('IPN_NETWORK_PROFILES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('RAF_ACCUM_RULES');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('RAF_RULE_AXIS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GSM_MERGE_NETWORKS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GSM_SERVICE_MERGED_NETWORKS');
    INSERT INTO TEMP_GRANTS_LIST(table_name) VALUES ('GSM_MERGE_NW_MAPPING');
    commit;
    
  EXCEPTION
    WHEN OTHERS THEN
    RAISE;
  END;  
 
  FOR c_table_cur IN (SELECT table_name FROM TEMP_GRANTS_LIST ORDER BY table_name) LOOP
    v_sql := 'GRANT SELECT ON '||c_table_cur.table_name||' TO '||v_rep_user;
    EXECUTE IMMEDIATE v_sql;
  
    IF c_table_cur.table_name IN ('IPN_IBR_CELL_REPORT_DATA', 'DEBUG_GEN_W_SDR') THEN
      v_sql := 'GRANT DELETE ON '||c_table_cur.table_name||' TO '||v_rep_user;
      EXECUTE IMMEDIATE v_sql;
    END IF;  
  END LOOP;

  BEGIN
    v_sql := 'DROP TABLE TEMP_GRANTS_LIST';
    EXECUTE IMMEDIATE v_sql;
  EXCEPTION
    WHEN OTHERS THEN
      RAISE;
  END;
  
EXCEPTION
  WHEN OTHERS THEN
    RAISE;
END;    
/

PROMPT Grant Permissions Step Finished Successfully
PROMPT ======================================
PROMPT
spool off
exit 
 

awk example
input:
. . exported "LAB_QANFV_ALLQQ"."VSSP"     10.03 KB       6 rows

output:
SELECT * FROM GG_REF_TABLES WHERE table_name ='VSSP';

Command examples
Parse to SELECT
less exp_gg_LAB_QANFV_ALLQQ.log | grep expo | awk '{print $4}' | sort -u | sed s/\"//g | sed s/\\.//g| sed s/LAB_QANFV_ALLQQ/"SELECT * FROM GG_REF_TABLES WHERE table_name =\\'"/ | awk '{print $0 "'\''"}' | awk -F: '{print $1}' | sort -u | awk '{print $0";"}' > ~iu/check_tables.sql

Parse to INSERT
less exp_gg_LAB_QANFV_ALLQQ.log | grep expo | awk '{print $4}' | sort -u | sed s/\"//g | sed s/\\.//g | sed s/LAB_QANFV_ALLQQ/"INSERT INTO GG_REF_EXPDP (table_name, is_replicated, expdp, impdp) VALUES (\\'"/ | awk '{print $0 "'\'',NULL,'\''Y'\'',NULL)"}' | awk -F: '{print $1}' | sort -u | awk '{print $0";"}'

Thursday, May 21, 2020

CC PL/SQL- Conditional Compilation in PL/SQL

============================
General
============================
Conditional Compilation in PL/SQL (aka CC PLSQL)

Was added in Oracle 10.1.0.4
Conditional compilation allows PL/SQL code to be tailored by altering the source code based on compiler directives.  

Compiler flags are identified by the "$$" prefix
Conditional control is provided by the $IF-$THEN-$ELSE syntax.

The result of conditional compilation is the removal of code. 
PL/SQL is an interpreted language, so removing unnecessary code may produce performance improvements in some circumstances.

============================
Example
============================
Basic example

$$some_flag

$IF <condition_1> $THEN 
  do_this_01
$ELSIF <condition_2> $THEN 
  do_this_02
$ELSE 
  do_this_03
$END

Procedure example

CREATE OR REPLACE PROCEDURE debug (p_text  IN  VARCHAR2) IS
  $IF $$debug_on $THEN
    l_text  VARCHAR2(32767);
  $END
BEGIN
  $IF $$debug_on $THEN
    $IF DBMS_DB_VERSION.VER_LE_10_1 $THEN
      l_text := SUBSTR(p_text, 1 ,233);
    $ELSE
      l_text := p_text;
    $END

    $IF $$show_date $THEN
      DBMS_OUTPUT.put_line(TO_CHAR(SYSDATE, 'DD-MON-YYYY HH24:MI:SS') || ': ' || l_text);
    $ELSE
      DBMS_OUTPUT.put_line(p_text);
    $END
  $ELSE
    NULL;
  $END
END debug;
/

The debug_on flag acts as an on/off switch.
A value of FALSE will result in an empty procedure. 
A value of TRUE is running the IF $$debug $THEN code.

Once the procedure is compiled the complete source is stored in the database, including the conditional code directives.

SET PAGESIZE 30
SELECT text
  FROM user_source
 WHERE name = 'DEBUG'
   AND type = 'PROCEDURE';

TEXT
------------------------------------------------------------------
PROCEDURE debug (p_text  IN  VARCHAR2) AS
  $IF $$debug_on $THEN
    l_text  VARCHAR2(32767);
  $END
BEGIN
  $IF $$debug_on $THEN
    $IF DBMS_DB_VERSION.VER_LE_10_1 $THEN
      l_text := SUBSTR(p_text, 1 ,233);
    $ELSE
      l_text := p_text;
    $END

    $IF $$show_date $THEN
      DBMS_OUTPUT.put_line(TO_CHAR(SYSDATE, 'DD-MON-YYYY HH24:MI:SS') || ': ' || l_text);
    $ELSE
      DBMS_OUTPUT.put_line(p_text);
    $END
  $ELSE
    NULL;
  $END
END debug;

21 rows selected.

The DBMS_PREPROCESSOR.print_post_processed_source() procedure of the DBMS_PREPROCESSOR package displays the post-processed source.

SET SERVEROUTPUT ON SIZE UNLIMITED
BEGIN
  DBMS_PREPROCESSOR.print_post_processed_source (
    object_type => 'PROCEDURE',
    schema_name => 'TEST',
    object_name => 'DEBUG');
END;
/

PROCEDURE debug (p_text  IN  VARCHAR2) AS
BEGIN
NULL;
END debug;

PL/SQL procedure successfully completed.

Without compile flags, we get  an empty debug procedure. 
Setting the debug_on and show_date compiler flags results in debug messages printed with a date prefix.

ALTER PROCEDURE debug COMPILE PLSQL_CCFLAGS = 'debug_on:TRUE, show_date:TRUE' REUSE SETTINGS;

BEGIN
  DBMS_PREPROCESSOR.print_post_processed_source (
    object_type => 'PROCEDURE',
    schema_name => 'TEST',
    object_name => 'DEBUG');
END;
/

PROCEDURE debug (p_text  IN  VARCHAR2) AS
l_text  VARCHAR2(32767);
BEGIN
l_text := p_text;
DBMS_OUTPUT.put_line(TO_CHAR(SYSDATE, 'DD-MON-YYYY HH24:MI:SS') || ': ' || l_text);
END debug;

PL/SQL procedure successfully completed.


Switching off the show_date compiler flag results in debug messages printed without a date prefix.

ALTER PROCEDURE debug COMPILE PLSQL_CCFLAGS = 'debug_on:TRUE, show_date:FALSE' REUSE SETTINGS;

SET SERVEROUTPUT ON SIZE UNLIMITED
BEGIN
  DBMS_PREPROCESSOR.print_post_processed_source (
    object_type => 'PROCEDURE',
    schema_name => 'TEST',
    object_name => 'DEBUG');
END;
/
PROCEDURE debug (p_text  IN  VARCHAR2) AS
l_text  VARCHAR2(32767);
BEGIN
l_text := p_text;
DBMS_OUTPUT.put_line(l_text);
END debug;

PL/SQL procedure successfully completed.

SQL>


============================
REUSE SETTINGS
============================
The compile-time value of Conditional Compilation parameters is stored with the metadata of the PL/SQL unit being compiled.

If you omit any parameter from this clause and you specify REUSE SETTINGS, then if a value was specified for the parameter in an earlier compilation of this PL/SQL unit, the database uses that earlier value. 

If you do not specify REUSE SETTINGS ,then the database obtains the value for that parameter from the session environment.
If that one was not set in session environment, then the parameter value is set to NULL.

Tuesday, May 5, 2020

sqloader example. Loading several files types via bash and sqloader.

====================
General
====================
Load via bash and sqloader to three tables.

====================
Files
====================
ora_env.ini
ipn_profile_load.ini
main_ipn_conv_load.sh
load_handler.sh
sql_loader_cmd.sh
sql_loader_vrs_configuration.ctl
sql_loader_black_list.ctl
sql_loader_mccmnc.ctl

====================
Code
====================

==================================
ora_env.ini
==================================
ORA_VER=1120
ORACLE_SID=igt
ORACLE_BASE=/software/oracle
ORACLE_HOME=/software/oracle/112

==================================
ipn_profile_load.ini
==================================
DB_USER=ipn_conv
DB_PASS=ipn_conv
DB_INST=igt
DELIMITER="==================================="

WORK_DIR=/starhome/iu/workarea/IPN_LOAD
SCRIPTS_DIR=/starhome/iu/workarea/IPN_LOAD/scripts

LOG_FILE=/starhome/iu/workarea/IPN_LOAD/logs/ipn_load.log

INPUT_DIR=/starhome/iu/workarea/IPN_LOAD/files/input
PROCESS_DIR=/starhome/iu/workarea/IPN_LOAD/files/in_process
HANDLED_FILES_DIR=/starhome/iu/workarea/IPN_LOAD/files/handled
LOG_DIR=/starhome/iu/workarea/IPN_LOAD/logs

INPUT_FILE_NAME_ZIP=moco_managed_IPN_LOAD*dat.gz
INPUT_FILE_NAME=moco_managed_IPN_LOAD*dat

PATTERN_EXAMPLE=IPN_LOAD_[0-9]{2}.[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}.csv
INPUT_FILE_PATTERN_1=vrs_configuration.csv
INPUT_FILE_PATTERN_2=black_list.csv
INPUT_FILE_PATTERN_3=mccmnc.csv

FLOW_1=vrs_configuration
FLOW_2=black_list
FLOW_3=mccmnc

SQL_LOADER_DIR=/starhome/iu/workarea/IPN_LOAD/loader_code
SQL_LOADER_DIR_IN=/starhome/iu/workarea/IPN_LOAD/loader_code/in
SQL_LOADER_DIR_BAD=/starhome/iu/workarea/IPN_LOAD/loader_code/bad
SQL_LOADER_DIR_LOG=/starhome/iu/workarea/IPN_LOAD/loader_code/log
SQL_LOADER_DIR_OLD=/starhome/iu/workarea/IPN_LOAD/loader_code/old

SQL_LOADER_DAT_FILE=/starhome/iu/workarea/IPN_LOAD/loader_code/in/load_data.dat
SQL_LOADER_BAD_FILE=/starhome/iu/workarea/IPN_LOAD/loader_code/bad/bad_load_data.dat
SQL_LOADER_HIST_FILE=/starhome/iu/workarea/IPN_LOAD/loader_code/old/load_data.old
SQL_LOADER_LOG_FILE=/starhome/iu/workarea/IPN_LOAD/loader_code/log/load_load.log

CTL_FILE_1=/starhome/iu/workarea/IPN_LOAD/loader_code/sql_loader_vrs_configuration.ctl
CTL_FILE_2=/starhome/iu/workarea/IPN_LOAD/loader_code/sql_loader_black_list.ctl
CTL_FILE_3=/starhome/iu/workarea/IPN_LOAD/loader_code/sql_loader_mccmnc.ctl

==================================
main_ipn_conv_load.sh
==================================
!/bin/bash
INI_FILE=/starhome/iu/workarea/IPN_LOAD/scripts/ipn_profile_load.ini

#=====================
#functions()
#=====================
get_value(){
 FILE_NAME=$1
 VALUE_NAME=$2
 #echo "Looking for $VALUE_NAME in file $FILE_NAME"
 v_return_value=`grep -w $VALUE_NAME $FILE_NAME | sed s/\=/" "/ | awk '{print $2}'`
 echo $v_return_value
}

write_to_log(){
 echo $1
 echo $1 >> $LOG_FILE
}

create_folder(){
 FOLDER_NAME=$1
 if [[ ! -d ${FOLDER_NAME} ]]; then
   mkdir ${FOLDER_NAME}
 fi
}
#=====================
#set variables
#=====================
DELIMITER=$(get_value ${INI_FILE} DELIMITER)
LOG_FILE=$(get_value ${INI_FILE} LOG_FILE)
RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`
this_script=`basename $0`

#---------------------
#main()
#---------------------

echo "LOG_FILE = ${LOG_FILE}"
write_to_log ""
write_to_log $DELIMITER
write_to_log "$this_script .........: Starting at ${RUN_DATE}"
write_to_log $DELIMITER
write_to_log ""

#write_to_log "INPUT_DIR"
INPUT_DIR=$(get_value ${INI_FILE} INPUT_DIR)
create_folder ${INPUT_DIR}

#write_to_log "PROCESS_DIR"
PROCESS_DIR=$(get_value ${INI_FILE} PROCESS_DIR)
create_folder ${PROCESS_DIR}

#write_to_log "HANDLED_FILES_DIR"
HANDLED_FILES_DIR=$(get_value ${INI_FILE} HANDLED_FILES_DIR)
create_folder ${HANDLED_FILES_DIR}

#write_to_log "LOG_DIR"
LOG_DIR=$(get_value ${INI_FILE} LOG_DIR)
create_folder ${LOG_DIR}

#write_to_log "SQL_LOADER_DIR"
SQL_LOADER_DIR=$(get_value ${INI_FILE} SQL_LOADER_DIR)
create_folder ${SQL_LOADER_DIR}

#write_to_log "SQL_LOADER_DIR_IN"
SQL_LOADER_DIR_IN=$(get_value ${INI_FILE} SQL_LOADER_DIR_IN)
create_folder ${SQL_LOADER_DIR_IN}

#write_to_log "SQL_LOADER_DIR_BAD"
SQL_LOADER_DIR_BAD=$(get_value ${INI_FILE} SQL_LOADER_DIR_BAD)
create_folder ${SQL_LOADER_DIR_BAD}

#write_to_log "SQL_LOADER_DIR_LOG"
SQL_LOADER_DIR_LOG=$(get_value ${INI_FILE} SQL_LOADER_DIR_LOG)
create_folder ${SQL_LOADER_DIR_LOG}

#write_to_log "SQL_LOADER_DIR_OLD"
SQL_LOADER_DIR_OLD=$(get_value ${INI_FILE} SQL_LOADER_DIR_OLD)
create_folder ${SQL_LOADER_DIR_OLD}

write_to_log ""
. /etc/sh/orash/oracle_login.sh igt
WORK_AREA=/starhome/iu/workarea/IPN_LOAD/scripts

cd ${WORK_AREA}
./load_handler.sh
exit_status=$?
RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`
write_to_log ""
write_to_log $DELIMITER
write_to_log "Finished Running $this_script. Process status: $exit_status"
write_to_log $DELIMITER
write_to_log ""

mv ${LOG_FILE} ${LOG_FILE}_${RUN_DATE}
write_to_log "Log File: ${LOG_FILE}_${RUN_DATE}"

exit $exit_status

==================================
load_handler.sh
==================================
INI_FILE=/starhome/iu/workarea/IPN_LOAD/scripts/ipn_profile_load.ini

#=====================
#functions()
#=====================
get_value(){
 FILE_NAME=$1
 VALUE_NAME=$2
 #echo "Looking for $VALUE_NAME in file $FILE_NAME"
 v_return_value=`grep -w $VALUE_NAME $FILE_NAME | sed s/\=/" "/ | awk '{print $2}'`
 echo $v_return_value
}

write_to_log(){
 echo $1
 echo $1 >> $LOG_FILE
}


load_file(){
 INPUT_FILE_PATTERN=$1
 FLOW_TYPE=$2
 #write_to_log "Input Files in ${INPUT_DIR}:"
 find ${INPUT_DIR} -type f -printf "%f\n"
 #write_to_log "Looking for ${INPUT_FILE_PATTERN} inside ${INPUT_DIR}"
 find ${INPUT_DIR} -type f | egrep ${INPUT_FILE_PATTERN}

 files_num=`find ${INPUT_DIR} -type f | egrep ${INPUT_FILE_PATTERN} 2>/dev/null | wc -l`
 #write_to_log "$this_script ...: Number of Input Files: ${INPUT_DIR}/*${INPUT_FILE_PATTERN}:  $files_num"

 if [[ $files_num == 0 ]];then
  write_to_log "$this_script ...: No Files to Process"
  return 0
 fi

 if [[ $files_num > 1 ]];then
  write_to_log "$this_script ...: More than 1 input File. Exiting !!! "
  exit 1
 fi

 file_name=`find ${INPUT_DIR} -type f | egrep ${INPUT_FILE_PATTERN}`
 BASENAME=`basename $file_name`
 write_to_log "Loading File: $file_name"
 #write_to_log "cp $file_name ${PROCESS_DIR}/${BASENAME}"
 #Remove Empty Lines
 less $file_name | grep -v ^$ > ${PROCESS_DIR}/${BASENAME}

 #!!!!!!!!!!!!!!!!!!!!!!!
 #Remove the file
 rm $file_name
 #!!!!!!!!!!!!!!!!!!!!!!!

 #cp $file_name ${PROCESS_DIR}/${BASENAME}
 #write_to_log "./sql_loader_cmd.sh ${INI_FILE} ${file_name} ${FLOW_TYPE}"

 ./sql_loader_cmd.sh ${INI_FILE} ${PROCESS_DIR}/${BASENAME} ${FLOW_TYPE}
 #write_to_log "$this_script .............: Finished "
 mv ${PROCESS_DIR}/${BASENAME} ${HANDLED_FILES_DIR}/${BASENAME}
 #write_to_log $DELIMITER
}

#=====================
#set variables
#=====================
RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`
SCRIPTS_DIR=$(get_value ${INI_FILE} SCRIPTS_DIR)
LOG_DIR=$(get_value ${INI_FILE} LOG_DIR)
INPUT_FILE_PATTERN_1=$(get_value ${INI_FILE} INPUT_FILE_PATTERN_1)
INPUT_FILE_PATTERN_2=$(get_value ${INI_FILE} INPUT_FILE_PATTERN_2)
INPUT_FILE_PATTERN_3=$(get_value ${INI_FILE} INPUT_FILE_PATTERN_3)

DELIMITER=$(get_value ${INI_FILE} DELIMITER)
LOG_FILE=$(get_value ${INI_FILE} LOG_FILE)
WORK_DIR=$(get_value ${INI_FILE} WORK_DIR)
INPUT_DIR=$(get_value ${INI_FILE} INPUT_DIR)

INPUT_DIR=$(get_value ${INI_FILE} INPUT_DIR)
PROCESS_DIR=$(get_value ${INI_FILE} PROCESS_DIR)
HANDLED_FILES_DIR=$(get_value ${INI_FILE} HANDLED_FILES_DIR)


FLOW_1=$(get_value ${INI_FILE} FLOW_1 )
FLOW_2=$(get_value ${INI_FILE} FLOW_2 )
FLOW_3=$(get_value ${INI_FILE} FLOW_3 )

#=====================
#main
#=====================
. /etc/sh/orash/oracle_login.sh igt
this_script=`basename $0`
echo "Log File is: $LOG_FILE"
touch ${LOG_FILE}
#write_to_log ""
#write_to_log $DELIMITER
#write_to_log "$this_script .............: Starting Run at $RUN_DATE"
#write_to_log $DELIMITER
cd ${SCRIPTS_DIR}

write_to_log ""
write_to_log "${DELIMITER}"
write_to_log "Start Loading file type ${INPUT_FILE_PATTERN_1} Flow: ${FLOW_1}"
load_file "${INPUT_FILE_PATTERN_1}" "${FLOW_1}"
ret_code=$?
if [[ $ret_code > 0 ]]; then
 exit $ret_code
fi
write_to_log ""

write_to_log ""
write_to_log "${DELIMITER}"
write_to_log "Start Loading file type ${INPUT_FILE_PATTERN_2} Flow: ${FLOW_2}"
sleep 2
load_file "${INPUT_FILE_PATTERN_2}" "${FLOW_2}"
ret_code=$?
if [[ $ret_code > 0 ]]; then
 exit $ret_code
fi
write_to_log ""


write_to_log ""
write_to_log "${DELIMITER}"
write_to_log "Start Loading file type ${INPUT_FILE_PATTERN_3} Flow: ${FLOW_3}"
sleep 2
load_file "${INPUT_FILE_PATTERN_3}" "${FLOW_3}"
ret_code=$?
if [[ $ret_code > 0 ]]; then
 exit $ret_code
fi
write_to_log ""

write_to_log "${DELIMITER}"
write_to_log ""

exit 0

==================================
sql_loader_cmd.sh
==================================
#!/bin/bash
INI_FILE=$1
FILE_NAME=$2
FLOW_TYPE=$3
#=====================
#functions()
#=====================
get_value(){
 FILE_NAME=$1
 VALUE_NAME=$2
 #echo "Looking for $VALUE_NAME in file $FILE_NAME"
 v_return_value=`grep -w $VALUE_NAME $FILE_NAME | sed s/\=/" "/ | awk '{print $2}'`
 echo $v_return_value
}
write_to_log(){
 echo $1
 echo $1 >> $LOG_FILE
}
#=====================
#set variables
#=====================
RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`
INPUT_FILE_PATTERN=$(get_value ${INI_FILE} INPUT_FILE_PATTERN)
INPUT_FILE_PATTERN_1=$(get_value ${INI_FILE} INPUT_FILE_PATTERN_1)
INPUT_FILE_PATTERN_2=$(get_value ${INI_FILE} INPUT_FILE_PATTERN_2)
INPUT_FILE_PATTERN_3=$(get_value ${INI_FILE} INPUT_FILE_PATTERN_3
)
LOG_FILE=$(get_value ${INI_FILE} LOG_FILE)
INPUT_DIR=$(get_value ${INI_FILE} INPUT_DIR)
SOURCE_DATA=$(get_value ${INI_FILE} SOURCE_DATA)
OUTPUT_DIR=$(get_value ${INI_FILE} OUTPUT_DIR)
INPUT_FILE_NAME=$(get_value ${INI_FILE} INPUT_FILE_NAME)
HANDLED_FILES_DIR=$(get_value ${INI_FILE} HANDLED_FILES_DIR)
SQL_LOADER_LOG_FILE=$(get_value ${INI_FILE} SQL_LOADER_LOG_FILE)

#CTL_FILE=$(get_value ${INI_FILE} CTL_FILE)
CTL_FILE_1=$(get_value ${INI_FILE} CTL_FILE_1)
CTL_FILE_2=$(get_value ${INI_FILE} CTL_FILE_2)
CTL_FILE_3=$(get_value ${INI_FILE} CTL_FILE_3)

FLOW_1=$(get_value ${INI_FILE} FLOW_1 )
FLOW_2=$(get_value ${INI_FILE} FLOW_2 )
FLOW_3=$(get_value ${INI_FILE} FLOW_3 )

SQL_LOADER_DAT_FILE=$(get_value ${INI_FILE} SQL_LOADER_DAT_FILE)
SQL_LOADER_BAD_FILE=$(get_value ${INI_FILE} SQL_LOADER_BAD_FILE)
SQL_LOADER_HIST_FILE=$(get_value ${INI_FILE} SQL_LOADER_HIST_FILE)
DB_USER=$(get_value ${INI_FILE} DB_USER)
DB_PASS=$(get_value ${INI_FILE} DB_PASS)
DB_INST=$(get_value ${INI_FILE} DB_INST)
DELIMITER=$(get_value ${INI_FILE} DELIMITER)

this_script=`basename $0`
write_to_log "$this_script ...: ----------------------------------------------------------"
write_to_log "$this_script ...: Starting ${this_script} at ${RUN_DATE}"
#write_to_log "$this_script ...: sqlldr ${DB_USER}/${DB_PASS}@${DB_INST} CONTROL=${CTL_FILE} LOG=${SQL_LOADER_LOG_FILE}"

. /etc/sh/orash/oracle_login.sh igt
. /starhome/iu/workarea/IPN_LOAD/scripts/ora_env.ini


#write_to_log "cp ${FILE_NAME} ${SQL_LOADER_DAT_FILE}"
cp ${FILE_NAME} ${SQL_LOADER_DAT_FILE}

if [[ $FLOW_TYPE == ${FLOW_1} ]]; then
 CTL_FILE=${CTL_FILE_1}
fi
if [[ $FLOW_TYPE == ${FLOW_2} ]]; then
 CTL_FILE=${CTL_FILE_2}
fi
if [[ $FLOW_TYPE == ${FLOW_3} ]]; then
 CTL_FILE=${CTL_FILE_3}
fi


if [[ -f ${FILE_NAME} ]]; then
  #write_to_log "$this_script ...: sqlldr ${DB_USER}/${DB_PASS}@${DB_INST} CONTROL=${CTL_FILE} LOG=${SQL_LOADER_LOG_FILE}"
  #write_to_log "$this_script ...: ls -l /starhome/iu/workarea/IPN_LOAD/loader_code/in/IPN_LOAD_data.dat"
  #write_to_log "$this_script ...: `ls -l /starhome/iu/workarea/IPN_LOAD/loader_code/in/IPN_LOAD_data.dat`"
  #write_to_log "$this_script ...: `env | grep ORA`"
  SQLDR_HOME=${ORACLE_HOME}/bin
  echo ${SQLDR_HOME}/sqlldr ${DB_USER}/${DB_PASS}@${DB_INST} CONTROL=${CTL_FILE} LOG=${SQL_LOADER_LOG_FILE}
  ${SQLDR_HOME}/sqlldr ${DB_USER}/${DB_PASS}@${DB_INST} CONTROL=${CTL_FILE} LOG=${SQL_LOADER_LOG_FILE}
  v_status=$?
  if [[ $v_status -ne 0 ]];then
    write_to_log "$this_script ...: ERROR in step: sqlldr ${DB_USER}/${DB_PASS}@${DB_INST} CONTROL=${CTL_FILE} LOG=${SQL_LOADER_LOG_FILE}"
    write_to_log "$this_script ...: Status: $v_status "
  fi
else
  write_to_log "$this_script ...: Error!! Could not find input file : ${FILE_NAME}"
fi


if [[ -f ${SQL_LOADER_LOG_FILE} ]]; then
  #write_to_log "$this_script ...: moving ${SQL_LOADER_LOG_FILE} : `ls -l ${SQL_LOADER_LOG_FILE}`"
  #write_to_log "$this_script ...: mv ${SQL_LOADER_LOG_FILE} ${SQL_LOADER_LOG_FILE}_${RUN_DATE}"
  mv ${SQL_LOADER_LOG_FILE} ${SQL_LOADER_LOG_FILE}_${RUN_DATE}
  write_to_log "sqloder log file: ${SQL_LOADER_LOG_FILE}_${RUN_DATE}"
else
  write_to_log "$this_script ...: ERROR! Cannot find Log File ${SQL_LOADER_LOG_FILE}"
fi

if [[ -f ${SQL_LOADER_BAD_FILE} ]]; then
  #write_to_log "$this_script ...: mv ${SQL_LOADER_BAD_FILE} ${SQL_LOADER_BAD_FILE}_${RUN_DATE}"
  mv ${SQL_LOADER_BAD_FILE} ${SQL_LOADER_BAD_FILE}_${RUN_DATE}
fi
if [[ -f ${SQL_LOADER_DAT_FILE} ]]; then
  #write_to_log "$this_script ...: moving ${SQL_LOADER_DAT_FILE}  : `ls -l ${SQL_LOADER_DAT_FILE} `"
  #write_to_log "$this_script ...: mv ${SQL_LOADER_DAT_FILE} ${SQL_LOADER_HIST_FILE}_${RUN_DATE}"
  mv ${SQL_LOADER_DAT_FILE} ${SQL_LOADER_HIST_FILE}_${RUN_DATE}
fi

RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`
write_to_log "$this_script ...: ----------------------------------------------------------"
write_to_log "$this_script ...: Finished ${this_script} in mode ${RUN_MODE} at ${RUN_DATE}"
write_to_log "$this_script ...: ----------------------------------------------------------"

exit $v_status

==================================
sql_loader_vrs_configuration.ctl
==================================
LOAD DATA
INFILE '/starhome/iu/workarea/IPN_LOAD/loader_code/in/load_data.dat'
BADFILE '/starhome/iu/workarea/IPN_LOAD/loader_code/bad/bad_load_data.dat'
DISCARDMAX 0
TRUNCATE
INTO TABLE IPN_CONV_CONFIGURATION
WHEN (COUNTRY != BLANKS)
FIELDS TERMINATED BY ','
(
country              "TRIM (:country)",
comunity             "TRIM (:comunity)",
network_name         "TRIM (:network_name)",
redirection_code     "TRIM (:redirection_code)",
network_type         "TRIM (:network_type)",
rdc_pct              "TRIM (:rdc_pct)",
mcc                  "TRIM (:mcc)",
mnc                  "TRIM (:mnc)",
mcc_mnc              "TRIM (:mcc_mnc)",
tomia_network_name   "TRIM (:tomia_network_name)",
tomia_mcc_mnc        "TRIM (:tomia_mcc_mnc)",
tomia_community      "TRIM (:tomia_community)",
aux                  "TRIM (:aux)",
tomia_configuration  "TRIM (:tomia_configuration)",
tomia_network_barring "TRIM (:tomia_network_barring)",
ts_last_modified     "SYSDATE"
)

==================================
sql_loader_black_list.ctl
==================================
LOAD DATA
INFILE '/starhome/iu/workarea/IPN_LOAD/loader_code/in/load_data.dat'
BADFILE '/starhome/iu/workarea/IPN_LOAD/loader_code/bad/bad_load_data.dat'
DISCARDMAX 0
TRUNCATE
INTO TABLE IPN_CONV_BLACK_LIST
FIELDS TERMINATED BY ','
(
imsi                 "TRIM (:imsi)",
ts_last_modified     SYSDATE
)

==================================
sql_loader_mccmnc.ctl
==================================
LOAD DATA
INFILE '/starhome/iu/workarea/IPN_LOAD/loader_code/in/load_data.dat'
BADFILE '/starhome/iu/workarea/IPN_LOAD/loader_code/bad/load_data.dat'
DISCARDMAX 0
TRUNCATE
INTO TABLE IPN_CONV_TOMIA_MCCMNC
FIELDS TERMINATED BY ','
(
mcc              "TRIM (:mcc)",
mnc              "TRIM (:mnc)",
mccmnc           "TRIM (:mccmnc)",
iso              "TRIM (:iso)",
country_name     "TRIM (:country_name)",
country_code     "TRIM (:country_code)",
network_name     "TRIM (:network_name)",
country_network  "TRIM (:country_network)",
ts_last_modified SYSDATE
)