Pages

Wednesday, June 28, 2017

"ORA-00900: invalid SQL statement" and "ORA-02064 Distributed operation not supported" Errors When calling to Remote PL/SQL Procedure

================================
General
================================
A. When calling to a remote PL/SQL procedure from sqlplus - all works fine,
But when calling to the same remote PL/SQL procedure from PL/SQL code, an ORA-00900: invalid SQL statement error is thrown.

B. When calling a PL/SQL directly, all works fine, but when calling same procedure vi db link, it fails with ORA-02064 Distributed operation not supported

Why is that?

================================
ORA-00900 Example
================================
In this example, a DB_LINK named KUKU is created to the remote database.

CREATE DATABASE LINK KUKU CONNECT TO REMOTE_USER IDENTIFIED BY REMOTE_PASSWORD USING 'REMOTE_HOST_NAME';

In the remote database, a getDate Procedure in Package PKG_TEST should be activated.

In sqlplus
From sqlplus, any of these versions work:
EXEC PKG_TEST.getDate@KUKU;
EXEC REMOTE_USER.PKG_TEST.getDate@KUKU;

When creating a Synonym, it can be also used, to simplify code.

CREATE SYNONYM REMOTE_PKG FOR REMOTE_USER.PKG_TEST@KUKU

SELECT * FROM USER_SYNONYMS WHERE synonym_name = 'KUKU'

SYNONYM_NAME        TABLE_OWNER      TABLE_NAME      DB_LINK
------------------- ---------------- --------------- ---------------
REMOTE_PKG          REMOTE_USER      PKG_TEST        KUKU

So in sqlplus, it would be:
EXEC REMOTE_PKG.getDate;

In PL/SQL
When activating the same code in PL/SQL it MUST be inside a BEGIN END block.
Any other syntax, such as EXECUTE IMMEDIATE <sql_string> would fail with "ORA-00900: invalid SQL statement".

This is the correct code:

BEGIN
  PKG_TEST.getDate;
  EXCEPTION
    WHEN OTHERS THEN
      WRITE_LOG(v_module_name,'Error Running: '||v_sql_str||' '||SQLERRM);
      RAISE;
END;


================================
ORA-02064 Example
================================
In this example, same DB_LINK named KUKU is created to the remote database.
The procedure now is a DML Procedre, performing an update, and a commit.

The remote procedure has these API:

PKG_TEST.setDate(v_date    IN DATE,

                v_result  OUT NUMBER);

When calling the remote procedure in PL/SQL, it would be:


DECLARE
  v_date   DATE;
  v_result NUMBER;
BEGIN
  v_date := SYSDATE;
  PKG_TEST.setDate(v_date,v_result);
  EXCEPTION
    WHEN OTHERS THEN
      WRITE_LOG(v_module_name,'Error Running: '||v_sql_str||' '||SQLERRM);
      RAISE;


END;


The returned error is:

ORA-02064: distributed operation not supported
ORA-06512: at "REMOTE_USER.PKG_TEST", line 491

Per Oracle documentation, 

Error Cause:
One of the following unsupported operations was attempted:

1. array execute of a remote update with a subquery that references a dblink, or

2. an update of a long column with bind variable and an update of a second column with a subquery that both references a dblink and a bind variable, or

3. a commit is issued in a coordinated session from an RPC procedure call with OUT parameters or function call.

Action:
Simplify remote update statement.

Per Oracle Metalink CALLING REMOTE PACKAGE RECEIVES ORA-2064 (Doc ID 1026597.6):
"This is Not A Bug. 
In the documentation for ORA-2064, it states that one of  the disallowed actions is "A commit is issued in a coordinated session from an RPC with OUT parameters." 
It happens that the return value of a function call counts as an OUT parameter for purposes of this rule. 

As a workaround for some cases Pragma AUTONOMOUS_TRANSACTION can be used in the called function or procedure in the remote site package."


So, a work around this problem would be to change the remote Procedure to be Autonomous Transaction.

PKG_TEST.setDate(v_date    IN DATE,
                v_result  OUT NUMBER) IS
PRAGMA AUTONOMOUS_TRANSACTION;


Tuesday, June 27, 2017

DBMS_STATS.GATHER_FIXED_OBJECTS_STATS, DBMS_STATS.GATHER_DICTIONARY_STATS and DBMS_STATS.GATHER_SYSTEM_STATS

======================================
General 

======================================
DBMS_STATS.GATHER_FIXED_OBJECTS_STATS
DBMS_STATS.GATHER_DICTIONARY_STATS
DBMS_STATS.GATHER_SYSTEM_STATS

======================================
DBMS_STATS.GATHER_FIXED_OBJECTS_STATS
======================================
This would collect statistics about Fixed objects. 
These are the X$ and K$ tables and their indexes. 
The V$ views in Oracle are defined in top of X$ tables (for example V$SQL and V$SQL_PLAN).

How to identify when DBMS_STATS.GATHER_FIXED_OBJECTS_STATS was executed in the database ?

SELECT v.name, ts.analyzetime 
FROM V$FIXED_TABLE v, SYS.TAB_STATS$ ts 
WHERE v.object_id = ts.obj#; 

no rows returned

SELECT COUNT(*) FROM sys.tab_stats$
count(*) was 0.

EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS(NULL);
This takes few minutes.

SELECT COUNT(*) FROM sys.tab_stats$
returns 761

======================================
DBMS_STATS.GATHER_DICTIONARY_STATS
======================================
This procedure gathers statistics for dictionary schemas 'SYS', 'SYSTEM'.

When was it last run?
Check the the last_analyzed column for tables owned by SYS.
SELECT MAX(LAST_ANALYZED) FROM DBA_TABLES WHERE owner = 'SYSTEM';

MAX(LAST_ANALYZED)
------------------
27-JUN-17

SELECT MAX(LAST_ANALYZED) FROM DBA_TABLES WHERE owner = 'SYS';

MAX(LAST_ANALYZED)
------------------
27-JUN-17

======================================
DBMS_STATS.GATHER_SYSTEM_STATS
======================================
This procedure gathers system statistics.

The actual gathered statistics would depend upon system being under workload, or not.

DBMS_STATS.GATHER_SYSTEM_STATS procedure gathers statistics relating to the performance of your systems I/O and CPU. 
Giving the optimizer this information makes its choice of execution plan more accurate, since it is able to weigh the relative costs of operations using both the CPU and I/O profiles of the system.


When was DBMS_STATS.GATHER_SYSTEM_STATS last run?
The output from DBMS_STATS.GATHER_SYSTEM_STATS is stored in the AUX_STATS$ table.

SELECT * FROM SYS.AUX_STATS$;

NAME                 PNAME                     PVAL1 PVAL2
-------------------- -------------------- ---------- --------------------
SYSSTATS_INFO        STATUS                          COMPLETED
SYSSTATS_INFO        DSTART                          10-26-2008 13:08
SYSSTATS_INFO        DSTOP                           10-26-2008 13:08
SYSSTATS_INFO        FLAGS                         1
SYSSTATS_MAIN        CPUSPEEDNW           1108.95499
SYSSTATS_MAIN        IOSEEKTIM                    10
SYSSTATS_MAIN        IOTFRSPEED                 4096
SYSSTATS_MAIN        SREADTIM
SYSSTATS_MAIN        MREADTIM
SYSSTATS_MAIN        CPUSPEED
SYSSTATS_MAIN        MBRC
SYSSTATS_MAIN        MAXTHR
SYSSTATS_MAIN        SLAVETHR

How to execute DBMS_STATS.GATHER_SYSTEM_STATS?

Option A. - noworkload
All databases come bundled with a default set of noworkload statistics, but they can be replaced with more accurate information. 
When gathering noworkload stats, the database issues a series of random I/Os and tests the speed of the CPU. 
As you can imagine, this puts a load on your system during the gathering phase.

EXEC DBMS_STATS.GATHER_SYSTEM_STATS;

Option B. - Workload
When initiated using the start/stop or interval parameters, the database uses counters to keep track of all system operations, giving it an accurate idea of the performance of the system. 
If workload statistics are present, they will be used in preference to noworkload statistics.

EXEC DBMS_STATS.GATHER_SYSTEM_STATS('START');
-- Wait some time, say 120 minutes, during workload hours
EXEC DBMS_STATS.GATHER_SYSTEM_STATS('STOP');
or
EXEC DBMS_STATS.GATHER_SYSTEM_STATS('interval', interval => 120); 

======================================
When to run these procedures?
======================================

DBMS_STATS.GATHER_FIXED_OBJECTS_STATS
- When there was a change to init.ora Instance parameters

DBMS_STATS.GATHER_DICTIONARY_STATS
- When there was a change to dictionary structure - new schema, etc.

DBMS_STATS.GATHER_SYSTEM_STATS
- When there was a major change to the host hardware.

======================================
Get Execution Plan
======================================
SET LINESIZE 200
SET PAGESIZE 0
SELECT sql_text, sql_id from V$SQL WHERE SQL_TEXT LIKE '%XXXX%';
SELECT * FROM table (DBMS_XPLAN.DISPLAY_CURSOR('1ddfkrjxhvpt8'));

Monday, June 19, 2017

delete_old_trace_files.sh. Linux script to delete Oracle log files

===========================
General
===========================
This a script that cleans up log file for Oracle listener and server.

===========================
Code
===========================

crontab entry
0 6 * * * /software/oracle/oracle/scripts/delete_old_trace_files.sh

delete_old_trace_files.sh


#!/bin/bash
ORA_INST=igt
DAYS_TO_KEEP=14
DAYS_TO_KEEP_LIST=7
LIST_SERVER=`hostname`

LISTENER_ROOT=/software/oracle/diag/tnslsnr/${LIST_SERVER}/lsnr_igt/trace
LISTENER_ROOT_ALERT=/software/oracle/diag/tnslsnr/${LIST_SERVER}/lsnr_igt/alert

find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trc" -mtime +${DAYS_TO_KEEP} -exec rm {} \;
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -name "*.trm" -mtime +${DAYS_TO_KEEP} -exec rm {} \;
find /software/oracle/diag/rdbms/${ORA_INST}/${ORA_INST}/trace -type f -size +1000M -exec rm {} \;

find $LISTENER_ROOT  -type f -mtime +${DAYS_TO_KEEP} -exec rm {} \;
find $LISTENER_ROOT_ALERT  -type f -mtime +${DAYS_TO_KEEP_LIST} -exec rm {} \;

mv -f  ${LISTENER_ROOT}/lsnr_igt.log_6  ${LISTENER_ROOT}/lsnr_igt.log_7
mv -f  ${LISTENER_ROOT}/lsnr_igt.log_5  ${LISTENER_ROOT}/lsnr_igt.log_6
mv -f  ${LISTENER_ROOT}/lsnr_igt.log_4  ${LISTENER_ROOT}/lsnr_igt.log_5
mv -f  ${LISTENER_ROOT}/lsnr_igt.log_3  ${LISTENER_ROOT}/lsnr_igt.log_4
mv -f  ${LISTENER_ROOT}/lsnr_igt.log_2  ${LISTENER_ROOT}/lsnr_igt.log_3
mv -f  ${LISTENER_ROOT}/lsnr_igt.log_1  ${LISTENER_ROOT}/lsnr_igt.log_2
mv -f  ${LISTENER_ROOT}/lsnr_igt.log    ${LISTENER_ROOT}/lsnr_igt.log_1


===========================
Evaluate the current status
===========================
Find biggest files
find /some/path -type f -printf '%s %p\n'|sort -nr | head -10 

Find biggest folder
du  -sh * |sort -nr | head -10

Thursday, June 8, 2017

Compare Regular Cursor vs Bulk Collect

=====================================
General
=====================================
Overview: 
There is a table, which is source table.
The records from this table should be processed, and inserted into a target table.
Once done, records from source table are deleted.

This Package handles same task in two ways:
convertSdr2CdrCursor
convertSdr2CdrBulk

convertSdr2CdrCursor
The records are selected via regular cursor, and processed one by one.

convertSdr2CdrBulk
The records are selected via BULK COLLECT, and processed one by one.

Since the processing is done one by one, there is no real advantage for using BULK COLLECT.

=====================================
The time to execute in both options
=====================================
convertSdr2CdrCursorDelete 50000 records using regular cursor
Time to Execute: 999(sec), rows deleted: 50000
50000/999=50 rows/sec

convertSdr2CdrBulk
Delete 50000 records using BULK COLLECT
Time to Execute: 1249(sec), rows deleted: 60762

60762/1249 = 48 rows/sec

Both options were far superior to not using bind variables.
Without using bind variables, the time was 6 rows/sec.

=====================================
Code
=====================================
CREATE OR REPLACE PACKAGE BODY SDR_2_CDR_CONVERT_ALEC AS

PROCEDURE write_to_log(p_data in varchar2) is
  pragma autonomous_transaction;

begin
  insert into SGA_W_LOG (procedure_name, data, ts_last_modified) values ('ConvertSdr2Cdr', p_data, sysdate);
  commit;

end;

-------------------------------------------------------
-- Process using Regular Cursor
-------------------------------------------------------

PROCEDURE convertSdr2CdrCursor (pProductName in varchar2) is

  cursor SdrCrs(pProductName in varchar2) is
   SELECT  cdr_id,
           outgoing_call_direction
     from DEBUG_GEN_W_NEW_SDR_ALEC
     where rownum < 50000
     and product_name = pProductName
     and ((pProductName = 'MOCO' and event_name = 'IDP') OR (pProductName <> 'MOCO'))
     for update;

  Vsql         VARCHAR2(5000);
  v_sql_type1  VARCHAR2(5000);
  v_sql_type2  VARCHAR2(5000);
  
  v_start_date DATE;
  v_end_date   DATE;
  v_seconds    NUMBER;
  
BEGIN
  v_start_date := SYSDATE;
  write_to_log('start running convertSdr2CdrCursor procedure at '||TO_CHAR(v_start_date,'YYYYMMDD hh24:mi:ss') );
  
  v_sql_type1 := GenerateCdr(1);
  v_sql_type2 := GenerateCdr(2);

  -- for each SDR create the mapped CDRs
  for SdrRec in SdrCrs(pProductName) loop

    begin

    Vsql := v_sql_type1;

-- create the insert statement and execute it
    Vsql := 'insert into CDR_ALEC select ' || Vsql || ' from DEBUG_GEN_W_NEW_SDR_ALEC where cdr_id = :b1';
    EXECUTE IMMEDIATE Vsql USING SdrRec.cdr_id;


    -- Generate CDR for Leg2
    if SdrRec.outgoing_call_direction is not null then
       Vsql :=v_sql_type2;
       Vsql := 'insert into CDR_ALEC select ' || Vsql || ' from DEBUG_GEN_W_NEW_SDR_ALEC where cdr_id = :b1';
       EXECUTE IMMEDIATE Vsql USING SdrRec.cdr_id;
    end if;

    delete DEBUG_GEN_W_NEW_SDR_ALEC WHERE CURRENT OF SdrCrs;

    EXCEPTION
      WHEN OTHERS THEN
            write_to_log('Error for CDR_ID: '|| SdrRec.Cdr_Id || ' - ' ||sqlerrm);
            UPDATE DEBUG_GEN_W_NEW_SDR_ALEC
              SET ROAMING_TYPE ='9'
            WHERE CURRENT OF SdrCrs;
    END;
  END LOOP;

  COMMIT;
  
  v_end_date := SYSDATE;
  write_to_log('convertSdr2CdrCursor procedure completed at '||TO_CHAR(v_end_date,'YYYYMMDD hh24:mi:ss'));
  v_seconds := ROUND((v_end_date - v_start_date)*1440*60);
  write_to_log('Time to Execute: '||v_seconds||'(sec)');
  
end convertSdr2CdrCursor;


-------------------------------------------------------
-- Process using BULK COLLECT
-------------------------------------------------------
PROCEDURE convertSdr2CdrBulk(p_product_name IN VARCHAR2) IS

  TYPE sdr_record IS RECORD (
      sdr_rowid               ROWID,
      cdr_id                  DEBUG_GEN_W_NEW_SDR_ALEC.cdr_id%TYPE,
      outgoing_call_direction DEBUG_GEN_W_NEW_SDR_ALEC.outgoing_call_direction%TYPE
  );

  TYPE sdr_record_tab IS TABLE OF sdr_record;
  v_sdr_records   sdr_record_tab;

  CURSOR sdr_cur IS
  SELECT  ROWID, cdr_id, outgoing_call_direction
    FROM DEBUG_GEN_W_NEW_SDR_ALEC
   WHERE 1=1
     AND product_name = p_product_name
     AND ((p_product_name = 'MOCO' and event_name = 'IDP') OR (p_product_name <> 'MOCO'));
   
  v_sql                        VARCHAR2(5000);
  v_insert_sql                 VARCHAR2(5000);
  v_sql_type1                  VARCHAR2(5000);
  v_sql_type2                  VARCHAR2(5000);
  v_outgoingCallDirection      SDR_2_CDR_MAPPING_NEW.sdr_column_name%TYPE;

  v_start_date                DATE;
  v_end_date                  DATE;
  v_seconds                   NUMBER;
  v_cdr_id                    DEBUG_GEN_W_NEW_SDR_ALEC.cdr_id%TYPE;
  c_limit                     NUMBER := 50000;
  
BEGIN
  
  v_start_date := SYSDATE;
  write_to_log('Start running ConvertSdr2CdrBulk procedure at '||TO_CHAR(v_start_date,'YYYYMMDD hh24:mi:ss') );
  
  v_sql_type1 := GenerateCdr(1);
  v_sql_type2 := GenerateCdr(2);

  SELECT sdr_column_name INTO v_outgoingCallDirection
    FROM SDR_2_CDR_MAPPING_NEW
   WHERE leg_id = 2
    AND cdr_column_name = 'CALL_DIRECTION';

  write_to_log('Before Opening Cursor');
  
   OPEN sdr_cur;
   --Loop on Cursor
   LOOP
     BEGIN   
       FETCH sdr_cur 
         BULK COLLECT INTO v_sdr_records LIMIT c_limit;

         IF sdr_cur%NOTFOUND THEN
             write_to_log('No More Data to Fetch from DB');
         END IF;  
        
         write_to_log('v_sdr_records.COUNT : '||v_sdr_records.COUNT );
         write_to_log('Processing Data');
         --Loop on the Bulk records
         FOR indx IN 1 .. v_sdr_records.COUNT LOOP
          
           BEGIN
             -- create the insert statement and execute it
             v_sql := v_sql_type1;           
             v_insert_sql := 'INSERT INTO CDR_ALEC SELECT ' || v_sql || ' FROM DEBUG_GEN_W_NEW_SDR_ALEC WHERE cdr_id = :b1';
             v_cdr_id :=  v_sdr_records(indx).cdr_id;
             EXECUTE IMMEDIATE v_insert_sql USING v_cdr_id;

             --check whether to create 2nd CDR for outgoing leg
             IF v_sdr_records(indx).outgoing_call_direction IS NOT NULL THEN
               v_sql := v_sql_type2;
               v_insert_sql := 'INSERT INTO CDR_ALEC SELECT ' || v_sql || ' FROM DEBUG_GEN_W_NEW_SDR_ALEC WHERE cdr_id = :b1';               
               EXECUTE IMMEDIATE v_insert_sql USING  v_sdr_records(indx).cdr_id;
             END IF; 

             DELETE DEBUG_GEN_W_NEW_SDR_ALEC WHERE DEBUG_GEN_W_NEW_SDR_ALEC.rowid = v_sdr_records(indx).sdr_rowid;
           EXCEPTION
             WHEN OTHERS THEN
               write_to_log('ConvertSdr2CdrBulk Error for CDR_ID: '|| v_sdr_records(indx).cdr_id || ' Error Details: ' ||SQLERRM);
               UPDATE DEBUG_GEN_W_NEW_SDR_ALEC
               SET ROAMING_TYPE ='9'
             WHERE cdr_id =  v_sdr_records(indx).cdr_id;
           END;   
                

        END LOOP;
        EXIT WHEN sdr_cur%NOTFOUND;

     EXCEPTION
       WHEN OTHERS THEN
         write_to_log('Unexpected Error in ConvertSdr2CdrBulk: '||SQLERRM);          
     END;
     
     END LOOP;
     CLOSE sdr_cur;
     COMMIT;
  v_end_date := SYSDATE;
  write_to_log('ConvertSdr2CdrBulk procedure completed at '||TO_CHAR(v_end_date,'YYYYMMDD hh24:mi:ss'));
  v_seconds := ROUND((v_end_date - v_start_date)*1440*60);
  write_to_log('Time to Execute: '||v_seconds||'(sec)');
  

END ConvertSdr2CdrBulk;

END SDR_2_CDR_CONVERT_ALEC;