Pages

Sunday, June 18, 2023

Data Pump is stuck at RBA: 0, Timestamp: Not Available

 
GGSCI 44> SEND DPM_S_01 STATUS
Sending STATUS request to EXTRACT DPM_S_01 ...
EXTRACT DPM_S_01 (PID 2637)
  Current status: Recovery complete: At EOF
  Current read position:
  Sequence #: 441
  RBA: 0
  Timestamp: Not Available
  Extract Trail: /software/ogg/191/dirdat/01/out/es
ALTER EXTRACT DPM_S_01, BEGIN NOW

GGSCI 45> INFO DPM_S_01

EXTRACT    DPM_S_01  Last Started 2023-06-18 20:12   Status STOPPED
Checkpoint Lag       00:00:00 (updated 00:00:03 ago)
Log Read Checkpoint  File /software/ogg/191/dirdat/01/out/es000000441
                     First Record  RBA 0

GGSCI 46> SEND DPM_S_01 STATUS

Sending STATUS request to EXTRACT DPM_S_01 ...
EXTRACT DPM_S_01 (PID 2637)
  Current status: Recovery complete: At EOF

  Current read position:
  Sequence #: 441
  RBA: 0
  Timestamp: Not Available
  Extract Trail: /software/ogg/191/dirdat/01/out/es

  Current write position:
  Sequence #: 63214
  RBA: 0
  Timestamp: 2023-06-18 20:14:21.631210
  Extract Trail: /software/ogg/191/dirdat/01/in/es


GGSCI 49> ALTER EXTRACT DPM_S_01, BEGIN NOW
EXTRACT altered.

GGSCI 50> START EXTRACT DPM_S_01
Sending START request to MANAGER ...
EXTRACT DPM_S_01 starting


EXTRACT    DPM_S_01  Last Started 2023-06-18 20:15   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:06 ago)
Process ID           7431
Log Read Checkpoint  File /software/ogg/191/dirdat/01/out/es000032125
                     2023-06-18 17:29:50.354524  RBA 181742

GGSCI 55> SEND DPM_S_01 STATUS
Sending STATUS request to EXTRACT DPM_S_01 ...
EXTRACT DPM_S_01 (PID 7431)
  Current status: Recovery complete: At EOF
  Current read position:
  Sequence #: 32125
  RBA: 182808
  Timestamp: 2023-06-18 20:15:46.000000
  Extract Trail: /software/ogg/191/dirdat/01/out/es
  Current write position:
  Sequence #: 63214
  RBA: 4759
  Timestamp: 2023-06-18 20:15:55.830421
  Extract Trail: /software/ogg/191/dirdat/01/in/es

Data Pump Extract STATS "No active extraction maps". - 2nd case

GGSCI 258> SEND DPM_S_01 STATS
Sending STATS request to EXTRACT DPM_S_01 ...
No active extraction maps.

GGSCI 259> SEND DPM_S_01 STATUS
Sending STATUS request to EXTRACT DPM_S_01 ...
EXTRACT DPM_S_01 (PID 5923)
  Current status: In recovery[1]: At EOF
  Current read position:
  Sequence #: 0
  RBA: 0
  Timestamp: Not Available
  Extract Trail: /software/ogg/191/dirdat/01/out/es  < local server:  /software/ogg/191/dirdat/01/out/es000032125
  Current write position:
  Sequence #: 0
  RBA: 0
  Timestamp: 2023-06-18 16:33:50.811444
  Extract Trail: /software/ogg/191/dirdat/01/in/es


GGSCI 264> STOP EXTRACT DPM_S_01 (or SEND EXTRACT DPM_S_01, FORCESTOP )
GGSCI 265> ALTER EXTRACT DPM_S_01, EXTSEQNO 32125 EXTRBA 0

GGSCI 266> START EXTRACT DPM_S_01

GGSCI 267> SEND DPM_S_01 STATUS

Sending STATUS request to EXTRACT DPM_S_01 ...
EXTRACT DPM_S_01 (PID 3307)
  Current status: Recovery complete: At EOF
  Current read position:
  Sequence #: 32125
  RBA: 120958
  Timestamp: 2023-06-18 19:17:46.000000
  Extract Trail: /software/ogg/191/dirdat/01/out/es
  Current write position:
  Sequence #: 63213
  RBA: 123599
  Timestamp: 2023-06-18 19:18:29.275072
  Extract Trail: /software/ogg/191/dirdat/01/in/es

Extract Data Pump: "No active extraction maps" - Just no traffic...

GGSCI> SEND DPM_I_01 STATS
No active extraction maps.

GGSCI> SEND DPM_I_01 STATUS

Sending STATUS request to EXTRACT DPM_I_01 ...
EXTRACT DPM_I_01 (PID 23644)
  Current status: Recovery complete: At EOF
  Current read position:
  Sequence #: 865
  RBA: 96421
  Timestamp: 2023-06-18 18:54:07.000000
  Extract Trail: /software/ogg/191/dirdat/01/out/ei  < On the local server
  Current write position:
  Sequence #: 4622
  RBA: 98279
  Timestamp: 2023-06-18 18:54:50.694153
  Extract Trail: /software/ogg/191/dirdat/01/in/ei < on remote server

Update to a table...  
Now there is data

GGSCI > SEND DPM_I_01 STATS

Sending STATS request to EXTRACT DPM_I_01 ...

Start of Statistics at 2023-06-18 19:03:24.

Output to /software/ogg/191/dirdat/01/in/ei:

Extracting from OGG.GG_HEARTBEAT_SEED to OGG.GG_HEARTBEAT_SEED:

*** Total statistics since 2023-06-18 17:29:55 ***
        Total inserts                                      0.00
        Total updates                                     96.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                  96.00

*** Daily statistics since 2023-06-18 17:29:55 ***
        Total inserts                                      0.00
        Total updates                                     96.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                  96.00

*** Hourly statistics since 2023-06-18 19:00:00 ***
        Total inserts                                      0.00
        Total updates                                      4.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                   4.00

*** Latest statistics since 2023-06-18 17:29:55 ***
        Total inserts                                      0.00
        Total updates                                     96.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                  96.00

Extracting from MY_USER.SFI_CUSTOMER_PROFILE to MY_USER.SFI_CUSTOMER_PROFILE:

*** Total statistics since 2023-06-18 17:29:55 ***
        Total inserts                                      0.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                   1.00

*** Daily statistics since 2023-06-18 17:29:55 ***
        Total inserts                                      0.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                   1.00

*** Hourly statistics since 2023-06-18 19:00:00 ***
        Total inserts                                      0.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                   1.00

*** Latest statistics since 2023-06-18 17:29:55 ***
        Total inserts                                      0.00
        Total updates                                      1.00
        Total deletes                                      0.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                   1.00

End of Statistics.

Tuesday, June 13, 2023

ORA-00379 no free buffers available in buffer pool for block size 16k

Error :ORA-00379 no free buffers available in buffer pool for block size 16k 

General
There has been no memory allocated to 8K or 16K or 32K block buffers cache. 
Explicitly allocating memory to the non-default block buffers will resolve the ORA-00379: no free buffers available in buffer pool DEFAULT for block size 8K or 16K or 32K errors

To see current status:
SELECT name, value 
  FROM V$PARAMETER 
 WHERE name LIKE '%cache_size';

To resolve the issue:
set DB_nK_CACHE_SIZE to a certain value and restart the instance
( Replace the n with the failing block size 8K or 16K or 32K as appropriate )

For example:

ALTER SYSTEM SET  DB_16K_CACHE_SIZE=100M SCOPE = SPFILE;

Restart the instance.


Sunday, June 4, 2023

buffer busy waits vs free buffer waits

buffer busy waits vs free buffer waits

buffer busy waits - waites to application process to commit/rollback on block
free buffer waits - waites to DBWR process to wtite dirty blocks

=================
Buffer busy waits (%)
=================
This wait happens when a session wants to access a database block in the buffer cache but it cannot because the buffer is busy. 

Another session is modifying the block and the contents of the block are in flux during the modification. 
The session modifying the block marks the block header with a flag letting other users know a change is taking place and to wait until the complete change is applied.
While the block is being changed, the block is marked as unreadable by others.

The problem is when there is a hot block, such as the first block on the free list of a table, with high concurrent inserts.

The two main cases where this wait can occur are:
Another session is reading the block into the buffer.
Another session holds the buffer in an incompatible mode to our request.

Solution:
1. Tune the UNDO tablespace
2. Check I/O

===================
Estimating for UNDO size
===================
1. SQL 1
  
V$UNDOSTAT
SELECT 'Required Undo Tablespace Size Using Statistics: '
 || DBMS_UNDO_ADV.required_undo_size(3600)||'Mb' required_undo_size FROM DUAL;

Required Undo Tablespace Size Using Statistics: 36936 MB

2. SQL 2
SELECT ROUND(d.undo_size/(1024*1024)) "ACTUAL UNDO SIZE [MByte]",
      SUBSTR(e.value,1,25) "UNDO RETENTION [Sec]",
      ROUND((TO_NUMBER(e.value) * TO_NUMBER(f.value) * g.undo_block_per_sec) / (1024*1024))  "NEEDED UNDO SIZE [MByte]"
 FROM (
   SELECT SUM(a.bytes) undo_size
     FROM v$datafile a,
          v$tablespace b,
          dba_tablespaces c
    WHERE c.contents = 'UNDO'
      AND c.status = 'ONLINE'
      AND b.name = c.tablespace_name
      AND a.ts# = b.ts#
  ) d,
 v$parameter e,
 v$parameter f,
 (
 SELECT MAX(undoblks/((end_time-begin_time)*3600*24)) undo_block_per_sec FROM v$undostat
 ) g
 WHERE e.name = 'undo_retention'
 AND f.name = 'db_block_size';

 
ACTUAL UNDO SIZE [Mb] UNDO RETENTION [Sec]     NEEDED UNDO SIZE [MB]
--------------------- ------------------------ ---------------------
                32768                   3600                 32562


Actual size of UNDO tablespace
SELECT maxbytes/1024/1024 AS max_mb  
  FROM DBA_DATA_FILES 
 WHERE tablespace_name LIKE '%UNDO%';

max_mb
------
32767

Check UNDO sergments status
SELECT status,
       ROUND (sum_bytes / (1024*1024), 0) as MB,
       
ROUND ((sum_bytes / undo_size) * 100, 0) as PERC
  FROM
(
  
SELECT status, sum(bytes) sum_bytes
    
FROM DBA_UNDO_EXTENTS
  group by status
),
(
  
SELECT sum(a.bytes) undo_size
    FROM DBA_TABLESPACES c
    join v$tablespace b on b.name = c.tablespace_name
    join v$datafile a on a.ts# = b.ts#
   WHERE c.contents = 'UNDO'
     AND c.status = 'ONLINE'
);

STATUS            MB       PERC
--------- ---------- ----------
ACTIVE            63          0
EXPIRED         6837         21
UNEXPIRED      20246         62

ACTIVE extents
- the system is using 
UNEXPIRED extents - are used for read consistency 
EXPIRED extents - can be reused.

SELECT status,
       ROUND(sum_bytes / (1024*1024), 0) as MB,
       ROUND((sum_bytes / undo_size) * 100, 0) as PERC,
       DECODE(status, 
       'UNEXPIRED', ROUND((sum_bytes / undo_size * factor) * 100, 0),
       'EXPIRED',   0,
       ROUND((sum_bytes / undo_size) * 100, 0)) FULL
FROM
(
 
SELECT status, sum(bytes) sum_bytes
   FROM DBA_UNDO_EXTENTS
 GROUP BY status
),
(
 select sum(a.bytes) undo_size
 from dba_tablespaces c
 join v$tablespace b on b.name = c.tablespace_name
 join v$datafile a on a.ts# = b.ts#
 where c.contents = 'UNDO'
 and c.status = 'ONLINE'
),
(
 select tuned_undoretention, u.value, u.value/tuned_undoretention factor
 from v$undostat us
 join (select max(end_time) end_time from v$undostat) usm
    on usm.end_time = us.end_time
 join (select name, value from v$parameter) u
    on u.name = 'undo_retention'
);

STATUS            MB       PERC       FULL
--------- ---------- ---------- ----------
UNEXPIRED      19997         61         61
EXPIRED         7214         22          0
ACTIVE           102          0          0


SELECT tuned_undoretention, u.value, 
       u.value/tuned_undoretention factor
FROM V$UNDOSTAT US
JOIN (select max(end_time) end_time FROM V$UNDOSTAT) usm
on usm.end_time = us.end_time
JOIN (select name, value FROM V$PARAMETER) u
on u.name = 'undo_retention';

tuned_undoretention      value     factor
------------------- ---------- ---------- 
               3600       3600          1

--Longest running query
SELECT max(maxquerylen)logest_query_sec 
  FROM v$undostat;

logest_query_sec
----------------
            1939

   
--Estimate UNDO based on logest_query_sec - estimate to avoid ORA-01555
SELECT
  ROUND(MAX(undoblks/600)*8192*max(maxquerylen)/(1024*1024)) 
  AS "UNDO in MB"
FROM V$UNDOSTAT;

UNDO in MB
----------
     17538
 

Find SQL responsible for buffer busy waits
 
SELECT SESS.sql_hash_value, 
       SW.p1 file#, 
       SW.p2 block#, 
       SW.p3 reason
  FROM V$SESSION_WAIT SW,
       V$SESSION SESS
 WHERE 
SW.event = 'buffer busy waits'
   AND 
SW.sid = SESS.sid;


===============================
By example
===============================

Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                           Total Wait       Avg   % DB Wait
Event                                Waits Time (sec)      Wait   time Class
------------------------------ ----------- ---------- --------- ------ --------
DB CPU                                         3764.6             63.3
db file sequential read          1,020,186     1066.9    1.05ms   17.9 User I/O
log file sync                       42,985      826.1   19.22ms   13.9 Commit
log buffer space                       383      155.6  406.23ms    2.6 Configur
buffer busy waits                  200,170      113.6  567.55us    1.9 Concurre
enq: TX - index contention          15,792       52.3    3.31ms     .9 Concurre
SQL*Net more data from client    3,418,373       48.5   14.18us     .8 Network
direct path read                    13,588       40.7    3.00ms     .7 User I/O
SQL*Net message from dblink          2,580       34.9   13.53ms     .6 Network
cursor: mutex S                     31,054         34    1.10ms     .6 Concurre



=============
log buffer space
=============
The log buffer space wait event occurs when server processes write data into the log buffer faster than the LGWR process can write it out
As the LGWR process writes entries to disk, user processes can reuse the space in the log buffer for new entries. 
If the Log Buffer is too small, user processes wait for Log Buffer Space until the LGWR flushes the redo information in memory to disk.
The Log Buffer Space wait event could be an indication of slow disks and/or slow log file switches.

SELECT name, value FROM V$PARAMETER 
 WHERE name LIKE 'log_buffer%' 
 ORDER BY by name;

name        value
----------- --------------
log_buffer  111370240

=============
Log File Sync
=============
"log file sync" event is the time it takes for the log writer (LGWR) to write to the redo log file. 
The Oracle "log file sync" wait event is triggered when a user session issues a commit (or a rollback). 
The user session will signal or post the LGWR to write the log buffer to the redo log file. 
When the LGWR has finished writing, it will post the user session. 
The wait is entirely dependent on LGWR to write out the necessary redo blocks and send confirmation of its completion back to the user session. 
The wait time is sometimes called "commit latency".

The P1 parameter in V$SESSION_WAIT is defined as follows for the log file sync wait event:

P1 = buffer#
All changes up to this buffer number (in the log buffer) must be flushed to disk and the writes confirmed.
The wait is for LGWR to flush up to this buffer#.

=================
Reducing Log File Sync
=================
What to check?

Average time for this event should be examined. 
If the average wait time is low, but the number of waits is high, then the application might be committing after every row, rather than batching COMMITs. 

Reducing the overall number of commits by batching transactions can be very beneficial.

If the SQL statement is a SELECT statement, review the Oracle Auditing settings. 
If Auditing is enabled for SELECT statements, Oracle could be spending time writing and commit data to the AUDIT$ table.

SELECT * FROM V$SESSION_WAIT 
 WHERE event LIKE '%sync%';

If the average wait time is high, then examine the other log related waits for the session, to see where the session is spending most of its time. 

If a session continues to wait on the same buffer# then the SEQ# column of V$SESSION_WAIT should increment every second. 

If not then the local session has a problem with wait event timeouts. 

If the SEQ# column is incrementing then the blocking process is the LGWR process. 
Check to see what LGWR is waiting on as it may be stuck.

Try to reduce resource contention. 
Check the number of transactions (commits + rollbacks) each second, from V$SYSSTAT.
SELECT * FROM V$SYSSTAT WHERE name LIKE '%transac%' AND value > 0

Increase the size for  log files, up to several Gb, say 8Gb, can reduce the time and occurrence of wait events and improve performance.
Oracle recommends 3 to 4 switches per hour. 
Higher frequency can increase log file sync waits.

Is the application committing to often?
If the application commits to often it can cause high waits on log file sync since each commit flushes redo data from the redo buffer to the redo logs. 
Oracle has a recommendation that user calls per (commits+rollbacks) should not be lower than 30, if it is the application is committing to frequent. 

For example:

Key Instance Activity Stats               DB/Inst: IGT/igt  Snaps: 31891-31892
-> Ordered by statistic name
Statistic                                     Total     per Second     per Trans
-------------------------------- ------------------ -------------- -------------
db block changes                         73,448,532       20,359.6         977.8
execute count                             9,482,394        2,628.5         126.2
logons cumulative                             2,521            0.7           0.0
opened cursors cumulative                   320,466           88.8           4.3
parse count (total)                       2,151,065          596.3          28.6
parse time elapsed                           13,731            3.8           0.2
physical reads                            3,944,428        1,093.4          52.5
physical writes                           8,680,011        2,406.1         115.6
redo size                            39,938,539,788   11,070,768.7     531,663.2
session cursor cache hits                   338,288           93.8           4.5
session logical reads                   165,008,626       45,739.6       2,196.6
user calls                               15,048,551        4,171.4         200.3
user commits                                 75,116           20.8           1.0
user rollbacks                                    4            0.0           0.0
workarea executions - onepass                     0            0.0           0.0
workarea executions - optimal               390,480          108.2           5.2

15,048,551/(75,116+4)=200
avg user calls per commit.

Thursday, June 1, 2023

sftp from server to server with crontab task

===============
Send files from server A to server B from crontab
===============
Step A - Generate a commands file
Step B - Execute this commands file from bash script


generate_sftp_script.sh

#!/bin/bash
REMOTE_SERVER=back_server_01
REMOTE_PATH=/starhome/ES/ora_online
WORK_DIR=/home/shdaemon/ftp_to_mng
BACKUP_DIR=/backup/ora_online/for_backup
SFTP_SCRIPT=ftp_commands.txt
LOG_FILE=sftp_to_mng.log
RUN_DATE=`date +"%Y%m%d"_"%H%M%S"`

backup_dir_last=`ls -ltr ${BACKUP_DIR} | grep -v log | grep 202 | tail -1 | awk '{print $9}'`

cd ${WORK_DIR}
echo "echo Begin sftp to $REMOTE_SERVER at $RUN_DATE" >>$LOG_FILE
rm -f ${SFTP_SCRIPT} 2>/dev/null
echo "lcd ${BACKUP_DIR}/${backup_dir_last}" >> ${SFTP_SCRIPT}
echo "mkdir  ${backup_dir_last}" >>${SFTP_SCRIPT}
echo "cd ${backup_dir_last}" >> ${SFTP_SCRIPT}
for f in `ls -1 $BACKUP_DIR/$backup_dir_last/`
do
 echo "put $f" >> ${SFTP_SCRIPT}
done
echo "bye" >> ${SFTP_SCRIPT}

main_sftp.sh
#!/bin/bash
sftp -b ftp_commands.txt someuser@back_server_01

generated ftp_commands.txt
lcd /backup/ora_online/for_backup/20230601_0300
cd /starhome/ES/ora_online
mkdir  20230601_0300
cd 20230601_0300
put file_1
put file_2
put file_3
put file_4
bye