pcs cluster switch from cluster mode to local mount mode

Stop the cluster mode

root@rock8-19c-2:~>% pcs property set maintenance-mode=true

root@rock8-19c-2:~>% pcs status

Cluster name: igt055_cluster

Status of pacemakerd: 'Pacemaker is running' (last updated 2024-04-11 11:03:14Z)

Cluster Summary:

  * Stack: corosync

  * Current DC: rock8-19c-1 (version 2.1.5-8.1.el8_8-a3f44794f94) - partition with quorum

  * Last updated: Thu Apr 11 11:03:14 2024

  * Last change:  Thu Apr 11 11:03:09 2024 by root via cibadmin on rock8-19c-2

  * 2 nodes configured

  * 6 resource instances configured

              *** Resource management is DISABLED ***

  The cluster will not attempt to start, stop or recover services

Node List:

  * Online: [ rock8-19c-1 rock8-19c-2 ]

Full List of Resources:

  * Resource Group: ora_igt_rg (unmanaged):

    * db1_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * online_igt_fs     (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * exp_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * db2_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * ora_igt_vip       (ocf::heartbeat:IPaddr2):        Started rock8-19c-2 (unmanaged)

    * ora_igt_ap        (lsb:dbora_ctl):         Started rock8-19c-2 (unmanaged)

Daemon Status:

  corosync: active/enabled

  pacemaker: active/enabled

  pcsd: active/enabled

Can change oracle archive mode now

For example:

oracle@rock8-19c-2:~>% sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Apr 11 11:15:45 2024

Version 19.22.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.22.0.0.0

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 8589932704 bytes

Fixed Size                  8960160 bytes

Variable Size            4328521728 bytes

Database Buffers         4244635648 bytes

Redo Buffers                7815168 bytes

Database mounted.

SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> ARCHIVE LOG LIST;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /oracle_db/db1/db_igt/arch

Oldest online log sequence     496

Next log sequence to archive   498

Current log sequence           498

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> exit

Now start cluster mode

root@rock8-19c-2:~>% pcs property set maintenance-mode=false

root@rock8-19c-2:~>% pcs resource show

Warning: This command is deprecated and will be removed. Please use 'pcs resource status' instead.

  * Resource Group: ora_igt_rg:

    * db1_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * online_igt_fs     (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * exp_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * db2_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * ora_igt_vip       (ocf::heartbeat:IPaddr2):        Started rock8-19c-2

    * ora_igt_ap        (lsb:dbora_ctl):         Started rock8-19c-2

root@rock8-19c-2:~>% pcs resource disable ora_igt_rg

root@rock8-19c-2:~>% pcs resource enable ora_igt_rg

root@rock8-19c-2:~>% pcs status

Cluster name: igt055_cluster

Status of pacemakerd: 'Pacemaker is running' (last updated 2024-04-11 12:34:54Z)

Cluster Summary:

  * Stack: corosync

  * Current DC: rock8-19c-1 (version 2.1.5-8.1.el8_8-a3f44794f94) - partition with quorum

  * Last updated: Thu Apr 11 12:34:54 2024

  * Last change:  Thu Apr 11 11:35:59 2024 by root via cibadmin on rock8-19c-2

  * 2 nodes configured

  * 6 resource instances configured

Node List:

  * Online: [ rock8-19c-1 rock8-19c-2 ]

Full List of Resources:

  * Resource Group: ora_igt_rg:

    * db1_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * online_igt_fs     (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * exp_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * db2_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * ora_igt_vip       (ocf::heartbeat:IPaddr2):        Started rock8-19c-2

    * ora_igt_ap        (lsb:dbora_ctl):         Started rock8-19c-2

Daemon Status:

  corosync: active/enabled

  pacemaker: active/enabled

  pcsd: active/enabled

ORA-12547: TNS:lost contact

ERROR:
ORA-12547: TNS:lost contact

=============================================
Correct permissions should be:
ls -l ${ORACLE_HOME}bin/oracle
-rwsr-s--x 1 oracle dba 409028888 Aug 27  2019 bin/oracle

But now:
ls -l ${ORACLE_HOME}bin/oracle
-rwxr-x--x 1 oracle dba 457057360 Mar 28 08:07 /software/oracle/19c/bin/oracle
=============================================

How to fix:
Per Oracle technote "Troubleshooting ORA-12547 TNS: Lost Contact (Doc ID 555565.1)"
Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission
su - oracle
cd $ORACLE_HOME/bin/
chmod 6751 oracle

drbd sync SOW

A. Activity Overview 

1. drdb syncrinization

B. Prerequisite and Pre-checks:

Steps:
1. On servers BD902G and BD901G
pcs status

Server which is running oracle service ora_igt_rg  is Active. (Primary)
The other server is Passive (Secondary)

2. drbd General Note:
Data is written on Active server by oracle service, and synchronized by drbd to Standby server.
drbd has its own CLI, drbdadm

Command are run per LV Group (db2, db1, Ora_Exp, Ora_Online). Each Volume sync can take few minutes to complete.
drbdadm status during synchronization, will show progress, in percent.
drbdadm commands should be run as root.
drbdadm commads are run either on Active (Primary), or on Passive (Secondary) node, per context, see below.

3. Check status before starting synchronization
Run drbdadm status on Active server

Expected result:
connection:StandAlone – meaning there is no sync with the Secondary node

root>% drbdadm status
Ora_Exp role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone
Ora_Online role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone
db1 role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone
db2 role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone

4. Synchronization commands.
Yellow: commands on secondary site
Green: commands on primary site

drbdadm status
drbdadm status

drbdadm secondary Ora_Exp
drbdadm disconnect Ora_Exp
drbdadm -- --discard-my-data connect Ora_Exp
drbdadm connect Ora_Exp
drbdadm status
 
drbdadm secondary Ora_Online
drbdadm disconnect Ora_Online
drbdadm -- --discard-my-data connect Ora_Online
drbdadm connect Ora_Online
drbdadm status

drbdadm secondary db1
drbdadm disconnect db1
drbdadm -- --discard-my-data connect db1
drbdadm connect db1
drbdadm status

drbdadm secondary db2
drbdadm disconnect db2
drbdadm -- --discard-my-data connect db2
drbdadm connect db2
drbdadm status
 
drbdadm status
drbdadm status

1.After synchronization check status
Now, Primary node is aware of secondary node
Secondary node is aware on Primary node.
In this example:
DBD901G - is Primary
DBD902G - is Secondary


root@DBD901G:~>% drbdadm  status
Ora_Exp role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate

Ora_Online role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate

db1 role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate

db2 role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate


root@DBD902G:~>% drbdadm  status
Ora_Exp role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

Ora_Online role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

db1 role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

db2 role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

completed!!

Locks in Oracle

V$LOCKED_OBJECT

SELECT LO.session_id, LO.oracle_username, LO.os_user_name, 

       OBJ.object_name , OBJ.owner
  FROM V$LOCKED_OBJECT LO, 
       DBA_OBJECTS OBJ
WHERE LO.object_id = OBJ.object_id;

V$LOCKED_OBJECT + V$SESSION

SELECT SESS.sid, SESS.serial#, SESS.schemaname, SESS.machine, 

       SESS.program, SESS.module, SESS.blocking_session,

       LO.os_user_name, OBJ.object_name , OBJ.owner,

       SESS.sql_id,

'ALTER SYSTEM KILL SESSION ''' || SESS.sid || ',' || SESS.serial# ||''';' kill_session

  FROM V$LOCKED_OBJECT LO, 

       DBA_OBJECTS OBJ, 

       V$SESSION SESS

WHERE LO.object_id = OBJ.object_id

  AND LO.session_id = SESS.sid;

V$LOCK

SELECT A.sid "blocking session", 

       B.sid "blocked session"  , 

       DECODE (A.block,1, A.sid ||' is Blocking '||B.sid,'Not Blocking'),  

       A.type "Lock Type"

FROM V$LOCK A, 

     V$LOCK B 

WHERE A.sid != B.sid 

  AND A.id1 = B.id1  

  AND A.id2 = B.id2 

  AND A.block = 1

  AND B.request > 0;

V$SESSION - lock time

SELECT 

  sid, serial#, username, machine, program,

  blocking_session "blocking_session",

  sid "blocked_session",

  serial# "blocked_serial#", 

  seconds_in_wait/60 "wait_time(minutes)"

FROM V$SESSION

WHERE blocking_session IS NOT NULL

ORDER BY blocking_session;

V$SESSION - lock time + lock info

SELECT 

  ROUND(V_SESS_LOCKED.seconds_in_wait/60) "WAIT_TIME (MIN)",

  V_SESS_LOCKED.sid       locked_sid, 

  V_SESS_LOCKED.serial#   locked_serial, 

  V_SESS_LOCKED.username  locked_username, 

  V_SESS_LOCKED.machine   locked_machine, 

  V_SESS_LOCKED.program   locked_program,

  V_SESS_LOCKED.blocking_session blocking_session,

  V_SESS_LOCKED.sql_id    locked_sql,

  V_SESS_LOCKING.sid      locking_session,

  V_SESS_LOCKING.serial#  locking_serial,

  V_SESS_LOCKING.username locking_username,

  V_SESS_LOCKING.machine  locking_machine,

  V_SESS_LOCKING.program  locking_program,

  V_SESS_LOCKING.sql_id   locking_sql

FROM V$SESSION V_SESS_LOCKED,

     V$SESSION V_SESS_LOCKING

WHERE V_SESS_LOCKED.blocking_session IS NOT NULL

  AND V_SESS_LOCKED.blocking_session = V_SESS_LOCKING.sid

ORDER BY ROUND(V_SESS_LOCKED.seconds_in_wait/60) DESC;

ORA-01873: the leading precision of the interval is too small

SELECT * FROM USER_JOBS
ORA-01873: the leading precision of the interval is too small

SELECT * FROM DBA_SCHEDULER_JOBS
ORA-01882: timezone region not found

After a removal of the suspected job, and recreating it again the issue was fixed.

"suspected job" - there was a lock in database, and the job session was terminated with ALTER SYSTEM KILL SESSION command.

BEGIN

 DBMS_JOB.remove(23);
 commit;
END;
/
SELECT * FROM USER_JOBS

Now is working!

Oracle Patch types

Oracle Patch types

Interim patches - contain a single bug fix or a collection of bug fixes provided as required

Diagnostic patches - intended to help diagnose or verify a fix

Bundle Patch Updates (BPUs) - a cumulative collection of fixes for a specific product or component

Patch Set Updates (PSUs) - a cumulative collection of high impact, low risk, and proven fixes for a specific product or component and Security Patch Updates

Security Patch Updates (SPU) - a cumulative collection of security bug fixes.  SPUs were formerly known as Critical Patch Updates (CPU).

System Patch - contains several sub-patches in a format that can be used by OPatchAuto.

Merge Label Request (MLR) - a merge of two or more fixes. MLR creation requires a label for the new set of merged code and a Patch Set Exception.

What to install?

RU - Release Update

Release Update

RU - Release Update

Oracle recommends that you keep your database software current by applying Release Updates (RUs).

RUs are listed in Primary Note for Database Proactive Patch Program (Doc ID 888.1)

Release Updates (RUs) are release quarterly: Third Tuesday of January, April, July and October. 

Each RU will be given a maximum of six Monthly Recommended Patches (MRPs), released monthly.

RUs are highly tested bundles of critical fixes which enable you to avoid known issues. 

They usually contain the following type of fixes: security, regression (bug), optimizer, and functional.

Monthly Recommended Patches (MRP)

Starting with update 19.17, Oracle is providing MRPs for Linux x86-64 to provide proactive patching between Release Updates.

Which patches to install for Oracle 19?

A. Oracle Database 19c Important Recommended One-off Patches (Doc ID 555.1)

This documents lists the latest patches for Oracle 19
for example:

19.22 DB RU - Patch 35943157

19.21 DB RU - Patch 35643107 

19.20 DB RU - Patch 35320081

19.19 DB RU - Patch 35042068

B. Primary Note for Database Proactive Patch Program (Doc ID 888.1)

This documents lists patches for All oracle versions.
for example:

ORA-12034: materialized view log on "SCHEMA"."MY_SERVICE" younger than last refresh

Issue: Refresh Materialized View is giving this error:

ERROR at line 1:

ORA-12034: materialized view log on "SCHEMA"."MY_MVIEW" younger than last refresh

ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794

ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851

ORA-06512: at "SYS.DBMS_IREFRESH", line 683

ORA-06512: at "SYS.DBMS_REFRESH", line 195

ORA-06512: at "SCHEMA.REFRESH_PKG", line 19

ORA-06512: at line 2

Solution:

Complete refresh and then Fast Refresh the Materialized View

Example

BEGIN

 DBMS_MVIEW.refresh('MY_MVIEW' ,'C');

END;

/

BEGIN

 DBMS_MVIEW.refresh('MY_MVIEW' ,'F');

END;

/

Code Generator

-----------------------------------------

--Generate DBMS_REFRESH.refresh script

-----------------------------------------

COLUMN sql_cmd FORMAT A120 WORD_WRAP

SET LINESIZE 120

SET PAGESIZE 0

SET HEADING OFF

SET FEEDBACK OFF

SET NEWPAGE NONE

SET VERIFY OFF

SET ECHO OFF

SET SHOW OFF

SET TERMOUT OFF

-----------------------------------------

--Generate DBMS_REFRESH.refresh script

-----------------------------------------

COLUMN sql_cmd FORMAT A120 WORD_WRAP

SET LINESIZE 120

SET PAGESIZE 0

SET HEADING OFF

SET FEEDBACK OFF

SET NEWPAGE NONE

SET ECHO OFF

SET TERMOUT OFF

spool manual_refresh_mviews.sql 

PROMPT SET FEEDBACK ON

SELECT 'spool refresh_'||name||'.log;'||CHR(10)||

'PROMPT Start DBMS_MVIEW.refresh() of '||name||CHR(10)||

'BEGIN'||CHR(10)||

' DBMS_MVIEW.refresh('''||name||''' ,''C'');' ||CHR(10)||

' DBMS_MVIEW.refresh('''||name||''' ,''F'');' ||CHR(10)||

' COMMIT;'||CHR(10)||

'END;'||CHR(10)||

'/'||CHR(10)||

'spool off;' as sql_cmd

FROM DBA_REFRESH_CHILDREN

WHERE rname = 'MASTER_GROUP'

order by name;

PROMPT spool off

spool off;

@manual_refresh_mviews.sql