pcs cluster switch from cluster mode to local mount mode

Stop the cluster mode

root@rock8-19c-2:~>% pcs property set maintenance-mode=true

root@rock8-19c-2:~>% pcs status

Cluster name: igt055_cluster

Status of pacemakerd: 'Pacemaker is running' (last updated 2024-04-11 11:03:14Z)

Cluster Summary:

  * Stack: corosync

  * Current DC: rock8-19c-1 (version 2.1.5-8.1.el8_8-a3f44794f94) - partition with quorum

  * Last updated: Thu Apr 11 11:03:14 2024

  * Last change:  Thu Apr 11 11:03:09 2024 by root via cibadmin on rock8-19c-2

  * 2 nodes configured

  * 6 resource instances configured

              *** Resource management is DISABLED ***

  The cluster will not attempt to start, stop or recover services

Node List:

  * Online: [ rock8-19c-1 rock8-19c-2 ]

Full List of Resources:

  * Resource Group: ora_igt_rg (unmanaged):

    * db1_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * online_igt_fs     (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * exp_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * db2_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2 (unmanaged)

    * ora_igt_vip       (ocf::heartbeat:IPaddr2):        Started rock8-19c-2 (unmanaged)

    * ora_igt_ap        (lsb:dbora_ctl):         Started rock8-19c-2 (unmanaged)

Daemon Status:

  corosync: active/enabled

  pacemaker: active/enabled

  pcsd: active/enabled

Can change oracle archive mode now

For example:

oracle@rock8-19c-2:~>% sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Apr 11 11:15:45 2024

Version 19.22.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.22.0.0.0

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 8589932704 bytes

Fixed Size                  8960160 bytes

Variable Size            4328521728 bytes

Database Buffers         4244635648 bytes

Redo Buffers                7815168 bytes

Database mounted.

SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> ARCHIVE LOG LIST;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /oracle_db/db1/db_igt/arch

Oldest online log sequence     496

Next log sequence to archive   498

Current log sequence           498

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> exit

Now start cluster mode

root@rock8-19c-2:~>% pcs property set maintenance-mode=false

root@rock8-19c-2:~>% pcs resource show

Warning: This command is deprecated and will be removed. Please use 'pcs resource status' instead.

  * Resource Group: ora_igt_rg:

    * db1_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * online_igt_fs     (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * exp_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * db2_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * ora_igt_vip       (ocf::heartbeat:IPaddr2):        Started rock8-19c-2

    * ora_igt_ap        (lsb:dbora_ctl):         Started rock8-19c-2

root@rock8-19c-2:~>% pcs resource disable ora_igt_rg

root@rock8-19c-2:~>% pcs resource enable ora_igt_rg

root@rock8-19c-2:~>% pcs status

Cluster name: igt055_cluster

Status of pacemakerd: 'Pacemaker is running' (last updated 2024-04-11 12:34:54Z)

Cluster Summary:

  * Stack: corosync

  * Current DC: rock8-19c-1 (version 2.1.5-8.1.el8_8-a3f44794f94) - partition with quorum

  * Last updated: Thu Apr 11 12:34:54 2024

  * Last change:  Thu Apr 11 11:35:59 2024 by root via cibadmin on rock8-19c-2

  * 2 nodes configured

  * 6 resource instances configured

Node List:

  * Online: [ rock8-19c-1 rock8-19c-2 ]

Full List of Resources:

  * Resource Group: ora_igt_rg:

    * db1_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * online_igt_fs     (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * exp_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * db2_igt_fs        (ocf::heartbeat:Filesystem):     Started rock8-19c-2

    * ora_igt_vip       (ocf::heartbeat:IPaddr2):        Started rock8-19c-2

    * ora_igt_ap        (lsb:dbora_ctl):         Started rock8-19c-2

Daemon Status:

  corosync: active/enabled

  pacemaker: active/enabled

  pcsd: active/enabled

ORA-12547: TNS:lost contact

ERROR:
ORA-12547: TNS:lost contact

=============================================
Correct permissions should be:
ls -l ${ORACLE_HOME}bin/oracle
-rwsr-s--x 1 oracle dba 409028888 Aug 27  2019 bin/oracle

But now:
ls -l ${ORACLE_HOME}bin/oracle
-rwxr-x--x 1 oracle dba 457057360 Mar 28 08:07 /software/oracle/19c/bin/oracle
=============================================

How to fix:
Per Oracle technote "Troubleshooting ORA-12547 TNS: Lost Contact (Doc ID 555565.1)"
Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission
su - oracle
cd $ORACLE_HOME/bin/
chmod 6751 oracle

drbd sync SOW

A. Activity Overview 

1. drdb syncrinization

B. Prerequisite and Pre-checks:

Steps:
1. On servers BD902G and BD901G
pcs status

Server which is running oracle service ora_igt_rg  is Active. (Primary)
The other server is Passive (Secondary)

2. drbd General Note:
Data is written on Active server by oracle service, and synchronized by drbd to Standby server.
drbd has its own CLI, drbdadm

Command are run per LV Group (db2, db1, Ora_Exp, Ora_Online). Each Volume sync can take few minutes to complete.
drbdadm status during synchronization, will show progress, in percent.
drbdadm commands should be run as root.
drbdadm commads are run either on Active (Primary), or on Passive (Secondary) node, per context, see below.

3. Check status before starting synchronization
Run drbdadm status on Active server

Expected result:
connection:StandAlone – meaning there is no sync with the Secondary node

root>% drbdadm status
Ora_Exp role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone
Ora_Online role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone
db1 role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone
db2 role:Primary
  disk:UpToDate
  DBD902G connection:StandAlone

4. Synchronization commands.
Yellow: commands on secondary site
Green: commands on primary site

drbdadm status
drbdadm status

drbdadm secondary Ora_Exp
drbdadm disconnect Ora_Exp
drbdadm -- --discard-my-data connect Ora_Exp
drbdadm connect Ora_Exp
drbdadm status
 
drbdadm secondary Ora_Online
drbdadm disconnect Ora_Online
drbdadm -- --discard-my-data connect Ora_Online
drbdadm connect Ora_Online
drbdadm status

drbdadm secondary db1
drbdadm disconnect db1
drbdadm -- --discard-my-data connect db1
drbdadm connect db1
drbdadm status

drbdadm secondary db2
drbdadm disconnect db2
drbdadm -- --discard-my-data connect db2
drbdadm connect db2
drbdadm status
 
drbdadm status
drbdadm status

1.After synchronization check status
Now, Primary node is aware of secondary node
Secondary node is aware on Primary node.
In this example:
DBD901G - is Primary
DBD902G - is Secondary


root@DBD901G:~>% drbdadm  status
Ora_Exp role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate

Ora_Online role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate

db1 role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate

db2 role:Primary
  disk:UpToDate
  SRV902G role:Secondary
    peer-disk:UpToDate


root@DBD902G:~>% drbdadm  status
Ora_Exp role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

Ora_Online role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

db1 role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

db2 role:Secondary
  disk:UpToDate
  DBD901G role:Primary
    peer-disk:UpToDate

completed!!