Welcome Guest ( Log In | Register )


 
 
 
 
 
 

 
 
Oracle 

Performance Tuning Reference poster
 
Oracle training in Linux 

commands
 
Oracle training Weblogic Book
 
Easy Oracle Jumpstart
 
Oracle training & performance tuning books
 
Burleson Consulting Remote DB Administration
 
 
 
Reply to this topicStart new topic
> Problem with Oracle Grid Infrastructure, Single Instance 2 node cluster with ASM
MATH
post Oct 11 2017, 04:08 AM
Post #1


Newbie
*

Group: Possible Spammers
Posts: 1
Joined: 11-October 17
Member No.: 52,201



Hi. Ive recently taken over an Oracle Grid Infrastructure solution containing a single db instance shared between 2 nodes running ASM. After reboot of the 2 nodes, ssh connectivity between the 2 nodes was lost, and the the 2 nodes were seing each other as offline.

We re-established the ssh connection between the nodes, but were not able to get the nodes to see each other as an online cluster member, and there were not showing up in olsnodes.

Ive tried removing the secondary node, removing grid infrastructure software, and addnode but with same result.... One this I am also confused about is that the setup doesnt contain a mgmt database...


Any ideas ?


Details:

Clusterware Software Version is : 12.1.0.2.0


node1:

[root@node1 ~]# crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@node1 ~]# olsnodes
node1

[oracle@node1 bin]$ olsnodes node2
PRCO-15: Not a cluster member

[root@node1 ~]# crsctl relocate res MYDB.db -n node2
CRS-2546: Server 'node2' is not online
CRS-4000: Command Relocate failed, or completed with errors.

node2:

[root@node2 ~]# crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@node2 ~]# olsnodes
node2

[oracle@node2 bin]$ olsnodes node1
PRCO-15: Not a cluster member


When running ols.pl:


Node1:

[oracle@node1 ~]$ ./ols.pl -f -v
Local Time Now : 2017-10-10 09:48:18



The Cluster Nodes are : node1
The Local Node is : node1
The Remote Nodes are :

Major Clusterware Software Version is : 12.1.0.2.0
Major Clusterware Active Version is : 12.1.0.2.0
Major Clusterware Release Version is : 12.1.0.2.0

CRS_HOME is installed at : /software/cluster/12.1.0.2/grid
CRS_BASE is installed at : /data/oracle/grid
CRS_OWNER is : oracle
CRS_GROUP is : oinstall

ORACLE_HOMES[0] is installed at : /software/db/12.1.0.2/db_1 (on Local FS)
ORACLE_BASES[0] is installed at : /data/oracle (on Local FS)
ORACLE_OWNERS[0] is : oracle
ORACLE_GROUPS[0] is : oinstall


All databases created :


NODE_NAME NODE_ID NODE_STATE NODE_ROLE
========= ======= ========== =========
node1 1 Active Hub



Cluster Name : clu01
SCAN Name : clu01
SCAN Listeners : LISTENER_SCAN1 (Port: TCP:1521)
LISTENER_SCAN2 (Port: TCP:1521)
LISTENER_SCAN3 (Port: TCP:1521)
GNS Status : not configured



Node VIP Version : 12.1.0.2.0
Local Node VIPs : ora.node1.vip node1-vip (static)



NIC Subnet Netmask Type
=== ====== ======= ====
Oracle Interfaces : <REMOVED> global public
eth1 <REMOVED> global cluster_interconnect

eth0 eth1
==== ====
node1 : <REMOVED>

OCR Location : '+DATA'

Voting Disk Location : '+DATA'


Cluster Mode : Standard Cluster with Standard ASM

Hub Node connects Leaf Node
======== ======== =========
node1(1,Active) <--- None


WARNING: Error occurred during running "/software/cluster/12.1.0.2/grid/bin/srvctl status mgmtdb" :
PRCD-1120 : The resource for database _mgmtdb could not be found.
PRCR-1001 : Resource ora.mgmtdb does not exist
256
WARNING: Error occurred during running "/software/cluster/12.1.0.2/grid/bin/srvctl config mgmtdb" :
PRCD-1120 : The resource for database _mgmtdb could not be found.
PRCR-1001 : Resource ora.mgmtdb does not exist
256
MGMTDB Status : disabled and isn't running now
MGMTDB HOME : N/A
MGMTDB Spfile : N/A
MGMTDB Instance : N/A

MGMTLSNR Status : enabled and is running on node1
MGMTLSNR HOME : <CRS home>
MGMTLSNR Port : TCP:1521
Detailed state : <REMOVED>


WARNING: Error occurred during running "/software/cluster/12.1.0.2/grid/bin/srvctl config listener -asmlistener -all" :
PRCN-2044 : No listener exists
256


Subnet :
End points :
Owner :
Home :
Status :


DISKGROUP REDUNDANCY AU COMPATIBILITY DB_COMPATIBILITY SIZE_MB FREE_MB USABLE_MB PATH
========= ========== ==== ============= ================ ======= ======= ========= ====
ACFS EXTERN 1MB 12.1.0.0.0 11.2.0.0.0 10239 1948 1948 /dev/oracleasm/disks/ACFS001
DATA EXTERN 4MB 12.1.0.0.0 10.1.0.0.0 2047960 828928 828928 /dev/oracleasm/disks/DATA008
/dev/oracleasm/disks/DATA003
/dev/oracleasm/disks/DATA001
/dev/oracleasm/disks/DATA009
/dev/oracleasm/disks/DATA010
/dev/oracleasm/disks/DATA002
/dev/oracleasm/disks/DATA006
/dev/oracleasm/disks/DATA005
/dev/oracleasm/disks/DATA007
/dev/oracleasm/disks/DATA004
FRA EXTERN 4MB 11.2.0.0.0 11.2.0.0.0 614388 357360 357360 /dev/oracleasm/disks/FRA003
/dev/oracleasm/disks/FRA002
/dev/oracleasm/disks/FRA001


VOLUME_NAME VOLUME_DEVICE DG_NAME MOUNTPOINT USAGE VOL_SIZE_MB
=========== ============= ======= ========== ===== ===========
ACFS /dev/asm/acfs-195 ACFS USMCA_NULL USMCA_NULL 8192


ASM Host connects Client
======== ======== ======
node1(+ASM1) <--- '+ASM1(node1)'
node1(+ASM1) <--- 'MYDB'


CSS Master : node1

OCR/CRSD Master : node1

CRSD PE Master : node1

CRSD PE Standby : N/A

<node1> 10-09 22:03:39.418486 My R:3 = Min R:3 R of OCR:1 R of CRS Standby:0 R of ASM Inst:2 OCR on ASM:1 ASM mode:1

<node1> 10-09 22:04:00.397371 PE Role|State Update: old role [MASTER] new [MASTER]; old state [Starting] new [Running]

CTSS Master : node1

UI Master : node1

ONS Master : node1

ONSNET Master : node1

CHM Master : N/A

CHM Replica : REPLICA has been deprecated from 12c

CSS Diagwait : 0 s
CSS Disktimeout : 200 s
CSS Misscount : 30 s
CSS Leafmisscount : 30 s
CSS Reboottime : 3 s

OCR Local/Writer connects ASM Instance
================ ======== ============
node1(Hub,OCR Writer) ---> node1(+ASM1)


node2:


[oracle@node2 ~]$ ./ols.pl -f -v
Local Time Now : 2017-10-10 09:51:54

The Cluster Nodes are : node2
The Local Node is : node2
The Remote Nodes are :

Major Clusterware Software Version is : 12.1.0.2.0
Major Clusterware Active Version is : 12.1.0.2.0
Major Clusterware Release Version is : 12.1.0.2.0

CRS_HOME is installed at : /software/cluster/12.1.0.2/grid
CRS_BASE is installed at : /data/oracle/grid
CRS_OWNER is : oracle
CRS_GROUP is : oinstall

All databases created :

NODE_NAME NODE_ID NODE_STATE NODE_ROLE
========= ======= ========== =========
node2 2 Active Hub

Cluster Name : clu01
SCAN Name : clu01
SCAN Listeners : LISTENER_SCAN1 (Port: TCP:1521)
LISTENER_SCAN2 (Port: TCP:1521)
LISTENER_SCAN3 (Port: TCP:1521)
GNS Status : not configured

Node VIP Version : 12.1.0.2.0
Local Node VIPs : ora.node1.vip node1-vip (static)
ora.node2.vip node2-vip (static)

NIC Subnet Netmask Type
=== ====== ======= ====
Oracle Interfaces : eth0 <REMOVED> global public
eth1 <REMOVED> global cluster_interconnect

eth0 eth1
==== ====
node2 : <REMOVED>

OCR Location : '+DATA'

Voting Disk Location : '+DATA'

Cluster Mode : Standard Cluster with Standard ASM

Hub Node connects Leaf Node
======== ======== =========
node2(2,Active) <--- None


WARNING: Error occurred during running "/software/cluster/12.1.0.2/grid/bin/srvctl status mgmtdb" :
PRCD-1120 : The resource for database _mgmtdb could not be found.
PRCR-1001 : Resource ora.mgmtdb does not exist
256
WARNING: Error occurred during running "/software/cluster/12.1.0.2/grid/bin/srvctl config mgmtdb" :
PRCD-1120 : The resource for database _mgmtdb could not be found.
PRCR-1001 : Resource ora.mgmtdb does not exist
256
MGMTDB Status : disabled and isn't running now
MGMTDB HOME : N/A
MGMTDB Spfile : N/A
MGMTDB Instance : N/A

MGMTLSNR Status : enabled and is running on node2
MGMTLSNR HOME : <CRS home>
MGMTLSNR Port : TCP:1521
Detailed state : <REMOVED>

WARNING: Error occurred during running "/software/cluster/12.1.0.2/grid/bin/srvctl config listener -asmlistener -all" :
PRCN-2044 : No listener exists
256

Subnet :
End points :
Owner :
Home :
Status :

DISKGROUP REDUNDANCY AU COMPATIBILITY DB_COMPATIBILITY SIZE_MB FREE_MB USABLE_MB PATH
========= ========== ==== ============= ================ ======= ======= ========= ====
ACFS EXTERN 1MB 12.1.0.0.0 11.2.0.0.0 10239 1948 1948 /dev/oracleasm/disks/ACFS001
DATA EXTERN 4MB 12.1.0.0.0 10.1.0.0.0 2047960 832116 832116 /dev/oracleasm/disks/DATA009
/dev/oracleasm/disks/DATA002
/dev/oracleasm/disks/DATA003
/dev/oracleasm/disks/DATA010
/dev/oracleasm/disks/DATA007
/dev/oracleasm/disks/DATA001
/dev/oracleasm/disks/DATA008
/dev/oracleasm/disks/DATA006
/dev/oracleasm/disks/DATA005
/dev/oracleasm/disks/DATA004
FRA EXTERN 4MB 11.2.0.0.0 11.2.0.0.0 614388 358224 358224 /dev/oracleasm/disks/FRA001
/dev/oracleasm/disks/FRA002
/dev/oracleasm/disks/FRA003

VOLUME_NAME VOLUME_DEVICE DG_NAME MOUNTPOINT USAGE VOL_SIZE_MB
=========== ============= ======= ========== ===== ===========
ACFS /dev/asm/acfs-195 ACFS USMCA_NULL USMCA_NULL 8192

ASM Host connects Client
======== ======== ======
node2(+ASM2) <--- '+ASM2(node2)'
node2(+ASM2) <--- 'MYDB'


CSS Master : node2

OCR/CRSD Master : node2

CRSD PE Master : node2

CRSD PE Standby : N/A

<node2> 10-10 00:29:19.048687 My R:3 = Min R:3 R of OCR:1 R of CRS Standby:0 R of ASM Inst:2 OCR on ASM:1 ASM mode:1

<node2> 10-10 00:29:19.681778 PE Role|State Update: old role [MASTER] new [MASTER]; old state [Starting] new [Running]

CTSS Master : node2

UI Master : node2

ONS Master : node2

ONSNET Master : node2

CHM Master : node2

CHM Replica : REPLICA has been deprecated from 12c

CSS Diagwait : 0 s
CSS Disktimeout : 200 s
CSS Misscount : 30 s
CSS Leafmisscount : 30 s
CSS Reboottime : 3 s

OCR Local/Writer connects ASM Instance
================ ======== ============
node2(Hub,OCR Writer) ---> node2(+ASM2)
Go to the top of the page
 
+Quote Post
burleson
post Oct 11 2017, 05:06 AM
Post #2


Advanced Member
***

Group: Members
Posts: 13,577
Joined: 26-January 04
Member No.: 13



Hi math, and welcome to the forum!

This forum is to ask specific Oracle questions, but your post looks like a request for services!

For issues like this, you will want to open a service request on MOSC:

http://support.oracle.com.

From what I noticed, it looks like the listeners were not running.

Can't you simply re-start the cluster?


--------------------
Hope this helps . . .

Donald K. Burleson
Oracle Press author
Author of Oracle Tuning: The Definitive Reference
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 

Lo-Fi Version Time is now: 23rd October 2017 - 02:00 PM