由于os的netmask与ocr中的netmask不匹配而导致的ora.net1.network启动失败

CRS-2674 Clusterware Network Resource ora.net1.network Could not Start (文档 ID 1270186.1)

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

SYMPTOMS

2 node RAC on 11.2.0.1, after the network switch upgraded from 1GB to 10GB, clusterware network resource cannot be started:

[grid@node01 bin]$ ./srvctl start nodeapps -n node01
PRCR-1013 : Failed to start resource ora.net1.network
PRCR-1064 : Failed to start resource ora.net1.network on node node01
CRS-2674: Start of 'ora.net1.network' on 'node01' failed

<GRID_HOME>/log/<node>/crsd/crsd.log shows:

2010-11-07 23:07:45.066: [ CRSPE][1542510912] Sequencer for [ora.net1.network node01 1] has completed with error: CRS-0215: Could not start resource 'ora.net1.network'.

2010-11-07 23:07:45.066: [UiServer][1546713408] Container [ Name: ORDER
   MESSAGE:
   TextMessage[CRS-2674: Start of 'ora.net1.network' on 'node01' failed]
   MSGTYPE:
   TextMessage[1]
   OBJID:
   TextMessage[ora.ons node01 1]
   WAIT:
   TextMessage[0]
]

CAUSE

The problem is caused by netmask mismatch between ifconfig and OCR config.

In OCR:

node01_OCRDUMP
---------
[SYSTEM.CRSD.RESOURCES.ora!net1!network.CONFIG]
USR_ORA_IF=bond0~USR_ORA_NETMASK=255.255.255.192~USR_ORA_SUBNET=192.168.1.0~VERSION=11.2.0.1.0

# ifconfig

bond0 Link encap:Ethernet HWaddr 00:00:49:6F:E3:A5
inet addr:192.168.1.7 Bcast:192.168.1.127 Mask:255.255.255.128

OCR shows  netmask of 255.255.255.192 and ifconfig shows netmask of 255.255.255.128. Due to this mismatch, the ora.net1.network resource cannot be started.

Please note, if subnet IP is setting wrongly, it could also cause the same problem.

For example:

crsctl stat res ora.net1.network -f shows:

USR_ORA_IF=en7 
USR_ORA_NETMASK=255.255.254.0 
USR_ORA_SUBNET=10.16.68.89

While oifcfg iflist shows: 

en7   10.16.68.0
...

So the subnet IP for en7 should be 10.16.68.0, not 10.16.68.89.

SOLUTION

1. If OCR entry is correct, modify NETMASK setting for bond0 interface
in /etc/sysconfig/network-scripts/ifcfg-bond0, change NETMASK value from 255.255.255.128 to 255.255.255.192.

2. If OCR entry needs to be modified to match the OS setting, then use srvctl modify nodeapps command as root user. For example:

# srvctl modify nodeapps -n node01 -A node01-vip/255.255.255.128/bond0

For 11.2.0.2 onward, one can modify the network resource directly:
# srvctl modify network -k 1 -S <subnet IP>/255.255.255.128/bond0

Once network resource is modified, node VIP netmask will be changed automatically.

Above command can also be used to modify subnet IP if it is set wrongly, for example:
# srvctl modify network -k 1 -S 10.16.68.0/255.255.254.0/en7

3. After they match, network resource should be able to start.
$ ./srvctl start nodeapps -n node01