Oracle Server - Enterprise Edition - Version: 10.1.0.4 to 11.1.0.7 Information in this document applies to any platform. Oracle Server Enterprise Edition - Version: 10.1.0.4 to 11.1.0.7
Symptoms
The command : crs_stat -t output shows VIP is offline and trying to start it gives error : CRS-0215: Could not start resource 'ora.dbtest2.vip'.
Example: crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....st2.gsd application ONLINE ONLINE dbtest2 ora....st2.ons application ONLINE ONLINE dbtest2 ora....st2.vip application ONLINE OFFLINE
# ./srvctl start nodeapps -n dbtest2 dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2) dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2) dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2) dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2) CRS-1006: No more members to consider CRS-0215: Could not start resource 'ora.dbtest2.vip'. CRS-0210: Could not find resource ora.dbtest2.LISTENER_DBTEST2.lsnr.
To know what is the nodeapps configuration in particular for VIP, issue the command: srvctl config nodeapps -n <nodename> -a -g -s -l Example: /u01/crs/bin/srvctl config nodeapps -n dbtest2 -a -g -s -l VIP exists.: /dbtest2-vip/10.11.11.198/255.255.255.0/eri0 GSD exists. ONS daemon exists. Listener does not exist.
Further debug it by uncommenting the environment variable _USR_ORA_DEBUG=1 in the script $ORA_CRS_HOME/bin/racgvip OR simply as root user, issue the command : crsctl debug log res "ora.dbtest2.vip:5" You may turn off debugging with command : crsctl debug log res "ora.dbtest2.vip:0" Start the VIP using srvctl start nodeapps again. This will create a log for VIP starting problem for 10.2 and above version in directory $ORA_CRS_HOME/log/<nodename>/racg/*vip.log
Example: last lines of the *.vip.log shows -
2005-02-09 20:38:06.711: [ RACG][1] [5602][1][ora.dbtest2.vip]: 203800 [ 5604 ] Checking interface existance 203800 [ 5604 ] Calling getifbyip 203800 [ 5604 ] getifbyip: started for 10.11.11.198 203800 [ 5604 ] getifbyip: returning IF eri0:1 203800 [ 5604 ] Completed getifbyip eri0:1 203801 [ 5604 ] Completed with in 2005-02-09 20:38:06.711: [ RACG][1] [5602][1][ora.dbtest2.vip]: itial interface test 203801 [ 5604 ] checkIf: start for if=eri0 203801 [ 5604 ] checkIf: -z defaultgw 203801 [ 5604 ] defaultgw: started 203801 [ 5604 ] defaultgw: completed with 10.11.11.1 203801 [ 5604 ] checkIf: -n defaultgw 203804 [ 5604 ] checkIf: 2005-02-09 20:38:06.711: [ RACG][1] [5602][1][ora.dbtest2.vip]: in while, before sleep 203805 [ 5604 ] checkIf: in while, before sleep 203806 [ 5604 ] checkIf: checked if=eri0 failed Interface eri0 checked failed (host=dbtest2) 203806 [ 5604 ] checkIf: end for if=eri0 203806 [ 5604 ] Performing CRS_STAT testing 203806 2005-02-09 20:38:06.711: [ RACG][1] [5602][1][ora.dbtest2.vip]: [ 5604 ] Completed CRS_STAT testing 203806 [ 5604 ] Completed second gateway test 203806 [ 5604 ] Interface tests
Failed to start VIP 10.11.11.198 (host=dbtest2)
From above we see the VIP:10.11.11.198 is correct however oracle function checkIf is failing when trying to reach the default gateway IP:10.11.11.1
Changes
Either the default gateway has got changed to someother IP -or-
It is on a different network from the client (the network where the VIP is configured on) -or-
It is not being used anymore.
Cause
By default, the server's default gateway is used as a ping target during the Oracle RAC 10g VIP status check action. Upon a ping failure, Oracle will decide that the current interface where the VIP is running has failed, and will initiate an interface / internode VIP failover.
In above case, we used just one node for CRS installation, so the VIP coudn't failover to other nodes and thus reported additional error like : CRS-1006: No more members to consider CRS-0215: Could not start resource 'ora.dbtest2.vip'.
Solution
10.1.0.4 and above introduced a parameter FAIL_WHEN_DEFAULTGW_NOT_FOUND in the file $ORA_CRS_HOME/bin/racvip to address this problem.
The following steps will fix the VIP starting problem for above mentioned scenario.
1- stop nodeapps 2- As root, vi the script $ORA_CRS_HOME/bin/racgvip and change the value of variable FAIL_WHEN_DEFAULTGW_NOT_FOUND=0 . 3- start nodeapps and you should see the resources ONLINE
You may proceed with netca and dbca to create a RAC database after this.