CRS-1006 , CRS-0215 故障一例

安装好sles 10 sp3 + Oracle 10g RAC之后，在配置监听器时，总是提示主机bo2dbp上的监听服务已经在运行，忽略错误之后手动在bo2dbp节点上启，总是收到TNS-12545: Connect failed because target host or object does not exist错误信息。后来发现节点bo2dbp的vip总是漂移到另一节点bo2dbs上，原来这才是罪魁祸首 ??????

1、集群环境netca之后，节点bo2dbp监听无法启动
手动尝试启动监听仍不成功，查看了listener.ora以及监听日志未发现任何异常，查看主机/etc/hosts配置也正常
无奈之下，删除监听器
oracle@bo2dbp:~> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....BP.lsnr application ONLINE OFFLINE
ora.bo2dbp.gsd application ONLINE ONLINE bo2dbp
ora.bo2dbp.ons application ONLINE ONLINE bo2dbp
ora.bo2dbp.vip application ONLINE ONLINE bo2dbs
ora....BS.lsnr application ONLINE ONLINE bo2dbs
ora.bo2dbs.gsd application ONLINE ONLINE bo2dbs
ora.bo2dbs.ons application ONLINE ONLINE bo2dbs
ora.bo2dbs.vip application ONLINE ONLINE bo2dbs

2、节点bo2dbp.vip被漂移到bo2dbs
oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.bo2dbp.gsd application ONLINE ONLINE bo2dbp
ora.bo2dbp.ons application ONLINE ONLINE bo2dbp
ora.bo2dbp.vip application ONLINE ONLINE bo2dbs
ora.bo2dbs.gsd application ONLINE ONLINE bo2dbs
ora.bo2dbs.ons application ONLINE ONLINE bo2dbs
ora.bo2dbs.vip application ONLINE ONLINE bo2dbs

3、尝试为bo2dbp进行relocate也不听使唤
oracle@bo2dbp:~/product/10.2.0/crs_1/bin> ./crs_relocate ora.bo2dbp.vip -c bo2dbp
Attempting to stop `ora.bo2dbp.vip` on member `bo2dbs`
Stop of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.
Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`
Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.
Attempting to start `ora.bo2dbp.vip` on member `bo2dbs`
Start of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.
CRS-0217: Could not relocate resource 'ora.bo2dbp.vip'.

4、crs_stop 出马
oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_stop ora.bo2dbp.vip
Attempting to stop `ora.bo2dbp.vip` on member `bo2dbs`
Stop of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_start ora.bo2dbp.vip
Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`
Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.
Attempting to start `ora.bo2dbp.vip` on member `bo2dbs`
Start of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

我晕，又漂移到bo2dbs节点

5、狠一点，直接将节点2bo2dbs的crs停掉，这下没得漂了吧 :)
bo2dbs:/u01/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

6、露出马脚
oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_start ora.bo2dbp.vip
Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`
Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.
Remote start for `ora.bo2dbp.vip` failed on member `bo2dbs`
CRS-1006: No more members to consider

CRS-0215: Could not start resource 'ora.bo2dbp.vip'.

7、check log

oracle@bo2dbp:~/product/10.2.0/crs_1/log/bo2dbp/racg> tail -50 ora.bo2dbp.vip.log

2012-08-30 16:18:55.942: [ RACG][4250161648] [20571][4250161648][ora.bo2dbp.vip]: end for resource = ora.bo2dbp.vip, action = start, status = 1, time = 6.390s

2012-08-30 16:19:30.968: [ RACG][4041843184] [20878][4041843184][ora.bo2dbp.vip]: checkIf: Default gateway is not defined (host=bo2dbp)
Interface eth0 checked failed (host=bo2dbp)
Invalid parameters, or failed to bring up VIP (host=bo2dbp)

原来醉鬼祸首尽然是没有缺省网关，我倒倒倒......
记得之前的安装就碰到过在节点2上没有缺省网关导致vipca不能继续的情形。这次又碰上了。汗。。。
oracle@bo2dbp:~> cat /etc/sysconfig/network/routes
cat: /etc/sysconfig/network/routes: No such file or directory
oracle@bo2dbp:~> ssh bo2dbs cat /etc/sysconfig/network/routes
default 192.168.7.254 - -

使用root帐户为其追加网关后，一切ok.

oracle@bo2dbs:~> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.bo2dbp.gsd application ONLINE ONLINE bo2dbp
ora.bo2dbp.ons application ONLINE ONLINE bo2dbp
ora.bo2dbp.vip application ONLINE ONLINE bo2dbp
ora.bo2dbs.gsd application ONLINE ONLINE bo2dbs
ora.bo2dbs.ons application ONLINE ONLINE bo2dbs
ora.bo2dbs.vip application ONLINE ONLINE bo2dbs

8、问题环境
sles 10/sp3 + Oracle 10g(10.2.0.4)

9、小结:
a、细心细心再细心
b、根据相应的日志快速定位故障