ORA-15064 ORA-03113 - 测试库案例

-- ORA-15064 ORA-03113测试库案例  
  
错误日志 2013-09-12 16:20:00   
###############  
alert_prod1.log  
###############  
Thu Sep 12 16:19:17 2013  
NOTE: ASMB terminating  
Errors in file /prod/oracle/diag/rdbms/prod/prod2/trace/prod2_asmb_26138.trc:  
ORA-15064: communication failure with ASM instance  
ORA-03113: end-of-file on communication channel  
Process ID:  
Session ID: 119 Serial number: 42229  
Errors in file /prod/oracle/diag/rdbms/prod/prod2/trace/prod2_asmb_26138.trc:  
ORA-15064: communication failure with ASM instance  
ORA-03113: end-of-file on communication channel  
Process ID:  
Session ID: 119 Serial number: 42229  
ASMB (ospid: 26138): terminating the instance due to error 15064  
Instance terminated by ASMB, pid = 26138  
  
###############  
alert_+ASM2.log  
###############  
Thu Sep 12 16:19:17 2013  
NOTE: client exited [4512]  
Thu Sep 12 16:19:17 2013  
NOTE: ASMB process exiting, either shutdown is in progress  
NOTE: or foreground connected to ASMB was killed.  
Thu Sep 12 16:19:18 2013  
Received an instance abort message from instance 1  
Please check instance 1 alert and LMON trace files for detail.  
Thu Sep 12 16:19:18 2013  
Received an instance abort message from instance 1  
Please check instance 1 alert and LMON trace files for detail.  
LMS0 (ospid: 4477): terminating the instance due to error 481  
Instance terminated by LMS0, pid = 4477  
Thu Sep 12 16:20:06 2013  
MEMORY_TARGET defaulting to 285212672.  
* instance_number obtained from CSS = 2, checking for the existence of node 0...  
* node 0 does not exist. instance_number = 2  
Starting ORACLE instance (normal)  
  
#######################  
grid alertq1ebsdb02.log  
#######################  
2013-09-12 16:19:00.781  
[cssd(4138)]CRS-1612:Network communication with node q1ebsdb01 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.300 seconds  
2013-09-12 16:19:07.801  
[cssd(4138)]CRS-1611:Network communication with node q1ebsdb01 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.280 seconds  
2013-09-12 16:19:12.811  
[cssd(4138)]CRS-1610:Network communication with node q1ebsdb01 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.270 seconds  
2013-09-12 16:19:15.088  
[cssd(4138)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /prod/grid/product/11.2.0  
/crs_1/log/q1ebsdb02/cssd/ocssd.log.  
2013-09-12 16:19:15.088  
[cssd(4138)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/cssd/ocssd.log  
2013-09-12 16:19:15.225  
[cssd(4138)]CRS-1652:Starting clean up of CRSD resources.  
2013-09-12 16:19:15.520  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(4659)]CRS-5016:Process "/prod/grid/product/11.2.0/crs_1/bin/lsnrctl" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin" for  
action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oragrid/oraagent_oragrid.log"  
2013-09-12 16:19:15.519  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(4822)]CRS-5016:Process "/prod/oracle/product/11.2.0/db_1/bin/lsnrctl" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin" for  
 action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oraprod/oraagent_oraprod.log"  
2013-09-12 16:19:16.088  
[cssd(4138)]CRS-1608:This node was evicted by node 1, q1ebsdb01; details at (:CSSNM00005:) in /prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/cssd/ocssd.log.  
2013-09-12 16:19:16.512  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(4659)]CRS-5016:Process "/prod/grid/product/11.2.0/crs_1/opmn/bin/onsctli" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin"  
 for action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oragrid/oraagent_oragrid.log"  
2013-09-12 16:19:17.128  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(4659)]CRS-5016:Process "/prod/grid/product/11.2.0/crs_1/bin/lsnrctl" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin" for  
action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oragrid/oraagent_oragrid.log"  
2013-09-12 16:19:17.141  
[cssd(4138)]CRS-1654:Clean up of CRSD resources finished successfully.  
2013-09-12 16:19:17.142  
[cssd(4138)]CRS-1655:CSSD on node q1ebsdb02 detected a problem and started to shutdown.  
  
错误日志 2013-09-17 18:24:00   
###############  
alert_prod2.log  
###############  
Tue Sep 17 18:24:56 2013  
Errors in file /prod/oracle/diag/rdbms/prod/prod2/trace/prod2_asmb_22443.trc:  
ORA-15064: communication failure with ASM instance  
ORA-03113: end-of-file on communication channel  
Errors in file /prod/oracle/diag/rdbms/prod/prod2/trace/prod2_asmb_22443.trc:  
ORA-15064: communication failure with ASM instance  
ORA-03113: end-of-file on communication channel  
  
###############  
alert_+ASM2.log  
###############  
Tue Sep 17 18:24:56 2013  
NOTE: client exited [3309]  
Tue Sep 17 18:24:56 2013  
NOTE: ASMB process exiting, either shutdown is in progress  
NOTE: or foreground connected to ASMB was killed.  
opidcl aborting process unknown ospid (3320) as a result of ORA-29709  
Tue Sep 17 18:24:57 2013  
PMON (ospid: 3248): terminating the instance due to error 481  
Instance terminated by PMON, pid = 3248  
Tue Sep 17 18:25:30 2013  
MEMORY_TARGET defaulting to 285212672.  
* instance_number obtained from CSS = 2, checking for the existence of node 0...  
* node 0 does not exist. instance_number = 2  
  
#######################  
grid alertq1ebsdb02.log  
#######################  
[cssd(3003)]CRS-1612:Network communication with node q1ebsdb01 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.210 seconds  
2013-09-17 18:24:42.621  
[cssd(3003)]CRS-1611:Network communication with node q1ebsdb01 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.180 seconds  
2013-09-17 18:24:47.631  
[cssd(3003)]CRS-1610:Network communication with node q1ebsdb01 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.170 seconds  
2013-09-17 18:24:49.808  
[cssd(3003)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /prod/grid/product/11.2.0  
/crs_1/log/q1ebsdb02/cssd/ocssd.log.  
2013-09-17 18:24:49.808  
[cssd(3003)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/cssd/ocssd.log  
2013-09-17 18:24:49.951  
[cssd(3003)]CRS-1652:Starting clean up of CRSD resources.  
2013-09-17 18:24:50.955  
[cssd(3003)]CRS-1608:This node was evicted by node 1, q1ebsdb01; details at (:CSSNM00005:) in /prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/cssd/ocssd.log.  
2013-09-17 18:24:51.345  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(3428)]CRS-5016:Process "/prod/grid/product/11.2.0/crs_1/opmn/bin/onsctli" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin"  
 for action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oragrid/oraagent_oragrid.log"  
2013-09-17 18:24:51.999  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(3428)]CRS-5016:Process "/prod/grid/product/11.2.0/crs_1/bin/lsnrctl" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin" for  
action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oragrid/oraagent_oragrid.log"  
2013-09-17 18:24:52.002  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(3428)]CRS-5016:Process "/prod/grid/product/11.2.0/crs_1/bin/lsnrctl" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin" for  
action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oragrid/oraagent_oragrid.log"  
2013-09-17 18:24:56.280  
[/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin(3612)]CRS-5016:Process "/prod/oracle/product/11.2.0/db_1/bin/lsnrctl" spawned by agent "/prod/grid/product/11.2.0/crs_1/bin/oraagent.bin" for  
 action "check" failed: details at "(:CLSN00010:)" in "/prod/grid/product/11.2.0/crs_1/log/q1ebsdb02/agent/crsd/oraagent_oraprod/oraagent_oraprod.log"  
2013-09-17 18:24:56.289  
[cssd(3003)]CRS-1654:Clean up of CRSD resources finished successfully.  
2013-09-17 18:24:56.290  
[cssd(3003)]CRS-1655:CSSD on node q1ebsdb02 detected a problem and started to shutdown.  

原因分析：
1，grid的日志文件其实已经显示真正的原因了，节点2心跳网络与节点1失去联系，超过了30秒，导致rac出现了隔离，节点2被强制驱离，rac重启节点2的所有资源，包括ASM与DB的实例。
2，17号当天傍晚有网络调试，出现了短暂网络中断。