某客户管理系统Oracle RAC节点异常重启问题详细分析记录

一、故障概述

        某日10:58分左右客户管理系统数据库节点1所有实例异常重启，重启后业务恢复正常。经过分析发现，此次实例异常重启的是数据库节点1。

二、故障原因分析

1、数据库日志分析

        从节点1的数据库日志来看，10:58:49的时候数据库进程开始被abort，最终PMON进程因为481错误而终止实例,这个报错一般表示网络问题。

alert_reportdb1.log：

***********************************************************************

Sat Dec 07 10:58:49 XXXX

***********************************************************************

Fatal NI connect error 12537, connecting to:

 (LOCAL=NO)

Fatal NI connect error 12537, connecting to:

 (LOCAL=NO)

Fatal NI connect error 12537, connecting to:

 (LOCAL=NO)

  VERSION INFORMATION:

TNS for Linux: Version 11.2.0.4.0 - Production

Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production

TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production

TNS-12537: TNS:connection closed

    ns main err code: 12537

    ns secondary err code: 12560

TNS-12537: TNS:connection closed  

    nt main err code: 0

    ns secondary err code: 12560

    nt secondary err code: 0

    nt main err code: 0

TNS-12537: TNS:connection closed

    nt OS err code: 0

    nt secondary err code: 0

    ns secondary err code: 12560

    nt OS err code: 0

    nt main err co