一个CRS节点无法安装的故障分析

关注

原创 2024-03-04 11:47:54 931 阅读 AI 写同款· GEO 优化 ›

记一次因网卡心跳故障引发RAC节点重启故障分析 网卡心跳故障引发RAC节点重启故障分析,kernel: NETDEV WATCHDOG: eth1: transmit timed out,eth1: transmit timed out 阅读详情

前些天有一个朋友安装CRS的时候无法找到节点：

$./cluvfy/runcluvfy.sh stage -post hwos -n node1,node2 -verbose

notice: JAVA_HOME not set in the environment.

Performingpost-checks for hardware and operating system setup

Checking nodereachability...

Check: Nodereachability from node "node1"

DestinationNode                     Reachable?

------------------------------------ ------------------------

node1                                no

node2                                no

Result: Node reachability check failed from node "node1".

碰到这个问题，首先我问他hostname是否就是Node1，node2，而且这两个地址是否正确的在/etc/hosts里定义了。他回答是这些都没错，然后我问他网卡的顺序是否正确，在HP-UX下有时候私网和公网的网卡在两台服务器上的设备名不对应，也可能会出一些莫名其妙的错误。经过确认都没问题。

检查hosts文件和services文件的属性，也都没问题，world有r的权限。看样子问题不是出在这些地方。那么下面该怎么处理呢？

在常规方法无法找到问题的时候，使用操作系统跟踪工具对某个出问题的配置过程进行跟踪是十分有效的方法，我建议他用tusc跟踪一下，看看能有什么发现。

很快在tusc里面我们发现了疑点：

stat("/usr/sbin/ping",0x742ff5e4) ....................................................................= 0

access("/usr/sbin/ping", X_OK)........................................................................ = 0

pipe() ................................................................................................= 9 (10)

pipe()................................................................................................= 11 (12)

pipe()................................................................................................= 13 (14)

vfork() ............................................................ (returningas child ...) ......... , 2452

execve("/tmp/2437/jre/bin/HPUXChildWrapper", 0x1923d0, 0xf5230)....................................... [entry]

                             argv[0] @ 0x192410: "/tmp/2437/jre/bin/HPUXChildWrapper"

                             argv[1] @ 0x742ff460: "9"

                             argv[2] @ 0x742ff470: "12"

                             argv[3] @ 0x742ff480: "14"

                             argv[4] @ 0xffffffffc1be1140:

                             argv[5] @ 0x742ff038: "/usr/sbin/ping"

                             argv[6] @ 0x192170: "node1"

                             argv[7] @ 0x192188: "-n"

                             argv[8] @ 0x1921a0: "1"

                             argv[9] @ 0x1921e8: "-m"

                             argv[10] @ 0x192200: "3"

                              env[0] @ 0x77ff00f2: "_=/tmp/2437/jre/bin/java"

                              env[1] @ 0x77ff010b:"MANPATH=/usr/share/man/%L:/usr/share/man:/usr/contrib/man/%L:/usr/contrib/man:/usr/local/man/%L:/usr/local/man:/opt/mx/share/man:/opt/upgrade/share/man/%L:/opt/upgrade/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/resmon/share/man:/opt/gnome/man:/opt/openssl/man:/opt/openssl/prngd/man:/opt/wbem/share/man:/opt/hparray/share/man/%L:/opt/hparray/share/man:/opt/graphics/common/man:/usr/dt/share/man:/opt/samba/man:/opt/perl/man:/opt/ignite/share/man/%L:/opt/ignite/share/man:/opt/ssh/share/man"

                              env[2] @ 0x77ff034e: "SHLIB_PATH=/tmp/2437/jre/lib/PA_RISC2.0:/tmp/2437/jre/lib/PA_RISC2.0/server:/tmp/2437/jre/../lib/PA_RISC2.0:/tmp/2437/lib32:/tmp/2437/srvm/lib32:"

                              env[3] @ 0x77ff03e0: "PATH=/usr/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/hparray/bin:/opt/nettladm/bin:/opt/upgrade/bin:/opt/fcms/bin:/opt/pd/bin:/opt/resmon/bin:/opt/gnome/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/mozilla:/opt/wbem/bin:/opt/wbem/sbin:/opt/graphics/common/bin:/usr/sbin/diag/contrib:/opt/mx/bin:/opt/perl/bin:/opt/ssh/bin:."

                              env[4] @ 0x77ff051e: "COLUMNS=125"

                              env[5] @ 0x77ff052a: "ORACLE_BASE=/home/oracle/base"

                             env[6] @ 0x77ff0548: "EDITOR=vi"

                              env[7] @ 0x77ff0552: "LOGNAME=oracle"

                              env[8] @ 0x77ff0561: "CV_DESTLOC=/tmp"

                              env[9] @ 0x77ff0571: "ERASE=^H"

                              env[10] @ 0x77ff057a: "CV_JDKHOME=/tmp/2437/jre"

                              env[11] @ 0x77ff0593: "CRS_HOME=/home/oracle/crsHome"

                              env[12] @ 0x77ff05b1: "SHELL=/sbin/sh"

                              env[13] @ 0x77ff05c0: "HOME=/home/oracle"

                              env[14] @ 0x77ff05d2: "LD_LIBRARY_PATH=/tmp/2437/lib:"

                              env[15] @ 0x77ff05f1: "TERM=vt100"
                              env[16] @ 0x77ff05fc: "CV_HOME=/tmp/2437"
                              env[17] @ 0x77ff060e: "PWD=/home/oracle"
                              env[18] @ 0x77ff061f: "TZ=EAT-8"
                              env[19] @ 0x77ff0628: "LINES=47"
                              env[20] @ 0xffffffffc1be0fa8:
                              env[21] @ 0xffffffffc1be0fd0:

从上述的TRACE我们看出，检查过程中调用了Ping，估计CRS安装时候使用PING命令去检查节点的健康性。从后面的参数来看，我们看到其中有-n -m等ping的参数，于是我让他执行一下/usr/sbin/pingnode1 -n 1 -m 3 。

反馈的信息让我眼前一亮，居然发现ping没有-m选项：

# ping node1 -n1

PING node1: 64 byte packets

64 bytes from 192.168.1.150: icmp_seq=0. time=0. ms

----node1 PINGStatistics----

1 packets transmitted, 1 packets received, 0% packet loss

round-trip (ms) min/avg/max = 0/0/0

# ping node1 -n 1 -m 3

Usage: ping [-oprv] [-i address] [-t ttl] host [-n count]

ping [-oprv] [-i address] [-t ttl]host packet-size [[-n] count]