Oracle RAC心跳机制

1、网络心跳(Network HeartBeat,NHB)

       首先是确定集群节点之间的连通性，以便节点之间能够了解彼此的状态，而对于Oracle集群，这是通过节点间的网络心跳来实现的。对于Oracle集群，ocssd.bin守护进程每秒钟向集群的其他节点发送网络心跳（当然是通过集群的私网）。

[root@lissen-node2 trace]# ps -ef | grep ocssd.bin | grep -v grep
grid      5081     1  1 19:43 ?        00:01:07 /u19/app/19.2.0.0.0/grid/bin/ocssd.bin

[root@lissen-node2 trace]# grep NHB ocssd.trc
2019-11-28 22:37:37.228 :    CSSD:1378293504: [     INFO] clssnmCheckSplit: Node 1, lissen-node1, dead, last DHB (1574951827, 800434) after NHB (1574951827, 800284), but within reboot time 3000
2019-11-28 22:37:37.229 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434
2019-11-28 22:37:37.366 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434
2019-11-28 22:37:37.467 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434
2019-11-28 22:37:37.567 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434
2019-11-28 22:37:37.644 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434
2019-11-28 22:37:37.644 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434
2019-11-28 22:37:37.745 :    CSSD:1378293504: [     INFO] clssnmCheckKillStatus: Node 1, lissen-node1, down, due to timeout of DHB; last NHB TOD invariant clock 800284, TOD invariant clock 800434

2、磁盘心跳(Disk HeartBeat,DHB)

       磁盘心跳的主要目的就是当集群脑裂时帮助制定脑裂的解决方案。Oracle集群的每一个节点每秒钟都会向集群的所有表决盘注册本地节点的磁盘心跳信息（也就是说，所有VF [Voting File]中的信息是相同的），同时也会将自己能够联系到的集群中其他节点的信息，或者说本节点认为集群中的成员列表信息写入到表决盘中。一旦发生脑裂，CSS的重新配置线程就可以通过表决盘中的信息了解集群中节点之间的连通性，从而决定集群会分裂成几个子集群，以及每个子集群包含的节点情况和每个节点的状态。磁盘心跳是通过ocssd.bin守护进程来实现的。

root@lissen-node2 trace]# grep "DHB" ocssd.trc
2019-11-28 20:26:51.293 :    CSSD:2399300800: [     INFO] clssnmvDHBValidateNCopy: node 1, lissen-node1, has a disk HB, but no network HB, DHB has rcfg 470438296, wrtcnt, 1198, LATS 3797414, lastSeqNo 0, uniqueness 1574943471, timestamp 1574944010/3901454
2019-11-28 20:26:55.444 :    CSSD:2399300800: [     INFO] clssnmvDHBValidateNcopy: Saving DHB uniqueness for node(1/lissen-node1), latestInfo(1574943471), readInfo(1574943471), nodeInfoDHB(0)
2019-11-28 20:26:55.445 :    CSSD:2399300800: [     INFO] clssnmvDHBValidateNcopy: Setting LATS valid due to second DHB seen on disk(0x7fa97c3365a0) for node(1/lissen-node1) nodeStatus 0x3
......
2019-11-28 20:27:03.538 :    CSSD:1708447488: [     INFO] clssnmvDHBValidateNCopy: node 1, lissen-node1, has a disk HB, but no network HB, DHB has rcfg 470438296, wrtcnt, 1216, LATS 3809654, lastSeqNo 1207, uniqueness 1574943471, timestamp 1574944022/3913564
2019-11-28 20:36:07.604 :    CSSD:1384389824: [     INFO] clssnmvDHBValidateNCopy: node 1, lissen-node1, h1582630987 to node structure for node lissen-node1, number 1; previous unique value was 0
2020-02-25 19:43:57.557 :    CSSD:3694849216: [     INFO] clssnmvDHBValidateNCopy: node 1, lissen-node1, has a disk HB, but no network HB, DHB has rcfg 478125832, wrtcnt, 21815, LATS 4294942300, lastSeqNo 21780, uniqueness 1582630987, timestamp 1582631036/4294947150

3、本地心跳(Local HeartBeat,LHB)

       这种心跳机制是在11gR2版本中被引入的。这种心跳的作用是监控ocssd.bin进程以及本地节点的状态。在10g版本中，Oracle通过oclsomon和oprocd来实现。守护进程oclsomon.bin监控ocssd.bin进程的状态，oprocd.bin进程监控本节点是否出现了性能问题（例如，hang住）。从11.2.0.1版本开始，新的集群cssdagent和cssdmonitor被引入，它们的功能就是监控本地节点的ocssd.bin进程状态和本地节点的状态（是否hang住）。

[root@lissen-node2 trace]# ps -ef | grep cssdagent | grep -v grep
root      5045     1  0 19:43 ?        00:00:10 /u19/app/19.2.0.0.0/grid/bin/cssdagent
[root@lissen-node2 trace]# ps -ef | grep cssdmonitor | grep -v grep
root      5008     1  0 19:43 ?        00:00:10 /u19/app/19.2.0.0.0/grid/bin/cssdmonitor

4、网络心跳misscount和磁盘心跳disktimeout查询及设置

查询网络心跳NHB misscount
[root@lissen-node1 ~]# crsctl get css misscount;
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.

查询磁盘心跳DHB disktimeout
[root@lissen-node1 ~]# crsctl get css disktimeout;
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.

修改网络心跳NHB misscount
[root@lissen-node1 ~]# crsctl set css misscount 50;

CRS-4684: Successful set of parameter misscount to 50 for Cluster Synchronization Services.

[root@lissen-node1 ~]# crsctl set css misscount 30;
CRS-4684: Successful set of parameter misscount to 30 for Cluster Synchronization Services.

修改磁盘心跳disktimeout

crsctl set css disktimeout 300

5、脑裂和健忘

脑裂：网络心跳misscount(私网不通)会导致的问题

       由于网络心跳misscount会导致，每个节点都认为其他节点不存在了，这时候表决盘被引入，通过投票机制，获得最高票数或者最早到达的获得投票的，幸存，其他节点被踢出。在Oracle RAC中Voting Disk用来记录节点间成员的状态，出现脑裂时，仲裁哪个partition获得控制权，其他的partition被剔除,之后ORACLE RAC就会很不友善的给踢出去的机子重启，这就是为什么有时候机子会不断重启的原因。

[root@lissen-node2 trace]# tail -f alert.log 
2020-02-25 21:05:19.612 [OCSSD(5081)]CRS-1605: CSSD voting file is online: /dev/asm-disk5; details in /u19/app/grid/diag/crs/lissen-node2/crs/trace/ocssd.trc.
2020-02-25 21:05:19.635 [OCSSD(5081)]CRS-1605: CSSD voting file is online: /dev/asm-disk3; details in /u19/app/grid/diag/crs/lissen-node2/crs/trace/ocssd.trc.
2020-02-25 21:05:19.635 [OCSSD(5081)]CRS-1605: CSSD voting file is online: /dev/asm-disk4; details in /u19/app/grid/diag/crs/lissen-node2/crs/trace/ocssd.trc.

…

健忘：节点间配置文件不一致导致的。