oracle 11g RAC 双机双柜使用NFS解决仲裁磁盘

最新推荐文章于 2023-11-07 16:23:36 发布

原创最新推荐文章于 2023-11-07 16:23:36 发布 · 1.7k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#oracle 11g

rac 专栏收录该内容

13 篇文章

订阅专栏

本文通过实验展示了在Oracle RAC环境中实现ASM高可用性的方法。特别关注了使用NFS作为额外投票磁盘来增强系统冗余度的步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

概述：日常实践安装rac中当碰到双机双柜没有购买其他第三方存储软件时，需要使用oracle自动的asm存储管理方式时如何做到冗余度？我们可以使用ASM的normal or high 冗余度模式，大家都知道normal的数据冗余度是双份的，但是针对仲裁磁盘VTnormal的冗余模式时，磁盘的投票方式是大于一半磁盘再加一，也就是说双柜的时候需要来自三个不同的地方的磁盘，这里考虑使用NFS来做实验。

实验：

现有环境：

crs VT 磁盘组来自三个不同的目录磁盘：

集群状态：

[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2
ora.cvu
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac.db
1 ONLINE ONLINE rac1 Open
2 ONLINE ONLINE rac2 Open
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac2

对ocr文件做个备份：

[root@rac1 ~]# cd /u01/app/11.2.0/grid/bin/

[root@rac1 bin]# ./ocrconfig -manualbackup

rac1 2018/02/09 12:48:05 /u01/app/11.2.0/grid/cdata/cluster/backup_20180209_124805.ocr

rac1 2018/02/09 11:35:18 /u01/app/11.2.0/grid/cdata/cluster/backup_20180209_113518.ocr

rac1 2018/01/30 16:51:39 /u01/app/11.2.0/grid/cdata/cluster/backup_20180130_165139.ocr
[root@rac1 bin]# ./ocrconfig -showbackup
PROT-24: Auto backups for the Oracle Cluster Registry are not available

rac1 2018/02/09 12:48:05 /u01/app/11.2.0/grid/cdata/cluster/backup_20180209_124805.ocr

rac1 2018/02/09 11:35:18 /u01/app/11.2.0/grid/cdata/cluster/backup_20180209_113518.ocr

rac1 2018/01/30 16:51:39 /u01/app/11.2.0/grid/cdata/cluster/backup_20180130_165139.oc

ocr状态：

[grid@rac1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3112
Available space (kbytes) : 259008
ID : 833159346
Device/File Name : +crs
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

检验votedisk normal时需要访问的磁盘数为（n/2+1），手工断开一条链路，检查群集等状态：

[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5ef2ae0af41a4f28bf56d00afb5c0a2f (/dev/asm-diskb) [CRS]
2. ONLINE 69aedaedb7f64f3fbf627d63a94a99c6 (/dev/asm-diskc) [CRS]
Located 2 voting disk(s).
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2
ora.cvu
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac.db
1 ONLINE ONLINE rac1 Open
2 ONLINE ONLINE rac2 Open
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac2

此时只断开一条链路是不影响群集状态以及数据库正常使用的，因为crs磁盘组中的三个共享磁盘是来自不同的磁盘。但是如果在只有双机柜的情况下就会出现其中一个机柜会提供两块的磁盘，当这个机柜掉电等故障时将发生灾难性的事情。我们在本环境中在添加一个nfs文件的话能否解决问题呢？

实验：

NFS配置可以参考官方文档：https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=516839158258396&id=1674859.1&displayIndex=5&_afrWindowMode=0&_adf.ctrl-state=ppj9wr225_121#aref_section218

找个第三台服务配置nfs服务：

[root@six ~]# mkdir /votedisk

[root@six ~]# vi /etc/exports
/votedisk *(rw,sync,all_squash,anonuid=1000,anongid=6000)

[root@six ~]# chkconfig --level 345 nfs on
[root@six ~]# service nfs stop
Shutting down NFS daemon: [ OK ]
Shutting down NFS mountd: [ OK ]
Shutting down NFS quotas: [ OK ]
[root@six ~]# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: [ OK ]
Stopping RPC idmapd: [ OK ]
Starting RPC idmapd: [ OK ]
Starting NFS daemon: [ OK ]
[root@six ~]# exportfs -v
/votedisk <world>(rw,wdelay,root_squash,all_squash,no_subtree_check,anonuid=1000,anongid=6000)

在rac各节点进行挂载：

[root@rac1 voting_disk]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Jan 30 09:15:55 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_rac1-lv_root / ext4 defaults 1 1
UUID=f5f8ac26-9e59-494d-8f18-af87045c88b3 /boot ext4 defaults 1 2
/dev/mapper/vg_rac1-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
192.168.1.69:/votedisk /voting_disk nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 0 0

[root@rac1 voting_disk]# mount
/dev/mapper/vg_rac1-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
192.168.1.69:/votedisk on /voting_disk type nfs (rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,addr=192.168.1.69)

crs磁盘组新增nfs磁盘：

[root@rac1 ~]# dd if=/dev/zero of=/voting_disk/vote_1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 47.6771 s, 22.5 MB/s

SQL> alter diskgroup CRS add quorum failgroup FGQ DISK '/voting_disk/vote_1';
alter diskgroup CRS add quorum failgroup FGQ DISK '/voting_disk/vote_1'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15031: disk specification '/voting_disk/vote_1' matches no disks
ORA-15014: path '/voting_disk/vote_1' is not in the discovery set

SQL> show parameter asm_disk

NAME TYPE
------------------------------------ ----------------------
VALUE
------------------------------
asm_diskgroups string
DATA
asm_diskstring string
/dev/asm*
SQL> create pfile='/home/grid/asmpifle.ora' from spfile;

File created.

SQL> alter system set asm_diskstring='/dev/asm*','/voting_disk/vote*' scope=spfile;

System altered.

SQL> alter system set asm_diskstring='/dev/asm*','/voting_disk/vote*';

System altered.

SQL> alter diskgroup CRS add quorum failgroup FGQ DISK '/voting_disk/vote_1';

Diskgroup altered.

[root@rac1 voting_disk]# su - grid
[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5ef2ae0af41a4f28bf56d00afb5c0a2f (/dev/asm-diskb) [CRS]
2. ONLINE 69aedaedb7f64f3fbf627d63a94a99c6 (/dev/asm-diskc) [CRS]
3. ONLINE 303304b5b5204ff3bff8a48b0065cc79 (/voting_disk/vote_1) [CRS]
Located 3 voting disk(s).

模拟/dev/asm-diskc 磁盘断开是否影响群集、数据库运行，我们将/dev/asm-diskc 从crs组中删除测试：

SQL> alter diskgroup crs drop disk 'CRS_0001' rebalance power 4;

Diskgroup altered.

[root@rac1 voting_disk]# su - grid
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.cvu
1 ONLINE ONLINE rac2
ora.oc4j
1 ONLINE ONLINE rac2
ora.rac.db
1 ONLINE ONLINE rac1 Open
2 ONLINE ONLINE rac2 Open
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac1
[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5ef2ae0af41a4f28bf56d00afb5c0a2f (/dev/asm-diskb) [CRS]
2. ONLINE 69aedaedb7f64f3fbf627d63a94a99c6 (/dev/asm-diskc) [CRS]
3. ONLINE 303304b5b5204ff3bff8a48b0065cc79 (/voting_disk/vote_1) [CRS]
Located 3 voting disk(s).