Oracle RAC集群增加新共享硬盘并使用AFD的式增加新ASM磁盘组
Oracle RAC集群增加新共享硬盘并使用AFD的式增加新ASM磁盘组
此实验是在ASM共享磁盘的RAC环境下增加新共享硬盘并新建ASM磁盘组创建相应的表空间和表数据进行测试过程。参考资料包括了https://www.oracle.com/cn/technical-resources/article/database/asmfd.html Oracle ASM Filter Driver相关资料。
具体实验记录如下:
实验环境背景
数据库集群版本
[grid@rac2 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [19.0.0.0.0]
ASM磁盘开启的文件
ASMCMD> lsof
DB_Name Instance_Name Path
+ASM +ASM2 +ORS.255.1038877513
racdb racdb2 +DATA/RACDB/86B637B62FE07A65E053F706E80A27CA/DATAFILE/sysaux.266.1038939581
racdb racdb2 +DATA/RACDB/86B637B62FE07A65E053F706E80A27CA/DATAFILE/system.265.1038939581
racdb racdb2 +DATA/RACDB/86B637B62FE07A65E053F706E80A27CA/DATAFILE/undotbs1.267.1038939581
racdb racdb2 +DATA/RACDB/A4582AEF3E8210F9E0533C23100A3339/TEMPFILE/temp.268.1038939611
racdb racdb2 +DATA/RACDB/CONTROLFILE/current.261.1038938883
racdb racdb2 +DATA/RACDB/DATAFILE/sysaux.258.1038938777
racdb racdb2 +DATA/RACDB/DATAFILE/system.257.1038938723
racdb racdb2 +DATA/RACDB/DATAFILE/undotbs1.259.1038938803
racdb racdb2 +DATA/RACDB/DATAFILE/undotbs2.269.1038940101
racdb racdb2 +DATA/RACDB/DATAFILE/users.260.1038938803
racdb racdb2 +DATA/RACDB/ONLINELOG/group_1.262.1038938887
racdb racdb2 +DATA/RACDB/ONLINELOG/group_2.263.1038938887
racdb racdb2 +DATA/RACDB/ONLINELOG/group_3.270.1038940635
racdb racdb2 +DATA/RACDB/ONLINELOG/group_4.271.1038940637
racdb racdb2 +DATA/RACDB/TEMPFILE/temp.264.1038938901
ASM客户端信息
ASMCMD> lsct
DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group
+ASM CONNECTED 19.0.0.0.0 19.0.0.0.0 +ASM2 ORS
+ASM CONNECTED 19.0.0.0.0 19.0.0.0.0 +ASM2 DATA
_OCR CONNECTED - - rac2 ORS
racdb CONNECTED 19.0.0.0.0 19.0.0.0.0 racdb2 DATA
实验步骤
-
新增共享硬盘
本次新增硬盘到添加ASM新的磁盘的整个过程和创建搭建数据库时的ASM磁盘组类似,最后会有尝试使用ASMFD的新功能
(这是一个可以取代 ASMLIB 和 udev 设置的新功能)
直接绑定新硬盘然后加入数据库来使用。节点1:新增硬盘9,容量为50G 节点2:使用节点1的硬盘9,容量为50G -
使用udev配置Asm磁盘(两节点都做)
虚拟机在要之前配置共享存储时在各自的虚拟机安装路径中的vmx文件中添加disk.enableUUID = "TRUE"才会显示UUID。udev是一种工具,它能够根据系统中的硬件设备的状态动态更新设备文件,包括设备文件的创建,删除等。设备文件通常放在/dev目录下。使用udev后,在/dev目录下就只包含系统中真正存在的设备。
[root@rac1 ~]# fdisk -l //查看新增硬盘 Disk /dev/sdi: 53.7 GB, 53687091200 bytes, 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes //按步骤增加磁盘的分区1 root@rac1 ~]# fdisk /dev/sdi Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x9a414875. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): First sector (2048-104857599, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599): Using default value 104857599 Partition 1 of type Linux and of size 50 GiB is set Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
两节点分别fdisk -l ,显示都一致
Disk /dev/sdi: 53.7 GB, 53687091200 bytes, 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x9a414875 Device Boot Start End Blocks Id System /dev/sdi1 2048 104857599 52427776 83 Linux
但是节点2没有sdi1 且没有UUID显示
[root@rac2 ~]# partprobe /dev/sd sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1 sdf sdf1 sdg sdg1 sdh sdh1 sdi [root@rac2 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi1 [root@rac2 ~]# [root@rac2 ~]# partprobe /dev/sdi //通知内核sdi盘变化 [root@rac2 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi1 //可以出现UUID代表正常 36000c295d9c2913ec150e329c3465337
新增物理盘为ASM磁盘组成员,节点1:
[root@rac1 ~]# vi sdi1.sh #!/bin/bash for i in i ; do echo "KERNEL==\"sd?1\", SUBSYSTEM==\"block\", PROGRAM==\"/usr/lib/udev/scsi_id -g -u -d /dev/\$parent\", RESULT==\"`/usr/lib/udev/scsi_id -g -u -d /dev/sd\$i`\", SYMLINK+=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" done [root@rac1 ~]# ./sdi1.sh KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36000c295d9c2913ec150e329c3465337", SYMLINK+="asm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660" [root@rac1 ~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules //新增上述结果 [root@rac1 ~]# udevadm control --reload [root@rac1 ~]# udevadm trigger [root@rac1 ~]# ls -l /dev/sd?1 brw-rw---- 1 root disk 8, 1 May 13 11:37 /dev/sda1 brw-rw---- 1 grid asmadmin 8, 17 May 13 11:37 /dev/sdb1 brw-rw---- 1 grid asmadmin 8, 33 May 13 11:37 /dev/sdc1 brw-rw---- 1 grid asmadmin 8, 49 May 13 11:37 /dev/sdd1 brw-rw---- 1 grid asmadmin 8, 65 May 13 11:37 /dev/sde1 brw-rw---- 1 grid asmadmin 8, 81 May 13 11:37 /dev/sdf1 brw-rw---- 1 grid asmadmin 8, 97 May 13 11:37 /dev/sdg1 brw-rw---- 1 root disk 8, 113 May 13 11:37 /dev/sdh1 brw-rw---- 1 grid asmadmin 8, 129 May 13 11:37 /dev/sdi1
节点2:
[root@rac2 ~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules //增加上述结果 [root@rac2 ~]# udevadm control --reload [root@rac2 ~]# udevadm trigger [root@rac2 ~]# ls -l /dev/sd?1 brw-rw---- 1 root disk 8, 1 May 13 11:51 /dev/sda1 brw-rw---- 1 grid asmadmin 8, 17 May 13 11:51 /dev/sdb1 brw-rw---- 1 grid asmadmin 8, 33 May 13 11:51 /dev/sdc1 brw-rw---- 1 grid asmadmin 8, 49 May 13 11:51 /dev/sdd1 brw-rw---- 1 grid asmadmin 8, 65 May 13 11:51 /dev/sde1 brw-rw---- 1 grid asmadmin 8, 81 May 13 11:51 /dev/sdf1 brw-rw---- 1 grid asmadmin 8, 97 May 13 11:51 /dev/sdg1 brw-rw---- 1 root disk 8, 113 May 13 11:51 /dev/sdh1 brw-rw---- 1 grid asmadmin 8, 129 May 13 11:51 /dev/sdi1
-
查看现有ASM磁盘组的情况
[grid@rac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 19.0.0.0.0 - Production on Wed May 13 13:00:57 2020 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> show parameter asm_diskstring; //查看磁盘组支持的模式 NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskstring string /dev/asm*, AFD:* SQL> set lin 200 SQL> col path for a40 SQL> select group_number,disk_number,total_mb,name,failgroup,path from v$asm_disk order by 1; GROUP_NUMBER DISK_NUMBER TOTAL_MB NAME FAILGROUP PATH ------------ ----------- ---------- ------------------------------ ------------------------------ ---------------------------------------- 0 0 0 /dev/asm-diskf 0 1 0 /dev/asm-diskg 0 2 0 /dev/asm-diskc 0 3 0 /dev/asm-diske 0 4 0 /dev/asm-diskd 0 5 0 /dev/asm-diskb 0 6 0 /dev/asm-diski 1 0 20476 DATA1 DATA1 AFD:DATA1 1 1 20476 DATA2 DATA2 AFD:DATA2 2 0 20476 FRA1 FRA1 AFD:FRA1 3 0 1020 ORS1 ORS1 AFD:ORS1 GROUP_NUMBER DISK_NUMBER TOTAL_MB NAME FAILGROUP PATH ------------ ----------- ---------- ------------------------------ ------------------------------ ---------------------------------------- 3 1 1020 ORS2 ORS2 AFD:ORS2 3 2 1020 ORS3 ORS3 AFD:ORS3 13 rows selected.
可以从以上内容看到支持的有AFD模式的磁盘组并且所有的资源和数据都是写在此类磁盘组的上面,上述的磁盘组都是通过grid集群图形化界面或ASMCA磁盘管理软件界面创建的,创建后默认的模式是就是AFD。之前手动设置的udev rules文件通过查资料其实都可以不用做,以下方式使用shell命令行的形式新建AFD模式的磁盘组。
-
直接路径的方式创建ASM磁盘组
先直接使用/dev/asm-diski直接路径的方式创建
SQL> create diskgroup bak_test external redundancy disk '/dev/asm-diski' name baktest attribute 'au_size'='4M','compatible.asm' = '19.0.0.0', 'compatible.rdbms' = '19.0.0.0'; Diskgroup created. //查看新创建的磁盘组 SQL> select group_number,disk_number,total_mb,name,failgroup,path from v$asm_disk order by 1; GROUP_NUMBER DISK_NUMBER TOTAL_MB NAME FAILGROUP PATH ------------ ----------- ---------- ------------------------------ ------------------------------ ---------------------------------------- 0 0 0 /dev/asm-diskf 0 1 0 /dev/asm-diskg 0 2 0 /dev/asm-diskc 0 3 0 /dev/asm-diske 0 4 0 /dev/asm-diskd 0 5 0 /dev/asm-diskb 1 0 20476 DATA1 DATA1 AFD:DATA1 1 1 20476 DATA2 DATA2 AFD:DATA2 2 0 20476 FRA1 FRA1 AFD:FRA1 3 0 1020 ORS1 ORS1 AFD:ORS1 3 1 1020 ORS2 ORS2 AFD:ORS2 GROUP_NUMBER DISK_NUMBER TOTAL_MB NAME FAILGROUP PATH ------------ ----------- ---------- ------------------------------ ------------------------------ ---------------------------------------- 3 2 1020 ORS3 ORS3 AFD:ORS3 4 0 51196 BAKTEST BAKTEST /dev/asm-diski //这里可以看到模式是直接路径的方式 13 rows selected.
上述最后一行的信息可以看到新增磁盘组的路径是直接路径的方式,此方式和默认的方式都不同,应修改为AFD的模式。修改模式如下:
SQL> select name,state from v$asm_diskgroup; //查看现有磁盘组STATE信息 NAME STATE ------------------------------ ----------- DATA MOUNTED FRA MOUNTED ORS MOUNTED BAK_TEST MOUNTED SQL> select name,path from v$asm_disk_stat; //查案磁盘组路径信息 NAME PATH ------------------------------ ---------------------------------------- BAKTEST /dev/asm-diski ORS1 AFD:ORS1 ORS2 AFD:ORS2 ORS3 AFD:ORS3 DATA1 AFD:DATA1 DATA2 AFD:DATA2 FRA1 AFD:FRA1 7 rows selected. SQL> alter diskgroup BAK_TEST dismount; //dismount磁盘组 Diskgroup altered. SQL> select name,state from v$asm_diskgroup; NAME STATE ------------------------------ ----------- BAK_TEST DISMOUNTED DATA MOUNTED FRA MOUNTED ORS MOUNTED SQL> drop diskgroup BAKTEST; //删除磁盘组失败报错 drop diskgroup BAKTEST * ERROR at line 1: ORA-15039: diskgroup not dropped ORA-15073: diskgroup BAKTEST is mounted by another ASM instance 切换至节点2: [grid@rac2 ~]$ asmcmd ASMCMD> umount BAKTEST ASMCMD> lsdg SQL> drop diskgroup BAKTEST; //删除成功 Diskgroup dropped.
-
AFD方式创建新磁盘组
ASMCMD> afd_state //查看AFD状态 ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1' ASMCMD> afd_label baktest01 /dev/asm-diski ASMCMD> afd_lsdsk -------------------------------------------------------------------------------- Label Filtering Path ================================================================================ BAKTEST01 ENABLED /dev/asm-diski DATA1 ENABLED /dev/asm-diske DATA2 ENABLED /dev/asm-diskf FRA1 ENABLED /dev/asm-diskg ORS1 ENABLED /dev/asm-diskb ORS2 ENABLED /dev/asm-diskc ORS3 ENABLED /dev/asm-diskd SQL> create diskgroup bak_afdtest external redundancy disk 'AFD:BAKTEST01' //在AFD模式上创建磁盘组 SQL> alter diskgroup BAK_AFDTEST mount; //两节点分别mount SQL> CREATE tablespace AFD_test datafile '+BAK_AFDTEST' SIZE 100M; SQL> CREATE TABLE t_afd (n NUMBER) tablespace AFD_test; SQL> INSERT INTO t_afd VALUES(1); SQL> commit; Commit complete. SQL> select * from t_afd; //查询信息正常 N ---------- 1
后续
后续会继续整理相关AFD相关的知识,包括直接创建AFD硬盘不生成ASM磁盘组。