上一篇文章（修复ASM磁盘头（一）：http://blog.itpub.net/23135684/viewspace-1260502/）讨论了通过ASMCMD中的md_backup，md_restore命令是否能够修复ASM磁盘头，答案是否定的。这篇文章讨论通过Grid下的kfed工具对磁盘头进行修复。
一.检查实验环境。
# pwd
/u01/app/11.2.0/grid/bin
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA01.dg
ONLINE ONLINE localhost
ora.FRA01.dg
ONLINE ONLINE localhost
ora.LISTENER.lsnr
ONLINE ONLINE localhost
ora.asm
ONLINE ONLINE localhost Started
ora.ons
OFFLINE OFFLINE localhost
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE localhost
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE localhost
ora.orcl.db
1 ONLINE ONLINE localhost Open
相关服务都是ONLINE状态。
# lspv
hdisk0 00cc1ad4ef095bf0 rootvg active
hdisk1 00cc1ad46aff307f None
hdisk2 none None
hdisk3 none None
hdisk4 none None
# lsdev -Cc disk
hdisk0 Available 00-08-00 SAS Disk Drive
hdisk1 Available 00-08-00 SAS Disk Drive
hdisk2 Available 04-00-01 MPIO DS4700 Disk
hdisk3 Available 04-00-01 MPIO DS4700 Disk
hdisk4 Available 04-00-01 MPIO DS4700 Disk
服务器一共有5张磁盘，其中hdisk2，hdisk3和hdisk4是存储盘。
# su - grid
$ asmcmd -p
ASMCMD [+] > lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 51200 49637 0 49637 0 N DATA01/
MOUNTED EXTERN N 512 4096 1048576 51200 50980 0 50980 0 N FRA01/
ASMCMD [+] > lsdsk -G data01
Path
/dev/rhdisk3
ASMCMD [+] > lsdsk -G fra01
Path
/dev/rhdisk4
hdisk3属于DATA01磁盘组，hdisk4属于FRA01磁盘组。
# su - oracle
$ sql
SQL*Plus: Release 11.2.0.3.0 Production on Thu Sep 18 14:28:59 2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
SQL> set linesize 200
SQL> col name format a50
SQL> select name ,status from v$datafile;
NAME STATUS
-------------------------------------------------- -------
+DATA01/orcl/datafile/system.256.858607953 SYSTEM
+DATA01/orcl/datafile/sysaux.257.858607953 ONLINE
+DATA01/orcl/datafile/undotbs1.258.858607953 ONLINE
+DATA01/orcl/datafile/users.259.858607953 ONLINE
SQL> select name,open_mode from v$database;
NAME OPEN_MODE
-------------------------------------------------- --------------------
ORCL READ WRITE
SQL> select status from v$instance;
STATUS
------------
OPEN
数据库状态是OPEN的。
二.为ASM磁盘分配PVID。
1).停止所有CRS服务：
# pwd
/u01/app/11.2.0/grid/bin
# ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'localhost'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'localhost'
CRS-2677: Stop of 'ora.orcl.db' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.DATA01.dg' on 'localhost'
CRS-2673: Attempting to stop 'ora.FRA01.dg' on 'localhost'
CRS-2677: Stop of 'ora.DATA01.dg' on 'localhost' succeeded
CRS-2677: Stop of 'ora.FRA01.dg' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'localhost'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'localhost' succeeded
CRS-2677: Stop of 'ora.asm' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'localhost'
CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'localhost'
CRS-2677: Stop of 'ora.evmd' on 'localhost' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed
CRS-4133: Oracle High Availability Services has been stopped.
# ps -ef | grep d.bin
root 6815998 3211342 0 14:30:33 pts/0 0:00 grep d.bin
# ps -ef | grep ora_
root 3670204 3211342 0 14:30:36 pts/0 0:00 grep ora_
2).为所有ASM磁盘分配PVID：
# lspv
hdisk0 00cc1ad4ef095bf0 rootvg active
hdisk1 00cc1ad46aff307f None
hdisk2 none None
hdisk3 none None
hdisk4 none None
# chdev -l hdisk2 -a pv=yes
hdisk2 changed
# chdev -l hdisk3 -a pv=yes
hdisk3 changed
# chdev -l hdisk4 -a pv=yes
hdisk4 changed
# lspv
hdisk0 00cc1ad4ef095bf0 rootvg active
hdisk1 00cc1ad46aff307f None
hdisk2 00cc1ad48a3f2543 None
hdisk3 00cc1ad48a3f5a17 None
hdisk4 00cc1ad48a43b989 None
3).重新启动所有CRS服务：
# pwd
/u01/app/11.2.0/grid/bin
# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
经过一段时间后，ASM实例能正常的启动，但是所有的磁盘组都无法ONLINE：
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA01.dg
ONLINE OFFLINE localhost
ora.FRA01.dg
ONLINE OFFLINE localhost
ora.LISTENER.lsnr
ONLINE ONLINE localhost
ora.asm
ONLINE ONLINE localhost Started
ora.ons
OFFLINE OFFLINE localhost
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE localhost
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE localhost
ora.orcl.db
1 ONLINE OFFLINE Instance Shutdown
4).查看ASM实例告警日志：
# su - grid
$ cd $ORACLE_HOME/log/diag/asm/+asm/+ASM/trace
$ ls *.log
alert_+ASM.log
$ tail -200 alert*.log
......
ERROR: diskgroup DATA01 was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA01" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA01"
ERROR: ALTER DISKGROUP DATA01 MOUNT /* asm agent *//* {0:0:2} */
NOTE: cache registered group FRA01 number=1 incarn=0x408082f3
NOTE: cache began mount (first) of group FRA01 number=1 incarn=0x408082f3
Thu Sep 18 14:37:41 2014
NOTE: No asm libraries found in the system
ASM Health Checker found 1 new failures
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: cache dismounting (clean) group 1/0x408082F3 (FRA01)
NOTE: messaging CKPT to quiesce pins Unix process pid: 13238424, image: oracle@localhost (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x408082F3 (FRA01)
NOTE: cache ending mount (fail) of group FRA01 number=1 incarn=0x408082f3
NOTE: cache deleting context for group FRA01 1/0x408082f3
GMON dismounting group 1 at 4 for pid 18, osid 13238424
ERROR: diskgroup FRA01 was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "FRA01" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "FRA01"
ERROR: ALTER DISKGROUP FRA01 MOUNT /* asm agent *//* {0:0:2} */
......
ASM Health Checker found 1 new failures
ASM Health Checker found 1 new failures
hdisk3和hdisk4 ASM磁盘头已经被破坏。
三.使用kfed工具修复ASM磁盘头。
ASM disk header是ASM disk的第一个块，即AU#0 BLOCK#0，disk header中记录了ASM disk的重要信息，比如ASM disk名字，diskgroup名字，failure group名字，AU size等等。
1).使用kfed工具查看破坏后磁盘头内容：
# su - grid
$ which kfed
/u01/app/11.2.0/grid/bin/kfed
$ kfed read /dev/rhdisk3
kfbh.endian: 201 ; 0x000: 0xc9
kfbh.hard: 194 ; 0x001: 0xc2
kfbh.type: 212 ; 0x002: *** Unknown Enum *** <<<< 表示是否是磁盘头信息，这里的状态是被破坏的状态。
kfbh.datfmt: 193 ; 0x003: 0xc1
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
110285600 C9C2D4C1 00000000 00000000 00000000 [................]
110285610 00000000 00000000 00000000 00000000 [................]
Repeat 6 times
110285680 00CC1AD4 8A3F5A17 00000000 00000000 [.....?Z.........]
110285690 00000000 00000000 00000000 00000000 [................]
Repeat 246 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][212]
$ kfed read /dev/rhdisk4
kfbh.endian: 201 ; 0x000: 0xc9
kfbh.hard: 194 ; 0x001: 0xc2
kfbh.type: 212 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 193 ; 0x003: 0xc1
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
110285600 C9C2D4C1 00000000 00000000 00000000 [................]
110285610 00000000 00000000 00000000 00000000 [................]
Repeat 6 times
110285680 00CC1AD4 8A43B989 00000000 00000000 [.....C..........]
110285690 00000000 00000000 00000000 00000000 [................]
Repeat 246 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][212]
hdisk3和hdisk4磁盘头信息的条目都为0，且有明显的报错，说明磁盘头已经被破坏。
2).查看磁盘头备份信息：
在11.1.0.7之前ASM disk header没有自动备份，需要手动执行kfed read或者dd来备份，从11.1.0.7开始ASM disk header会自动备份到AU#1的倒数第二个block。对于AU size是1MB的DISKGROUP，每个AU包括block数量=1024KB/4KB=256个，因此备份信息位于AU#1的第254号block，执行下面的命令查看ASM disk header备份信息，备份信息与ASM disk header信息完全一致：
$ kfed read /dev/rhdisk3 AUNUM=1 BLKNUM=254 ausz=1048576 | more
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD <<<< 表示是磁盘头信息。
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 254 ; 0x004: blk=254
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 3830034039 ; 0x00c: 0xe449ae77
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA01_0000 ; 0x028: length=11 <<<< disk名称
kfdhdb.grpname: DATA01 ; 0x048: length=6 <<<< ASM磁盘组名称呢个
kfdhdb.fgname: DATA01_0000 ; 0x068: length=11 <<<< Failgroup名称
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 33007182 ; 0x0a8: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.crestmp.lo: 417121280 ; 0x0ac: USEC=0x0 MSEC=0x331 SECS=0xd MINS=0x6
kfdhdb.mntstmp.hi: 33007182 ; 0x0b0: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.mntstmp.lo: 427397120 ; 0x0b4: USEC=0x0 MSEC=0x264 SECS=0x17 MINS=0x6
kfdhdb.secsize: 512 ; 0x0b8: 0x0200 <<<< 扇区 size= 512 byte
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 <<<< 块大小 size = 4KB
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000 <<<< AU size =1 MB
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 51200 ; 0x0c4: 0x0000c800
kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000
kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi: 33007182 ; 0x0e4: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
......
$ kfed read /dev/rhdisk4 AUNUM=1 BLKNUM=254 ausz=1048576 | more
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD <<<< 表示是磁盘头信息。
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 254 ; 0x004: blk=254
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 3839790854 ; 0x00c: 0xe4de8f06
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: FRA01_0000 ; 0x028: length=10
kfdhdb.grpname: FRA01 ; 0x048: length=5
kfdhdb.fgname: FRA01_0000 ; 0x068: length=10
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 33007182 ; 0x0a8: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.crestmp.lo: 438000640 ; 0x0ac: USEC=0x0 MSEC=0x2d7 SECS=0x21 MINS=0x6
kfdhdb.mntstmp.hi: 33007182 ; 0x0b0: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.mntstmp.lo: 449592320 ; 0x0b4: USEC=0x0 MSEC=0x30f SECS=0x2c MINS=0x6
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 51200 ; 0x0c4: 0x0000c800
kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000
kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi: 33007182 ; 0x0e4: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
3).使用kfed repair命令修复ASM磁盘头：
恢复ASM disk header的方法是执行kfed repair命令，利用备份信息恢复ASM disk header：
$ kfed repair /dev/rhdisk3 aus=1048576
$ kfed repair /dev/rhdisk4 aus=1048576
4).查看恢复后的ASM磁盘头信息：
$ kfed read /dev/rhdisk3
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD <<<< 表示是磁盘头信息。
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 3830034057 ; 0x00c: 0xe449ae89
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA01_0000 ; 0x028: length=11 <<<< disk名称
kfdhdb.grpname: DATA01 ; 0x048: length=6 <<<< group名称
kfdhdb.fgname: DATA01_0000 ; 0x068: length=11 <<<< failure group名称
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 33007182 ; 0x0a8: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.crestmp.lo: 417121280 ; 0x0ac: USEC=0x0 MSEC=0x331 SECS=0xd MINS=0x6
kfdhdb.mntstmp.hi: 33007182 ; 0x0b0: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.mntstmp.lo: 427397120 ; 0x0b4: USEC=0x0 MSEC=0x264 SECS=0x17 MINS=0x6
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 <<<< ASM block size = 4KB
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000 <<<< ASM AU size =1 MB
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 51200 ; 0x0c4: 0x0000c800
kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000
kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi: 33007182 ; 0x0e4: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.grpstmp.lo: 416513024 ; 0x0e8: USEC=0x0 MSEC=0xdf SECS=0xd MINS=0x6
......
$ kfed read /dev/rhdisk4
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 3839791096 ; 0x00c: 0xe4de8ff8
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: FRA01_0000 ; 0x028: length=10
kfdhdb.grpname: FRA01 ; 0x048: length=5
kfdhdb.fgname: FRA01_0000 ; 0x068: length=10
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 33007182 ; 0x0a8: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.crestmp.lo: 438000640 ; 0x0ac: USEC=0x0 MSEC=0x2d7 SECS=0x21 MINS=0x6
kfdhdb.mntstmp.hi: 33007182 ; 0x0b0: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.mntstmp.lo: 449592320 ; 0x0b4: USEC=0x0 MSEC=0x30f SECS=0x2c MINS=0x6
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 51200 ; 0x0c4: 0x0000c800
kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000
kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi: 33007182 ; 0x0e4: HOUR=0xe DAYS=0x12 MNTH=0x9 YEAR=0x7de
kfdhdb.grpstmp.lo: 437550080 ; 0x0e8: USEC=0x0 MSEC=0x11f SECS=0x21 MINS=0x6
......
可以看到，恢复后的ASM磁盘头可以看到ASM磁盘组相关的信息，注意和损坏的进行对比，损坏的ASM磁盘头看不到这些信息，在最后有明显的报错。
四.重新MOUNT磁盘组，打开数据库。
1).检查当前CRS服务状态：
/u01/app/11.2.0/grid/bin
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA01.dg
ONLINE OFFLINE localhost
ora.FRA01.dg
ONLINE OFFLINE localhost
ora.LISTENER.lsnr
ONLINE ONLINE localhost
ora.asm
ONLINE ONLINE localhost Started
ora.ons
OFFLINE OFFLINE localhost
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE localhost
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE localhost
ora.orcl.db
1 ONLINE OFFLINE Instance Shutdown
2).加载ASM磁盘组：
# su - grid
$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Thu Sep 18 14:53:19 2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option
SQL> alter diskgroup data01 mount;
Diskgroup altered.
SQL> alter diskgroup fra01 mount;
Diskgroup altered.
3).检查CRS服务状态：
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA01.dg
ONLINE ONLINE localhost
ora.FRA01.dg
ONLINE ONLINE localhost
ora.LISTENER.lsnr
ONLINE ONLINE localhost
ora.asm
ONLINE ONLINE localhost Started
ora.ons
OFFLINE OFFLINE localhost
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE localhost
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE localhost
ora.orcl.db
1 ONLINE ONLINE localhost Open
4).检查数据库状态：
# su - oracle
$ sql
SQL*Plus: Release 11.2.0.3.0 Production on Thu Sep 18 14:54:17 2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
SQL> select status from v$instance;
STATUS
------------
OPEN
SQL> select name,open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
ORCL READ WRITE
参考文章：
《ASM常见错误 ORA-15063: ASM discovered an insufficient number of disks for diskgroup》：https://blogs.oracle.com/Database4CN/entry/asm%E5%B8%B8%E8%A7%81%E9%94%99%E8%AF%AF_ora_15063_asm_discovered
相关文章：
《 了解和使用kfed维护ASM元数据 》： http://blog.itpub.net/23135684/viewspace-1079197/
注意：在使用kfed工具修复ASM磁盘头之前，最好将磁盘上的PVID清除掉。
--end--
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/23135684/viewspace-1273077/，如需转载，请注明出处，否则将追究法律责任。
转载于:http://blog.itpub.net/23135684/viewspace-1273077/
扫一扫
612
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
