存储挂掉 重建oracle rac



9.24 存储挂掉   oracle 主节点重建   oracle 版本11gr2

crs ocr 资源还在不打算重新安装节点。重建后从备端重新同步数据

SQL> select group_number,name,FAILGROUP,REDUNDANCY from v$asm_disk;

GROUP_NUMBER NAME       FAILGROUP      REDUNDA
------------ ------------------------------ ------------------------------ -------
    0           UNKNOWN
    0           UNKNOWN
    0           UNKNOWN
    3 CRSVOL1       CRSVOL1      UNKNOWN

重新创建磁盘组

SQL> create diskgroup ARCHVOL1  external redundancy disk ''/dev/oracleasm/disks/ARCHVOL1';
create diskgroup ARCHVOL1  external redundancy disk ''/dev/oracleasm/disks/ARCHVOL1'
                                                    *
ERROR at line 1:
ORA-15106: missing or invalid operating system disk locator string

SQL> create diskgroup ARCHVOL1  external redundancy disk '/dev/oracleasm/disks/ARCHVOL1';
create diskgroup ARCHVOL1  external redundancy disk '/dev/oracleasm/disks/ARCHVOL1'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/disks/ARCHVOL1' matches no disks
ORA-15014: path '/dev/oracleasm/disks/ARCHVOL1' is not in the discovery set

SQL> show parameter asm_diskstring;

NAME         TYPE  VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring        string

设置asm_diskstring
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*';
alter system set asm_diskstring='/dev/oracleasm/disks/*'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-15014: path 'ORCL:CRSVOL1' is not in the discovery set

从报错信息中看出 当前处于正常MOUNTED状态的磁盘组  

并不是/dev/oracleasm/disk

SQL>  create diskgroup ARCHVOL1  external redundancy disk 'ORCL:ARCHVOL1';                                                      
Diskgroup created.

创建成功

SQL> create diskgroup DATAVOL1 external redundancy disk 'ORCL:DATAVOL1';

Diskgroup created.

SQL> create diskgroup VOL1 external redundancy disk 'ORCL:VOL1';
create diskgroup VOL1 external redundancy disk 'ORCL:VOL1'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15033: disk 'ORCL:VOL1' belongs to diskgroup "DATADG"

创建磁盘组不对
删除 新创建的磁盘组

SQL> drop diskgroup ARCHVOL1;

Diskgroup dropped.

SQL>  drop diskgroup DATAVOL1;

Diskgroup dropped.

SQL> select group_number,name,FAILGROUP,REDUNDANCY from v$asm_disk;

GROUP_NUMBER NAME       FAILGROUP      REDUNDA
------------ ------------------------------ ------------------------------ -------
    0           UNKNOWN
    0           UNKNOWN
    0           UNKNOWN
    3 CRSVOL1       CRSVOL1      UNKNOWN

SQL>

应当创建如下两个磁盘组

ARCHDG/

DATADG/

SQL>  create diskgroup ARCHDG  external redundancy disk 'ORCL:ARCHVOL1';

Diskgroup created.

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:ARCHVOL1,ORCL:VOL1';

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:ARCHVOL1,ORCL:VOL1';
 create diskgroup DATADG external redundancy disk 'ORCL:ARCHVOL1,ORCL:VOL1'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification 'ORCL:ARCHVOL1,ORCL:VOL1' matches no disks
ORA-15186: ASMLIB error function = [kfk_dscv_lib_init:disc],  error = [4],  mesg = [Invalid
discovery string]

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1';

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1';
 create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15033: disk 'ORCL:VOL1' belongs to diskgroup "DATADG"

SQL> drop diskgroup DATADG;
drop diskgroup DATADG
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup "DATADG" does not exist or is not mounted

SQL> drop diskgroup DATADG;
drop diskgroup DATADG
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup "DATADG" does not exist or is not mounted

必须把ORCL:VOL1  盘头信息擦除

1028  /etc/init.d/oracleasm deletedisk   VOL1
 1029  /etc/init.d/oracleasm createdisk VOL1  /dev/sde1
 1030  /etc/init.d/oracleasm scandisks;
 1031  /etc/init.d/oracleasm listdisks;

SQL> create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1';

Diskgroup created.

创建成功

用原来备份的pfile 启动 

用pfile 启动
SQL> startup nomount pfile='/u01/app/oracle/product/11.2.0/dbhome_1/dbs/pfile'
ORACLE instance started.

Total System Global Area 5010685952 bytes
Fixed Size      2261848 bytes
Variable Size   2835352744 bytes
Database Buffers  2164260864 bytes
Redo Buffers      8810496 bytes
SQL>

用rman 重新同步数据
 rman target sys/*****@racdg  auxiliary sys/***@racdb

duplicate target database for standby from active database;

发现  RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/26/2015 16:11:56
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile file number=00170 name=+DATADG/racdg/datafile/sae.436.888841821
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/26/2015 16:11:58
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy

好像是空间不足

再次执行  of Duplicate Db command at 09/26/2015 19:36:30
RMAN-05501: aborting duplication of target database
RMAN-06136: ORACLE error from auxiliary database: ORA-00200: control file could not be created
ORA-00202: control file: '+datadg'
ORA-17502: ksfdcre:4 Failed to create file +datadg
ORA-15041: diskgroup "DATADG" space exhausted 

删除数据文件

修改racdb1 的  vim /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora 

修改成
   RACDG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.124)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = racdg)
      (UR=A)
    )
  )

把racdg2  上的tnsnames.ora 修改成

重新开始复制

racdb
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    307199   306632                0          306632              0             N  ARCHDG/
MOUNTED  EXTERN  N         512   4096  1048576     10239     9759                0            9759              0             Y  DATA/
MOUNTED  EXTERN  N         512   4096  1048576   2098514  2098178                0         2098178              0             N  DATADG/
ASMCMD>

racdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    307196   201520                0          201520              0             N  ARCHDG/
MOUNTED  EXTERN  N         512   4096  1048576     10236     9840                0            9840              0             Y  DATA/
MOUNTED  EXTERN  N         512   4096  1048576   3460299  1308874                0         1308874              0             N  DATADG/

发现datadg 大小相差很大

怀疑  创建的diskgroup 有问题

发现asmlib 里racdb 的 datavol 明显有问题

重新对相应的磁盘分区

发现大小正常

重新同步

racdb2
datafile 177 switched to datafile copy
input datafile copy RECID=584 STAMP=891493391 file name=+DATADG/racdb/datafile/sae.410.891492627
Finished Duplicate Db at 27-SEP-2015 05:03:35

alter system set INSTANCE_NUMBER=1 scope=spfile sid='racdb1';
alter system set INSTANCE_NUMBER=2 scope=spfile sid='racdb2';
alter system set INSTANCE_NUMBER=3 scope=spfile sid='racdb3';
alter system set INSTANCE_NUMBER=4 scope=spfile sid='racdb4';

alter system set thread=1 scope=spfile sid='racdb1';
alter system set thread=2 scope=spfile sid='racdb2';
alter system set thread=3 scope=spfile sid='racdb3';
alter system set thread=4 scope=spfile sid='racdb4';

alter system set undo_tablespace='UNDOTBS1' sid='racdb1';
alter system set undo_tablespace='UNDOTBS2' sid='racdb2';
alter system set undo_tablespace='UNDOTBS3' sid='racdb3';
alter system set undo_tablespace='UNDOTBS4' sid='racdb4';

alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb1/cdump' scope=spfile sid='racdb1';

alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb2/cdump' scope=spfile sid='racdb2';
alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb3/cdump' scope=spfile sid='racdb3';
alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb4/cdump' scope=spfile sid='racdb4';

alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb1/trace' scope=spfile sid='racdb1';
alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb2/trace' scope=spfile sid='racdb2';
alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb3/trace' scope=spfile sid='racdb3';
alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb4/trace' scope=spfile sid='racdb4';

alter system set local_listener=’(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.3.129)(PORT=11521))))’ scope=both;