redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复---惜分飞

节点2 asm dismount导致redo写报错(ORA-00340,ORA-00345)，经过分析asm和系统日志,确认是由于多路径异常导致io异常

2022-01-24T23:44:39.966602+08:00 WARNING: group 4 is being dismounted. WARNING: ASMB force dismounting group 4 (REDO) due to ASM server dismount SUCCESS: diskgroup REDO was dismounted 2022-01-24T23:44:41.103783+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc: ORA-00345: redo log write error block 11961764 count 6 ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685' 2022-01-24T23:44:41.156809+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc: ORA-00340: IO error processing online log 10 of thread 2 ORA-00345: redo log write error block 11961764 count 6 ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685' Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc  (incident=1341402): ORA-340 [] [] [] [] [] [] [] [] [] [] [] [] Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF2/incident/incdir_1341402/XFF2_lgwr_228507_i1341402.trc 2022-01-24T23:44:41.505251+08:00 USER (ospid: 133928): terminating the instance due to error 340

由于节点2是突然crash，节点1做实例恢复失败，由于节点2的redo发生了写丢失,导致节点1实例恢复后库crash,进而是的该集群的相关数据库节点全部crash

2022-01-24T23:46:08.440519+08:00 Slave encountered ORA-10388 exception during crash recovery 2022-01-24T23:46:08.442854+08:00 Slave encountered ORA-10388 exception during crash recovery Abort recovery for domain 0, flags 4 2022-01-24T23:46:08.444531+08:00 Aborting crash recovery due to error 742 2022-01-24T23:46:08.444695+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_450481.trc: ORA-00742: Log read detects lost write in thread 2 sequence 63507 block 11941482 ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685' Abort recovery for domain 0, flags 4 2022-01-24T23:46:08.771108+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_450481.trc: ORA-00742: Log read detects lost write inthread 2 sequence 63507 block 11941482 ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685' ORA-742 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:17:165} */... 2022-01-24T23:46:10.143155+08:00 License high water mark = 33 2022-01-24T23:46:10.143752+08:00 USER (ospid: 451049): terminating the instance 2022-01-24T23:46:11.167337+08:00 Instance terminated by USER, pid = 451049

经过第三方强制拉库之后,数据库报ORA-600 kcbzib_kcrsds_1

2022-01-25T10:13:37.922332+08:00 Completed crash recovery at  Thread 2: RBA 5.3.16, nab 3, scn 0x00000a348a032122  0 data blocks read, 0 data blocks written, 0 redo k-bytes read 2022-01-25T10:13:38.071326+08:00 Thread 2 advanced to log sequence 6 (thread recovery) validate pdb 0, flags x4, valid 0, pdb flags x204 * validated domain 0, flags = 0x200 CRASH recovery complete: pdb 0 valid 1 (flags x4, pdb flags x200) Picked broadcast on commit scheme to generate SCNs Endian type of dictionary set to little 2022-01-25T10:13:38.389741+08:00 TT00: Gap Manager starting (PID:220505) 2022-01-25T10:13:38.646484+08:00 Thread 1 opened at log sequence 1   Current log# 1 seq# 1 mem# 0: +DATA/XFF/ONLINELOG/group_1.536.1094875107 Successful open of redo thread 1 2022-01-25T10:13:38.647243+08:00 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc  (incident=1879556): ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF1/incident/incdir_1879556/XFF1_ora_216590_i1879556.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. ***************************************************************** An internal routine has requested a dump of selected redo. This usually happens following a specific internal error, when analysis of the redo logs will help Oracle Support with the diagnosis. It is recommended that you retain all the redo logs generated (by all the instances) during the past 12 hours, in case additional redo dumps are required to help with the diagnosis. ***************************************************************** 2022-01-25T10:13:39.734366+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc: ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] 2022-01-25T10:13:39.734424+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] 2022-01-25T10:13:39.734499+08:00 Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] 2022-01-25T10:13:39.734536+08:00 Error 704 happened during db open, shutting down database Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc  (incident=1879557): ORA-00603: ORACLE server session terminated by fatal error ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF1/incident/incdir_1879557/XFF1_ora_216590_i1879557.trc 2022-01-25T10:13:39.894464+08:00 2022-01-25T10:13:40.446888+08:00 opiodr aborting process unknown ospid (216590) as a result of ORA-603 2022-01-25T10:13:40.470643+08:00 ORA-603 : opitsk aborting process License high water mark = 36 2022-01-25T10:13:40.471453+08:00 USER (ospid: 216590): terminating the instance due to error 704 2022-01-25T10:13:41.436133+08:00 opiodr aborting process unknown ospid (189796) as a result of ORA-1092 2022-01-25T10:13:41.439011+08:00 ORA-1092 : opitsk aborting process 2022-01-25T10:13:41.472060+08:00 PMON (ospid: 189585): terminating the instance due to error 704

该错误是12c之后才有的报错,由于文件异常导致,通过以前的解决经验,接手这个问题之后快速调整数据库文件头信息,顺利open库
参考以前相关blog内容：
Oracle 12c redo 丢失恢复
模拟19c数据库redo异常恢复
ORA-600 kcbzib_kcrsds_1报错
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

[oracle@xifenfei02 ~]$ sqlplus / as sysdba SQL*Plus: Release 12.2.0.1.0 Production on Wed Jan 26 00:31:50 2022 Copyright (c) 1982, 2016, Oracle.  All rights reserved. Connected to an idle instance. SQL> startup mount pfile='/tmp/pfile' ORACLE instance started. Total System Global Area 3.2320E+11 bytes Fixed Size                 29879248 bytes Variable Size            4.5634E+10 bytes Database Buffers         1.9059E+11 bytes Redo Buffers             1043861504 bytes In-Memory Area           8.5899E+10 bytes Database mounted. SQL> alter database open ; Database altered.