ORA-01558: out of transaction ID‘s in rollback segment SYSTEM---惜分飞

客户一个11.2.0.1的库,在重启之前报ORA-00604和ORA-01558: out of transaction ID’s in rollback segment SYSTEM错误

Sat Mar 16 05:51:44 2024 Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_smon_47709.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-01558: out of transaction ID's in rollback segment SYSTEM Sat Mar 16 05:51:55 2024 Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_smon_47709.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-01558: out of transaction ID's in rollback segment SYSTEM

数据库关闭之后无法open,报ORA-01092 ORA-00604 ORA-01558错误

SQL> alter database open; alter database open * ERROR at line 1: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00604: error occurred at recursive SQL level 1 ORA-01558: out of transaction ID's in rollback segment SYSTEM Process ID: 44308 Session ID: 769 Serial number: 3

对应的alert日志信息

Fri Aug 30 20:05:40 2024 alter database open Beginning crash recovery of 1 threads Fri Aug 30 20:06:08 2024  parallel recovery started with 32 processes Started redo scan Completed redo scan  read 0 KB redo, 0 data blocks need recovery Started redo application at  Thread 1: logseq 7583120, block 130906, scn 832944617073 Recovery of Online Redo Log: Thread 1 Group 6 Seq 7583120 Reading mem 0   Mem# 0: /OracleData/db_mc/xff/redo06.log Completed redo application of 0.00MB Completed crash recovery at  Thread 1: logseq 7583120, block 130906, scn 832944637074  0 data blocks read, 0 data blocks written, 0 redo k-bytes read Fri Aug 30 20:06:09 2024 Thread 1 advanced to log sequence 7583121 (thread open) Thread 1 opened at log sequence 7583121   Current log# 4 seq# 7583121 mem# 0: /OracleData/db_mc/xff/redo04.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Aug 30 20:06:09 2024 SMON: enabling cache recovery Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_ora_6066.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-01558: out of transaction ID's in rollback segment SYSTEM Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_ora_6066.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-01558: out of transaction ID's in rollback segment SYSTEM Error 604 happened during db open, shutting down database USER (ospid: 6066): terminating the instance due to error 604 Instance terminated by USER, pid = 6066 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (6066) as a result of ORA-1092

通过对异常block进行dump

index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num ------------------------------------------------------------------------------------------------  0x00    9    0x00  0xfffffffe  0x0059  0x00c1.ee6ab58f  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x01    9    0x00  0xfffffffe  0x0008  0x00c1.ee6ab5a7  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x02    9    0x00  0xfffffffe  0x005b  0x00c1.ee6ab59f  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x03    9    0x00  0xfffffffe  0x0007  0x00c1.ee6ab5b5  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x04    9    0x00  0xfffffffe  0x005f  0x00c1.ee6ab593  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x05    9    0x00  0xfffffffe  0x000a  0x00c1.ee6ab5ad  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x06    9    0x00  0xfffffffe  0x0019  0x00c1.ee6ab5dd  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x07    9    0x00  0xfffffffe  0x0013  0x00c1.ee6ab5b7  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x08    9    0x00  0xfffffffe  0x0035  0x00c1.ee6ab5a9  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x09    9    0x00  0xfffffffe  0x0002  0x00c1.ee6ab59d  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x0a    9    0x00  0xfffffffe  0x003c  0x00c1.ee6ab5af  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x0b    9    0x00  0xfffffffe  0x0011  0x00c1.ee6ab5d7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x0c    9    0x00  0xfffffffe  0x0026  0x00c1.ee6ab605  0x03000438  0x0000.000.00000000  0x00000001   0x00000000  0x0d    9    0x00  0xfffffffe  0x000f  0x00c1.ee6ab5c5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x0e    9    0x00  0xfffffffe  0x0014  0x00c1.ee6ab5d3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x0f    9    0x00  0xfffffffe  0x0012  0x00c1.ee6ab5c7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x10    9    0x00  0xfffffffe  0x0029  0x00c1.ee6ab5c1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x11    9    0x00  0xfffffffe  0x0018  0x00c1.ee6ab5d9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x12    9    0x00  0xfffffffe  0x001e  0x00c1.ee6ab5c9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x13    9    0x00  0xfffffffe  0x001b  0x00c1.ee6ab5b9  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x14    9    0x00  0xfffffffe  0x000b  0x00c1.ee6ab5d5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x15    9    0x00  0xfffffffe  0x0003  0x00c1.ee6ab5b3  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x16    9    0x00  0xfffffffe  0x0023  0x00c1.ee6ab5f1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x17    9    0x00  0xfffffffe  0x0021  0x00c1.ee6ab5bd  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x18    9    0x00  0xfffffffe  0x0006  0x00c1.ee6ab5db  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x19    9    0x00  0xfffffffe  0x0049  0x00c1.ee6ab5df  0x03000437  0x0000.000.00000000  0x00000001   0x00000000  0x1a    9    0x00  0xfffffffe  0x0043  0x00c1.ee6ab617  0x03000438  0x0000.000.00000000  0x00000001   0x00000000  0x1b    9    0x00  0xfffffffe  0x0017  0x00c1.ee6ab5bb  0x03000436  0x0000.000.00000000  0x00000001   0x00000000  0x1c    9    0x00  0xfffffffe  0x002e  0x00c1.ee6ab5e9  0x03000437  0x0000.000.00000000  0x00000001 &n