ORA-15080: synchronous I/O operation to a disk faile

1 问题概述

生产环境中，有一套Oracle RAC，突然反映说IO 错误: The Network Adapter could not establish the connection，并且节点掉了一个。同时备库、单实例库均无法访问。

于是查看trace log进行分析，发现以下报错且不断循环：

Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05 WARNING: Write Failed. group:1 disk:0 AU:2112 offset:0 size:173056

Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
WARNING: Write Failed. group:1 disk:0 AU:905 offset:721408 size:512Errors in file /u01/app/oracle/diag/rdbms/racdb_stan
b/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed

WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05
WARNING: failed to write mirror side 1 of virtual extent 36 logical extent 0 of file 266 in group 1 on disk 0 allocatio
111
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05 NOTE: Suppress further IO Write errors on group:1 disk:0

Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_pr00_31669.trc:
ORA-00333: redo log read error block 74851 count 1
ORA-00312: online log 6 thread 2: '+FRA/racdb_standby/onlinelog/redo06.log'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
NOTE: Suppress further IO Write errors on group:1 disk:0
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: Read Failed. group:1 disk:0 AU:2111 offset:574976 size:512
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
.......
.......

2 问题解决

经过排查，是由于存储掉电引起的故障，有存储的交换机出现异常导致。
rac环境在存储掉电后，两个节点的服务器都会自动重启。之前的rac环境掉了一个节点是由于服务器光驱内有安装光盘卡住，导致未重启成功。

服务器重启后，rac环境自动重启，单实例环境手动重启。

重启后需要注意：

• 无论单实例还是rac都要检查并重新开启监听
• 如果EM上有shell作业的也需要重新开启EM
• 受影响未执行备份作业的，为确保数据安全需要手动执行全备
• 逐一检查各项状态是否正常，比如存储是否重新挂载、备库是否有gap等