龊事一桩,解决 scsi 4:0:0:0: rejecting I/O to dead device

本文记录了一次服务器遇到I/O错误的问题排查与解决过程。作者通过检查系统日志发现了问题的根源在于不当的USB设备使用,并采取了一系列措施解决了问题。

今天登陆我们一台在上海的服务器,有同事说ftp无法正常上传文件,我想先看下分区的情况,于是:
df -h了一下,发现错误:
df: `/mnt': Input/output error
我吓了一条,dmesg了下,这下更的冷汗直冒:
scsi 4:0:0:0: rejecting I/O to dead device
FAT: Directory bread(block 32) failed
scsi 4:0:0:0: rejecting I/O to dead device
FAT: FAT read failed (blocknr 1)
scsi 4:0:0:0: rejecting I/O to dead device
FAT: FAT read failed (blocknr 1)
满屏这个信息,一般看到I/O err然后再来个read-only,基本上硬盘是出现物理坏道了,以前管理discuz!虚拟主机的两年中碰到过几次

这个时候我很紧张数据,赶紧把剩余的数据备份掉!!!速度很慢,等待中,我仔细检查了下系统日志。

 

我惊奇的发现:
Initializing USB Mass Storage driver...

MD,谁在我们机器上装过USB设备? 是谁!!!!! 我随便问了一下,同事告诉我,是edong那边要给我们装什么备案软件,需要提供机器的密码给他们,而我今天出去开会了,给过了,那问题清晰了! 长舒了一口气啊~~~
赶紧umount /mnt
umount: /mnt: device is busy

edong那帮人啊!
fuser /mnt 看了下是哪个进程使用此设备
fuser -km /mnt 强行干掉所有使用此设备的进程
umount /mnt

搞定!

[29015.022358] mpt3sas_cm0: search for end-devices: complete [29015.022358] mpt3sas_cm0: search for end-devices: start [29015.022358] mpt3sas_cm0: search for PCIe end-devices: complete [29015.022359] mpt3sas_cm0: search for expanders: start [29015.022376] expander present: handle(0x0017), sas_addr(0x5347379d0121f03f) [29015.022395] expander present: handle(0x0031), sas_addr(0x53473799ef5f403f) [29015.022410] mpt3sas_cm0: search for expanders: complete [29015.022415] mpt3sas_cm0: host reset: SUCCESS scmd(00000000fa36dd4f) [29067.001533] sd 20:0:1:0: Device offlined - not ready after error recovery [29067.001542] sd 20:0:1:0: [sdb] tag#5442 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=137s [29067.001544] sd 20:0:1:0: [sdb] tag#5442 CDB: Write(16) 8a 00 00 00 00 00 00 7f bf 40 00 00 00 20 00 00 [29067.001547] blk_update_request: I/O error, dev sdb, sector 8372032 op 0x1:(WRITE) flags 0x1000 phys_seg 4 prio class 0 [29067.004364] XFS (sdb1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x7fb740 len 32 error 5 [29067.044360] sd 20:0:12:0: Power-on or device reset occurred [29067.044489] sd 20:0:35:0: Power-on or device reset occurred [29067.044556] sd 20:0:30:0: Power-on or device reset occurred [29067.045517] sd 20:0:25:0: Power-on or device reset occurred [29067.046935] sd 20:0:13:0: Power-on or device reset occurred [29067.047438] sd 20:0:16:0: Power-on or device reset occurred [29067.047634] sd 20:0:33:0: Power-on or device reset occurred [29067.048062] sd 20:0:6:0: Power-on or device reset occurred [29067.049161] sd 20:0:26:0: Power-on or device reset occurred [29067.049320] sd 20:0:14:0: Power-on or device reset occurred [29067.049443] sd 20:0:21:0: Power-on or device reset occurred [29067.050046] sd 20:0:22:0: Power-on or device reset occurred [29067.050188] sd 20:0:34:0: Power-on or device reset occurred [29067.050220] sd 20:0:5:0: Power-on or device reset occurred [29067.050536] sd 20:0:3:0: Power-on or device reset occurred [29067.050600] sd 20:0:10:0: Power-on or device reset occurred [29067.051766] sd 20:0:36:0: Power-on or device reset occurred [29067.051872] sd 20:0:31:0: Power-on or device reset occurred [29067.052466] sd 20:0:19:0: Power-on or device reset occurred [29067.052849] sd 20:0:0:0: Power-on or device reset occurred [29067.158704] sd 20:0:17:0: Power-on or device reset occurred [29067.231964] sd 20:0:18:0: Power-on or device reset occurred [29067.257535] mpt3sas_cm0: removing unresponding devices: start [29067.257537] mpt3sas_cm0: removing unresponding devices: end-devices [29067.257541] mpt3sas_cm0: Removing unresponding devices: pcie end-devices [29067.257541] mpt3sas_cm0: removing unresponding devices: expanders [29067.257542] mpt3sas_cm0: removing unresponding devices: complete [29067.257546] sd 20:0:1:0: device_unblock and setting to running, handle(0x0019) [29067.257557] mpt3sas_cm0: scan devices: start [29067.257564] sd 20:0:1:0: rejecting I/O to offline device [29067.257566] blk_update_request: I/O error, dev sdb, sector 29324460136 op 0x0:(READ) flags 0x84700 phys_seg 128 prio class 0 [29067.257575] sd 20:0:1:0: rejecting I/O to offline device [29067.257578] blk_update_request: I/O error, dev sdb, sector 19329441936 op 0x1:(WRITE) flags 0x21800 phys_seg 2 prio class 0
最新发布
08-28
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值