问题:
Oracle归档日志备份成功,但2条channel有一个报错。
环境:
TSM服务器:2个Oracle node,4个DB2 node,共享8个drive。其中,两个oracle node分别共享其中4个drive。
报错的oracle node为RFIDP。
tsm: TSM>q path
Session established with server TSM: Linux/s390x
Server Version 6, Release 2, Level 4.0
Server date/time: 09/28/2015 14:26:33 Last access: 09/28/2015 13:06:41
Source Name Source Type Destination Destination On-Line
Name Type
----------- ----------- ----------- ----------- -------
TSM SERVER TS3500LIB1 LIBRARY Yes
TSM SERVER DRIVER10 DRIVE Yes
TSM SERVER DRIVER11 DRIVE Yes
TSM SERVER DRIVER12 DRIVE Yes
TSM SERVER DRIVER13 DRIVE Yes
TSM SERVER DRIVER14 DRIVE Yes
TSM SERVER DRIVER15 DRIVE Yes
TSM SERVER DRIVER16 DRIVE Yes
TSM SERVER DRIVER17 DRIVE Yes
JTFXPSTA1 SERVER DRIVER14 DRIVE Yes
JTFXPSTA1 SERVER DRIVER15 DRIVE Yes
JTFXPSTA1 SERVER DRIVER16 DRIVE Yes
JTFXPSTA1 SERVER DRIVER17 DRIVE Yes
RFIDPSTA1 SERVER DRIVER14 DRIVE Yes
RFIDPSTA1 SERVER DRIVER15 DRIVE Yes
RFIDPSTA1 SERVER DRIVER16 DRIVE Yes
RFIDPSTA1 SERVER DRIVER17 DRIVE Yes
排查过程:
检查TSM actlog
q actlog search=rfidp begint=11:00
从日志中发现, 有磁带mount失败。
进一步检查磁带mount失败的原因:
q actlog search=nj0174 begint=11:00
从日志中看到,driver14和driver16都无法mount这个磁带。
那么问题是在磁带上还是driver上呢?
首先检查改磁带状态:
tsm: TSM>q vol nj0174 f=d
Volume Name: NJ0174
Storage Pool Name: ORACLE_POOL
Device Class Name: 3592DEVC
Estimated Capacity: 12.0 T
Scaled Capacity Applied: 100
Pct Util: 22.9
Volume Status: Filling
Access: Read/Write
Pct. Reclaimable Space: 0.0
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 138
Write Pass Number: 1
Approx. Date Last Written: 09/28/2015 13:50:38
Approx. Date Last Read: 08/10/2015 01:10:30
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 08/10/2015 01:07:44
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: None
Logical Block Protected: No
tsm: TSM>
从上面的输出可以看到磁带状态正常。那么问题有可能在drive上。
继续查看actlog,关于driver14和driver16的记录:
tsm: TSM>q actlog search=driver14 begind=-1
tsm: TSM>q actlog search=driver16 begind=-1
这两个drive一直都无法mount磁带。问题可以确定就出在这两个drive上。
检查操作系统,带库所有drive状态。(drive过多,省略了若干)
发现driver14正常,driver16有一条路径是offline。
[root@TSM /]# lstape
FICON/ESCON tapes (found 0):
TapeNo BusID CuType/Model DevType/Model BlkSize State Op MedState
.
.
.
sg54 IBMtape13 31 13:0:1:1 IBM 03592E07 tapedrv
running
sg55 IBMtape14 32 14:0:0:1 IBM 03592E07 tapedrv
running
sg56 IBMchanger1 3 14:0:1:1 IBM 03584L22 changer
running
sg57 IBMtape15 33 14:0:2:1 IBM 03592E07 tapedrv
running
sg58 IBMtape16 34 15:0:0:1 IBM 03592E07 tapedrv
running
sg59 IBMtape17 35 15:0:1:1 IBM 03592E07 tapedrv
running
.
.
.
sg35 IBMtape14 32 8:0:0:1 IBM 03592E07 tapedrv
running
sg36 IBMchanger1 3 8:0:1:1 IBM 03584L22 changer
running
sg37 IBMtape15 33 8:0:2:1 IBM 03592E07 tapedrv
running
sg38 N/A 9:0:0:1 IBM 03592E07 tapedrv offline
sg39 IBMtape17 35 9:0:1:1 IBM 03592E07 tapedrv
running
[root@TSM /]#
登陆带库web控制台检查drive状态:
两个红框中的drive就是driver14和driver16。表面上看,状态正常。
对这两个drive做一次power cycle试试。
power cycle成功,在TSM server操作系统中,再次查看drive设备状态:driver16之前offline的路径已经变成running状态。
测试:
rman备份发起2个channel,该数据库节点在TSM server中配置了4条path,手动offline 2条,这样可以测试driver14和driver16的path。
tsm: TSM>q path
Session established with server TSM: Linux/s390x
Server Version 6, Release 2, Level 4.0
Server date/time: 09/28/2015 14:26:33 Last access: 09/28/2015 13:06:41
Source Name Source Type Destination Destination On-Line
Name Type
----------- ----------- ----------- ----------- -------
.
.
.
RFIDPSTA1 SERVER DRIVER14 DRIVE Yes
RFIDPSTA1 SERVER DRIVER15 DRIVE Yes
RFIDPSTA1 SERVER DRIVER16 DRIVE Yes
RFIDPSTA1 SERVER DRIVER17 DRIVE Yes
手动offline driver15和driver17
update path RFIDPSTA1 driver15 srctype=server desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=no
update path RFIDPSTA1 driver17 srctype=server desttype=drive DEVICE=/dev/IBMtape3 LIBRARY=ts3500lib1 online=no
tsm: TSM>update path RFIDPSTA1 driver15 srctype=server desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=no
ANR1726E A operation using the path from RFIDPSTA1 to TS3500LIB1 DRIVER15 is currently being used.
ANS8001I Return code 12.
driver15 in use,需要手动dismount磁带出来再进行offline
tsm: TSM>q mo
ANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.
ANR8329I 3592 volume NJ0080 is mounted R/W in drive DRIVER17 (/dev/IBMtape17), status: IDLE.
ANR8329I 3592 volume NJ0054 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.
ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.
ANR8329I 3592 volume NJ0035 is mounted R/W in drive DRIVER15 (/dev/IBMtape15), status: IDLE.
ANR8334I 5 matches found.
tsm: TSM>dismount vol nj0035
ANR8499I Command accepted.
手动dismount磁带成功。
再次查看mount状态:
tsm: TSM>q mo
ANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.
ANR8329I 3592 volume NJ0080 is mounted R/W in drive DRIVER17 (/dev/IBMtape17), status: IDLE.
ANR8329I 3592 volume NJ0054 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.
ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.
ANR8331I 3592 volume NJ0035 is mounted R/W in drive DRIVER15 (/dev/IBMtape15), status: DISMOUNTING.
ANR8334I 5 matches found.
dismount中
tsm: TSM>q mo
ANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.
ANR8329I 3592 volume NJ0054 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.
ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.
ANR8334I 4 matches found.
mount列表中已无driver15
再次手动offline driver15
tsm: TSM>update path RFIDPSTA1 driver15 srctype=server desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=no
ANR1722I A path from RFIDPSTA1 to TS3500LIB1 DRIVER15 has been updated.
offline成功
tsm: TSM>q path
Source Name Source Type Destination Destination On-Line
Name Type
----------- ----------- ----------- ----------- -------
TSM SERVER TS3500LIB1 LIBRARY Yes
TSM SERVER DRIVER10 DRIVE Yes
TSM SERVER DRIVER11 DRIVE Yes
TSM SERVER DRIVER12 DRIVE Yes
TSM SERVER DRIVER13 DRIVE Yes
TSM SERVER DRIVER14 DRIVE Yes
TSM SERVER DRIVER15 DRIVE Yes
TSM SERVER DRIVER16 DRIVE Yes
TSM SERVER DRIVER17 DRIVE Yes
JTFXPSTA1 SERVER DRIVER14 DRIVE Yes
JTFXPSTA1 SERVER DRIVER15 DRIVE Yes
JTFXPSTA1 SERVER DRIVER16 DRIVE Yes
JTFXPSTA1 SERVER DRIVER17 DRIVE Yes
RFIDPSTA1 SERVER DRIVER14 DRIVE Yes
RFIDPSTA1 SERVER DRIVER15 DRIVE No
RFIDPSTA1 SERVER DRIVER16 DRIVE Yes
RFIDPSTA1 SERVER DRIVER17 DRIVE No
重新运行归档日志备份脚本,测试driver14和driver16是否可用。
查看磁带mount情况,可以看到driver14和driver16已成功mount磁带,并且状态为R/W in use,说明备份正在写入磁带中。可以确定这两个drive已经可以正常使用了。
tsm: TSM>
tsm: TSM>q mo
ANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.
ANR8329I 3592 volume NJ0080 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.
ANR8330I 3592 volume NJ0035 is mounted R/W in drive DRIVER16 (/dev/IBMtape16), status: IN USE.
ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.
ANR8330I 3592 volume NJ0174 is mounted R/W in drive DRIVER14 (/dev/IBMtape14), status: IN USE.
ANR8334I 6 matches found.
继续等待,直到脚本执行完毕,检查脚本日志。备份成功,无报错。
[oracle@RFIDP log]$ tail /oracle/rmanscript/log/arch_092813.log
archive log filename=/oradata/arch/1_166187_777424034.dbf recid=166178 stamp=891610878
archive log filename=/oradata/arch/1_166188_777424034.dbf recid=166179 stamp=891611321
archive log filename=/oradata/arch/1_166189_777424034.dbf recid=166180 stamp=891611495
Finished backup at 28-SEP-15
released channel: t1
released channel: t2
Recovery Manager complete.
最后不要忘记把刚才offline的path重新online。
tsm: TSM>update path RFIDPSTA1 driver15 srctype=server desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=yes
ANR1722I A path from RFIDPSTA1 to TS3500LIB1 DRIVER15 has been updated.
tsm: TSM>update path RFIDPSTA1 driver17 srctype=server desttype=drive DEVICE=/dev/IBMtape3 LIBRARY=ts3500lib1 online=yes
ANR1722I A path from RFIDPSTA1 to TS3500LIB1 DRIVER17 has been updated.
tsm: TSM>
tsm: TSM>q path
Source Name Source Type Destination Destination On-Line
Name Type
----------- ----------- ----------- ----------- -------
TSM SERVER TS3500LIB1 LIBRARY Yes
TSM SERVER DRIVER10 DRIVE Yes
TSM SERVER DRIVER11 DRIVE Yes
TSM SERVER DRIVER12 DRIVE Yes
TSM SERVER DRIVER13 DRIVE Yes
TSM SERVER DRIVER14 DRIVE Yes
TSM SERVER DRIVER15 DRIVE Yes
TSM SERVER DRIVER16 DRIVE Yes
TSM SERVER DRIVER17 DRIVE Yes
JTFXPSTA1 SERVER DRIVER14 DRIVE Yes
JTFXPSTA1 SERVER DRIVER15 DRIVE Yes
JTFXPSTA1 SERVER DRIVER16 DRIVE Yes
JTFXPSTA1 SERVER DRIVER17 DRIVE Yes
RFIDPSTA1 SERVER DRIVER14 DRIVE Yes
RFIDPSTA1 SERVER DRIVER15 DRIVE Yes
RFIDPSTA1 SERVER DRIVER16 DRIVE Yes
RFIDPSTA1 SERVER DRIVER17 DRIVE Yes

1万+

被折叠的 条评论
为什么被折叠?



