https://access.redhat.com/solutions/344673
SOLUTION 已验证 - 已更新 2015年四月14日04:27 -
环境
- Red Hat Enterprise Linux 6.3
- DM-Multipath
问题
- After specifying the options "fast_io_fail_tmo" and "dev_loss_tmo" in multipath.conf, these values remain unchanged.
决议
An errata has been released for an internal BZ#974129 reported for the issue mentioned here. Updating the system to device-mapper-multipath-0.4.9-72.el6 or higher will resolve the issue.
With the current fix, multipath would be setting fast_io_fail_tmo value to higher than the existing dev_loss_tmo value. It first sets it to one less than the existing value. Then, if dev_loss_tmo is also being increased, it sets that and then sets fast_io_fail to the selected value.
That is, Multipath will correctly set any valid combination of fast_io_fail_tmo and dev_loss_tmo
根源
Mutipath sets fast_io_fail_tmo before setting dev_loss_tmo, and sysfs doesn't allow setting fast_io_fail_tmo to equal to or higher than dev_loss_tmo
As a consequence, if user set fast_io_fail_tmo to higher than the existing dev_loss_tmo value, multipath will be unable to set it.
诊断步骤
-
You can view the current values by the following commands :
-
fast_io_fail_tmo
$ for f in /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
- dev_loss_tmo
# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
These should be over-ridden by adding entries in /etc/multipath.conf:
defaults {
user_friendly_names yes
fast_io_fail_tmo <N>
dev_loss_tmo <N>
}
[root@localhost ~]# for f in /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-1:0-0:0x5005076303ffc4d6:off
rport-1:0-1:0x5005076303ffc4d6:off
rport-2:0-0:0x5005076303ffc4d6:off
rport-2:0-1:0x5005076303ffc4d6:off
rport-5:0-0:0x100000051e36126a:off
rport-5:0-1:0x100000051e36126a:off
rport-6:0-0:0x100000051e3637ba:off
rport-6:0-1:0x100000051e3637ba:off
[root@localhost ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-1:0-0:0x5005076303ffc4d6:8
rport-1:0-1:0x5005076303ffc4d6:8
rport-2:0-0:0x5005076303ffc4d6:8
rport-2:0-1:0x5005076303ffc4d6:8
rport-5:0-0:0x100000051e36126a:30
rport-5:0-1:0x100000051e36126a:30
rport-6:0-0:0x100000051e3637ba:30
rport-6:0-1:0x100000051e3637ba:30
- If we strace the multipathd daemon:
# strace -f -tt -T -v -x -o /tmp/stracing_multipathd.txt -s 4096 multipathd -v9
In the log file in /tmp you'll have something like:
/tmp/stracing_multipathd2.txt:1974 09:14:56.540154 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000048>
/tmp/stracing_multipathd2.txt-1974 09:14:56.540283 write(6, "\x33\x30\x00", 3) = -1 EINVAL (Invalid argument) <0.000024>
/tmp/stracing_multipathd2.txt-1974 09:14:56.540377 close(6) = 0 <0.000030>
--
/tmp/stracing_multipathd3.txt:2103 09:23:44.284830 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000055>
/tmp/stracing_multipathd3.txt-2103 09:23:44.284951 write(6, "30\0", 3) = -1 EINVAL (Invalid argument) <0.000021>
/tmp/stracing_multipathd3.txt-2103 09:23:44.285030 close(6) = 0 <0.000020>
--
/tmp/stracing_multipathd.txt:1887 09:00:00.782075 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000051>
/tmp/stracing_multipathd.txt-1887 09:00:00.782208 write(6, "\x33\x30\x00", 3) = -1 EINVAL (Invalid argument) <0.000024>
/tmp/stracing_multipathd.txt-1887 09:00:00.782301 close(6) = 0 <0.000023>
- Even when you have the correct order, you will still fail to set up dev_loss_tmo sometimes. This is because
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); donerport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:30
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:16
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done^C
[root@1950-pe2 ~]# echo 2147483647 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# echo 21 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# echo 2147483646 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo
21
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/
device/ fast_io_fail_tmo port_id port_state roles subsystem/ uevent
dev_loss_tmo node_name port_name power/ scsi_target_id supported_classes
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/node_name
0x200000e08b8f7684
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/roles
FCP Initiator
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-0/roles
FCP Target
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-2\:0-0/roles
FCP Target
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-2\:0-1/roles
FCP Initiator
[root@1950-pe2 ~]# echo 2147483647 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# echo 21 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:21
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:21
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
[root@1950-pe2 ~]# echo 600 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:600
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:21
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
-As we can see in drivers/scsi/scsi_transport_fc.c:
/*
* dev_loss_tmo: the default number of seconds that the FC transport
* should insulate the loss of a remote port.
* The maximum will be capped by the value of SCSI_DEVICE_BLOCK_MAX_TIMEOUT.
*/
static unsigned int fc_dev_loss_tmo = 60; /* seconds */
- And this is the max value according to drivers/scsi/scsi_priv.h:
#define SCSI_DEVICE_BLOCK_MAX_TIMEOUT 600 /* units in seconds */