Unable to set dev_loss_tmo and fast_io_fail_tmo in multipath.conf

https://access.redhat.com/solutions/344673

 SOLUTION 已验证 - 已更新 2015年四月14日04:27 - 

English 

环境

• Red Hat Enterprise Linux 6.3
• DM-Multipath

问题

• After specifying the options "fast_io_fail_tmo" and "dev_loss_tmo" in multipath.conf, these values remain unchanged.

决议

An errata has been released for an internal BZ#974129 reported for the issue mentioned here. Updating the system to device-mapper-multipath-0.4.9-72.el6 or higher will resolve the issue.

With the current fix, multipath would be setting fast_io_fail_tmo value to higher than the existing dev_loss_tmo value. It first sets it to one less than the existing value. Then, if dev_loss_tmo is also being increased, it sets that and then sets fast_io_fail to the selected value.

That is, Multipath will correctly set any valid combination of fast_io_fail_tmo and dev_loss_tmo

根源

Mutipath sets fast_io_fail_tmo before setting dev_loss_tmo, and sysfs doesn't allow setting fast_io_fail_tmo to equal to or higher than dev_loss_tmo

As a consequence, if user set fast_io_fail_tmo to higher than the existing dev_loss_tmo value, multipath will be unable to set it.

诊断步骤

• You can view the current values by the following commands :

• fast_io_fail_tmo

Raw

$ for f in /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done

• dev_loss_tmo

Raw

# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done

These should be over-ridden by adding entries in /etc/multipath.conf:

Raw

defaults {
        user_friendly_names yes
        fast_io_fail_tmo <N>
        dev_loss_tmo <N>
}

Raw

[root@localhost ~]#  for f in /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-1:0-0:0x5005076303ffc4d6:off
rport-1:0-1:0x5005076303ffc4d6:off
rport-2:0-0:0x5005076303ffc4d6:off
rport-2:0-1:0x5005076303ffc4d6:off
rport-5:0-0:0x100000051e36126a:off
rport-5:0-1:0x100000051e36126a:off
rport-6:0-0:0x100000051e3637ba:off
rport-6:0-1:0x100000051e3637ba:off
[root@localhost ~]#  for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-1:0-0:0x5005076303ffc4d6:8
rport-1:0-1:0x5005076303ffc4d6:8
rport-2:0-0:0x5005076303ffc4d6:8
rport-2:0-1:0x5005076303ffc4d6:8
rport-5:0-0:0x100000051e36126a:30
rport-5:0-1:0x100000051e36126a:30
rport-6:0-0:0x100000051e3637ba:30
rport-6:0-1:0x100000051e3637ba:30

• If we strace the multipathd daemon:

Raw

#  strace -f -tt -T -v -x -o /tmp/stracing_multipathd.txt -s 4096 multipathd -v9

In the log file in /tmp you'll have something like:

Raw

/tmp/stracing_multipathd2.txt:1974  09:14:56.540154 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000048>
/tmp/stracing_multipathd2.txt-1974  09:14:56.540283 write(6, "\x33\x30\x00", 3) = -1 EINVAL (Invalid argument) <0.000024>
/tmp/stracing_multipathd2.txt-1974  09:14:56.540377 close(6)          = 0 <0.000030>
--
/tmp/stracing_multipathd3.txt:2103  09:23:44.284830 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000055>
/tmp/stracing_multipathd3.txt-2103  09:23:44.284951 write(6, "30\0", 3) = -1 EINVAL (Invalid argument) <0.000021>
/tmp/stracing_multipathd3.txt-2103  09:23:44.285030 close(6)          = 0 <0.000020>
--
/tmp/stracing_multipathd.txt:1887  09:00:00.782075 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000051>
/tmp/stracing_multipathd.txt-1887  09:00:00.782208 write(6, "\x33\x30\x00", 3) = -1 EINVAL (Invalid argument) <0.000024>
/tmp/stracing_multipathd.txt-1887  09:00:00.782301 close(6)          = 0 <0.000023>

• Even when you have the correct order, you will still fail to set up dev_loss_tmo sometimes. This is because

Raw

[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); donerport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:30
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:16
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done^C
[root@1950-pe2 ~]# echo 2147483647 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# echo 21 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
[root@1950-pe2 ~]# echo 2147483646 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
21
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/
device/            fast_io_fail_tmo   port_id            port_state         roles              subsystem/         uevent             
dev_loss_tmo       node_name          port_name          power/             scsi_target_id     supported_classes  
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/node_name 
0x200000e08b8f7684
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/roles 
FCP Initiator
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-0/roles 
FCP Target
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-2\:0-0/roles 
FCP Target
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-2\:0-1/roles 
FCP Initiator
[root@1950-pe2 ~]# echo 2147483647 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# echo 21 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:21
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:21
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
[root@1950-pe2 ~]# echo 600 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:600
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:21
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16

-As we can see in drivers/scsi/scsi_transport_fc.c:

Raw

/*
 * dev_loss_tmo: the default number of seconds that the FC transport
 *   should insulate the loss of a remote port.
 *   The maximum will be capped by the value of SCSI_DEVICE_BLOCK_MAX_TIMEOUT.
 */
static unsigned int fc_dev_loss_tmo = 60;               /* seconds */

• And this is the max value according to drivers/scsi/scsi_priv.h:

Raw

#define SCSI_DEVICE_BLOCK_MAX_TIMEOUT 600 /* units in seconds */