iscsi Initiator 登录target时报错iscsiadm: Login I/O error, failed to receive a PDU

iscsi Initiator 登录target时报错iscsiadm: Login I/O error, failed to receive a PDU

[root@rac2 ~]# iscsiadm -m discovery -t st -p 10.10.10.111:3260 -I iser
iscsiadm: recv's end state machine bug?
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.10.10.111
iscsiadm: recv's end state machine bug?
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.10.10.111
iscsiadm: recv's end state machine bug?
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.10.10.111
iscsiadm: recv's end state machine bug?
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.10.10.111
iscsiadm: recv's end state machine bug?
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.10.10.111
iscsiadm: recv's end state machine bug?
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.10.10.111
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: Could not perform SendTargets discovery: encountered iSCSI login failure
[root@rac2 ~]#

但是ping 该ip地址是ok的。

[root@rac2 ~]# ping 10.10.10.111
PING 10.10.10.111 (10.10.10.111) 56(84) bytes of data.
64 bytes from 10.10.10.111: icmp_seq=1 ttl=64 time=0.156 ms
64 bytes from 10.10.10.111: icmp_seq=2 ttl=64 time=0.147 ms
64 bytes from 10.10.10.111: icmp_seq=3 ttl=64 time=0.109 ms
^C
--- 10.10.10.111 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.109/0.137/0.156/0.022 ms
[root@rac2 ~]#

根据redhat.com文章：
https://access.redhat.com/solutions/2158021
明确说明是存储端的问题。

建议使用 debug模式来排错

iscsiadm -m discovery -t st -d8 -p X.X.X.X

于是

iscsiadm -m discovery -t st -d8 -p 10.10.10.111:3260 -I iser

不过功力尚浅，看不出什么门道，先记录下来输出：

[root@rac2 ~]# iscsiadm -m discovery -t st -d8 -p 10.10.10.111:3260 -I iser
iscsiadm: ip 10.10.10.111, port 3260, tgpt -1
iscsiadm: Max file limits 1024 4096

iscsiadm: updating defaults from '/etc/iscsi/iscsid.conf'
iscsiadm: updated 'discovery.sendtargets.iscsi.MaxRecvDataSegmentLength', '32768' => '32768'
iscsiadm: updated 'node.startup', 'manual' => 'automatic'
iscsiadm: updated 'node.leading_login', 'No' => 'No'
iscsiadm: updated 'node.session.timeo.replacement_timeout', '120' => '120'
iscsiadm: updated 'node.conn[0].timeo.login_timeout', '30' => '15'
iscsiadm: updated 'node.conn[0].timeo.logout_timeout', '15' => '15'
iscsiadm: updated 'node.conn[0].timeo.noop_out_interval', '5' => '5'
iscsiadm: updated 'node.conn[0].timeo.noop_out_timeout', '5' => '5'
iscsiadm: updated 'node.session.err_timeo.abort_timeout', '15' => '15'
iscsiadm: updated 'node.session.err_timeo.lu_reset_timeout', '30' => '30'
iscsiadm: updated 'node.session.err_timeo.tgt_reset_timeout', '30' => '30'
iscsiadm: updated 'node.session.initial_login_retry_max', '4' => '8'
iscsiadm: updated 'node.session.cmds_max', '128' => '128'
iscsiadm: updated 'node.session.queue_depth', '32' => '32'
iscsiadm: updated 'node.session.xmit_thread_priority', '-20' => '-20'
iscsiadm: updated 'node.session.iscsi.InitialR2T', 'No' => 'No'
iscsiadm: updated 'node.session.iscsi.ImmediateData', 'Yes' => 'Yes'
iscsiadm: updated 'node.session.iscsi.FirstBurstLength', '262144' => '262144'
iscsiadm: updated 'node.session.iscsi.MaxBurstLength', '16776192' => '16776192'
iscsiadm: updated 'node.conn[0].iscsi.MaxRecvDataSegmentLength', '262144' => '262144'
iscsiadm: updated 'node.conn[0].iscsi.MaxXmitDataSegmentLength', '0' => '0'
iscsiadm: updated 'node.conn[0].iscsi.HeaderDigest', 'None' => 'None'
iscsiadm: updated 'node.session.nr_sessions', '1' => '1'
iscsiadm: updated 'node.session.iscsi.FastAbort', 'Yes' => 'Yes'
iscsiadm: in read_transports
iscsiadm: Adding new transport iser
iscsiadm: Matched transport iser

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'handle'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/handle'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/handle'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/handle' with attribute value '18446744072106209312'

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'caps'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/caps'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/caps'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/caps' with attribute value '0x89'

iscsiadm: Could not match iface[hw=,ip=,net_if=,iscsi_if=iser] to host.
iscsiadm: Looking for config file /var/lib/iscsi/send_targets/10.10.10.111,3260

iscsiadm: Looking for config file /var/lib/iscsi/send_targets/10.10.10.111,3260 config st_config.
iscsiadm: updated 'discovery.startup', 'manual' => 'manual'
iscsiadm: updated 'discovery.type', 'sendtargets' => 'sendtargets'
iscsiadm: updated 'discovery.sendtargets.address', '' => '10.10.10.111'
iscsiadm: updated 'discovery.sendtargets.port', '0' => '3260'
iscsiadm: updated 'discovery.sendtargets.auth.authmethod', 'None' => 'None'
iscsiadm: updated 'discovery.sendtargets.timeo.login_timeout', '15' => '15'
iscsiadm: updated 'discovery.sendtargets.use_discoveryd', 'No' => 'No'
iscsiadm: updated 'discovery.sendtargets.discoveryd_poll_inval', '30' => '30'
iscsiadm: updated 'discovery.sendtargets.reopen_max', '5' => '5'
iscsiadm: updated 'discovery.sendtargets.timeo.auth_timeout', '45' => '45'
iscsiadm: updated 'discovery.sendtargets.timeo.active_timeout', '30' => '30'
iscsiadm: updated 'discovery.sendtargets.iscsi.MaxRecvDataSegmentLength', '32768' => '32768'
iscsiadm: disc rec already exists
iscsiadm: Looking for config file /var/lib/iscsi/send_targets/10.10.10.111,3260

iscsiadm: starting sendtargets discovery, address 10.10.10.111:3260,
iscsiadm: in read_transports
iscsiadm: Updating transport iser
iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'handle'

iscsiadm: sysfs_attr_get_value: found in cache '/class/iscsi_transport/iser/handle'

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'caps'

iscsiadm: sysfs_attr_get_value: found in cache '/class/iscsi_transport/iser/caps'

iscsiadm: no authentication configured...
iscsiadm: sendtargets discovery to 10.10.10.111:3260 using isid 0x00023d000000
iscsiadm: resolved 10.10.10.111 to 10.10.10.111
iscsiadm: discovery timeouts: login 15, reopen_cnt 6, auth 45.
iscsiadm: in ctldev_open
iscsiadm: created NETLINK_ISCSI socket...
iscsiadm: setting iface iser, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: ktransport_ep_connect got handle 4
iscsiadm: in ktransport_ep_poll
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: in ktransport_ep_poll
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: iscsi_create_leading_conn discovery create session

iscsiadm: in kcreate_session
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: expecting event 28, got 106, handling...
iscsiadm: in ctldev_handle
iscsiadm: in nl_read
iscsiadm: ctldev_handle got event type 106

iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: iscsi_create_leading_conn discovery created session 18

iscsiadm: iscsi_create_leading_conn discovery create conn

iscsiadm: in kcreate_conn
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: iscsi_create_leading_conn discovery bind conn

iscsiadm: in kbind_conn
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: connected to discovery address 10.10.10.111
iscsiadm: discovery session to 10.10.10.111:3260 starting iSCSI login
iscsiadm: iscsi_create_session discovery set params

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_connection/connection18:0'/'ping_tmo'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_connection/connection18:0/ping_tmo'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_connection/connection18:0/ping_tmo'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_connection/connection18:0/ping_tmo' with attribute value '0'

iscsiadm: in kset_param
iscsiadm: set operational parameter 15 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: set operational parameter 17 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 18 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 19 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: set operational parameter 22 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: set operational parameter 23 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: set operational parameter 24 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: set operational parameter 25 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 26 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 27 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 28 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 35 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 30 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 31 to:
iscsiadm: 0
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 32 to:
iscsiadm: iser
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 34 to:
iscsiadm: iqn.1994-05.com.redhat:3fea8217eb5a
iscsiadm: in kset_param
iscsiadm: set operational parameter 39 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: set operational parameter 40 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: set operational parameter 41 to:
iscsiadm:
iscsiadm: in kset_param
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: set operational parameter 43 to:
iscsiadm: 1
iscsiadm: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d120000 exp_statsn 0
iscsiadm: >    InitiatorName=iqn.1994-05.com.redhat:3fea8217eb5a
iscsiadm: >    InitiatorAlias=rac2
iscsiadm: >    SessionType=Discovery
iscsiadm: >    HeaderDigest=None
iscsiadm: >    DataDigest=None
iscsiadm: >    DefaultTime2Wait=2
iscsiadm: >    DefaultTime2Retain=0
iscsiadm: >    IFMarker=No
iscsiadm: >    OFMarker=No
iscsiadm: >    ErrorRecoveryLevel=0
iscsiadm: >    MaxRecvDataSegmentLength=8192
iscsiadm: in ksend_pdu_begin
iscsiadm: send PDU began for hdr 48 bytes and data 244 bytes
iscsiadm: in kwritev
iscsiadm: wrote 48 bytes of PDU header
iscsiadm: in kwritev
iscsiadm: wrote 244 bytes of PDU data
iscsiadm: in ksend_pdu_end
iscsiadm: in __kipc_call
iscsiadm: in kwritev
iscsiadm: in nlpayload_read
iscsiadm: in nlpayload_read
iscsiadm: send PDU finished for conn 18:0
^Ciscsiadm: caught SIGINT, exiting... --------->>>>这是我执行ctrl+c的结果，否则会一直输出。
[root@rac2 ~]#

突然想到我曾经写过测试ib网络的文章：
http://blog.youkuaiyun.com/msdnchina/article/details/71133494

于是在服务器端（有10.10.10.111 这个ip地址的这一端）敲入：

ibping -S -C mlx4_1 -P 2

在客户端执行：ibping -c 10000 -f -C mlx4_1 -P 2 -L 0

[root@rac2 ~]# ibping -c 10000 -f -C mlx4_1 -P 2 -L 0
ibping: iberror: failed: can't resolve destination port 0
[root@rac2 ~]# 

其实，有经验的人，一下就看出来了，链路正常的话，-L后边的值不能为0，这正说明ib链路不通，虽然ping 10.10.10.111能通。