What is the best bonding mode for TCP traffic such as NFS, ISCSI, CIFS, etc?

最新推荐文章于 2024-08-22 00:00:00 发布

转载最新推荐文章于 2024-08-22 00:00:00 发布 · 345 阅读

文章标签：

#bond #mode #rhel

RHEL7 专栏收录该内容

432 篇文章

订阅专栏

本文围绕Red Hat Enterprise Linux中TCP流量绑定模式展开。指出NFS、Samba/CIFS等大流量场景存在TCP重传等问题，根源在于部分绑定模式无法保证TCP流按序传输。决议推荐使用Bonding Mode 1、2、4等模式，并给出诊断步骤。

https://access.redhat.com/solutions/2217521

SOLUTION 已验证 - 已更新 2018年八月13日12:45 -

English

环境

Red Hat Enterprise Linux (all versions)
Bonding or Teaming
Large streaming TCP traffic such as NFS, Samba/CIFS, ISCSI, rsync over SSH/SCP, backups

问题

What is the best bonding mode for TCP traffic such as NFS and Samba/CIFS?
NFS repeatedly logs nfs: server not responding, still trying when no network issue is present
A packet capture displays many TCP retransmission, TCP Out-of-order, RPC retransmission, when there should be no reason for this.

决议

Use a bonding mode which guarantees in-order delivery of TCP traffic such as:

Bonding Mode 1 (active-backup)
Bonding Mode 2 (balance-xor)
Bonding Mode 4 (802.3ad aka LACP)
Bonding Mode 5 (balance-tlb) with tlb_dynamic_lb=0
Bonding Mode 6 (balance-alb) with tlb_dynamic_lb=0

Note that Bonding Mode 2 (balance-xor) requires an EtherChannel or similar configured on the switch, and Mode 4 (802.3ad) requires an EtherChannel with LACP on the switch. Bonding Mode 1 (active-backup) requires no switch configuration.

Bonding Modes 5 (balance-tlb) and 6 (balance-alb) do not require switch configuration. Mode 5 has no capability to balance traffic back into the bond. Mode 6 balances transmit by intercepting ARP requests, so may not be suitable for all situations such as where traffic mostly goes through a default gateway.

For advice on configuring bonding, refer to How do I configure a bonding device on Red Hat Enterprise Linux (RHEL)?

For advice on picking a specific hash policy for your traffic, refer to Why are all interfaces not used in bonding Mode 2 or Mode 4?

根源

The following bonding modes:

Bonding Mode 0 (round-robin)
Bonding Mode 3 (broadcast)
Bonding Mode 5 (balance-tlb) with tlb_dynamic_lb=1
Bonding Mode 6 (balance-alb) with tlb_dynamic_lb=1

Do not guarantee in-order delivery of TCP streams, as each packet of a stream may be transmitted down a different slave, and no switch guarantees that packets received in different switchports will be delivered in order.

Given the following example configuration:

Raw

.---------------------------.
| bond0 in 0 (round-robin)  |
'---------------------------'
| eth0 | eth1 | eth2 | eth3 |
'--=---'--=---'---=--'---=--'
   |      |       |      |
   |      |       |      |
.--=------=-------=------=--.
|          switch           |
'---------------------------'

The bond system may send traffic out each slave in a correct order, like ABCD ABCD ABCD, but the switch may forward this traffic in any random order, like CADB BDCA DACB.

As TCP on the receiver expects to be presented a TCP stream in-order, this causes the receiver to think it's missed packets and request retransmissions, to spend a great deal of time reassembling out-of-order traffic in to be in the correct order, and for the sender to waste bandwidth sending retransmissions which are not really required.

The following bonding modes:

Bonding Mode 1 (active-backup)
Bonding Mode 2 (balance-xor)
Bonding Mode 4 (802.3ad aka LACP)

Avoid this issue by transmitting traffic for one destination down the one slave. Mode 2 and Mode 4's balancing algorithm can be altered by the xmit_hash_policy bonding option, but they will never balance a single TCP stream down different ports, and so will avoid the problematic behaviour discussed above.

It is not possible to effectively balance a single TCP stream across multiple bonding or teaming devices. If higher speed is required for a single stream, then faster interfaces (and possibly faster network infrastructure) must be used.

This theory applies to all TCP streams. The most common occurrences of this issue are seen on high-speed long-lived TCP streams such as NFS, Samba/CIFS, ISCSI, rsync over SSH/SCP, and so on.