2.5. Link Aggregation and High Availability with Bonding

2.5. Link Aggregation and High Availability with Bonding

 

Networking vendors have long offered a functionality for aggregating bandwidth across multiple physical links to a switch. This allows a machine (frequently a server) to treat multiple physical connections to switch units as a single logical link. The standard moniker for this technology is IEEE 802.3ad, although it is known by the common names of trunking, port trunking and link aggregation. The conventional use of bonding under linux is an implementation of this link aggregation.

A separate use of the same driver allows the kernel to present a single logical interface for two physical links to two separate switches. Only one link is used at any given time. By using media independent interface signal failure to detect when a switch or link becomes unusable, the kernel can, transparently to userspace and application layer services, fail to the backup physical connection. Though not common, the failure of switches, network interfaces, and cables can cause outages. As a component of high availability planning, these bonding techniques can help reduce the number of single points of failure.

For more information on bonding, see the Documentation/networking/bonding.txt from the linux source code tree.

 

2.5.1. Link Aggregation

Bonding for link aggregation must be supported by both endpoints. Two linux machines connected via crossover cables can take advantage of link aggregation. A single machine connected with two physical cables to a switch which supports port trunking can use link aggregation to the switch. Any conventional switch will become ineffably confused by a hardware address appearing on multiple ports simultaneously.

Example 2.12. Link aggregation bonding

[root@real-server root]# modprobe bonding [root@real-server root]# ip addr add 192.168.100.33/24 brd + dev bond0 [root@real-server root]# ip link set dev bond0 up [root@real-server root]# ifenslave bond0 eth2 eth3 master has no hw address assigned; getting one from slave! The interface eth2 is up, shutting it down it to enslave it. The interface eth3 is up, shutting it down it to enslave it. [root@real-server root]# ifenslave bond0 eth2 eth3 [root@real-server root]# cat /proc/net/bond0/info Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 0 Slave Interface: eth3 MII Status: up Link Failure Count: 0

FIXME; Need an experiment here....maybe a tcpdump to show how the management frames appear on the wire.

This Beowulf software page describes in a bit more detail the rationale and a practical application of linux channel bonding (for link aggregation).

 

 

2.5.2. High Availability

Bonding support under linux is part of a high availability solution. For an entry point into the complexity of high availability in conjunction with linux, see the linux-ha.org site. To guard against layer two (switch) and layer one (cable) failure, a machine can be configured with multiple physical connections to separate switch devices while presenting a single logical interface to userspace.

The name of the interface can be specified by the user. It is commonly bond0 or something similar. As a logical interface, it can be used in routing tables and by tcpdump.

The bond interface, when created, has no link layer address. In the example below, an address is manually added to the interface. See Example 2.12, “Link aggregation bonding” for an example of the bonding driver reporting setting the link layer address when the first device is enslaved to the bond (doesn't that sound cruel!).

Example 2.13. High availability bonding

[root@real-server root]# modprobe bonding mode=1 miimon=100 downdelay=200 updelay=200 [root@real-server root]# ip link set dev bond0 addr 00:80:c8:e7:ab:5c [root@real-server root]# ip addr add 192.168.100.33/24 brd + dev bond0 [root@real-server root]# ip link set dev bond0 up [root@real-server root]# ifenslave bond0 eth2 eth3 The interface eth2 is up, shutting it down it to enslave it. The interface eth3 is up, shutting it down it to enslave it. [root@real-server root]# ip link show eth2 ; ip link show eth3 ; ip link show bond0 4: eth2: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 100 link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff 5: eth3: <BROADCAST,MULTICAST,NOARP,SLAVE,DEBUG,AUTOMEDIA,PORTSEL,NOTRAILERS,UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 100 link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff 58: bond0: <BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff

Immediately noticeable, there is a new flag in the ip link show output. The MASTER and SLAVE flags clearly report the nature of the relationship between the interfaces. Also, the Ethernet interfaces indicate the master interface via the keywords master bond0.

Note also, that all three of the interfaces share the same link layer address, 00:80:c8:e7:ab:5c.

FIXME; What doe DEBUG,AUTOMEDIA,PORTSEL,NOTRAILERS mean?