What are the recommendations for 10 Gigabit network adapter tuning parameters?

https://access.redhat.com/solutions/127143

What are the recommendations for 10 Gigabit network adapter tuning parameters?

 SOLUTION 已验证 - 已更新 2017年三月20日01:07 - 

English 

环境

• Red Hat Enterprise Linux 7
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 5
• 10 Gigabit Ethernet network interface adapters (10GbE)

问题

• What are the recommendations for 10 Gigabit network adapter tuning parameters?
• 10Gbit NIC with MTU of 9000. Testing using iperf between two systems only results in 6-7 gigabits/s transfer speed.
• What are the expected and recommended tuning parameters to configure to achieve 10Gbps connection wirespeed for streaming bulk transfers?

决议

The following parameters are very general recommendations to achieve faster streaming transfers over 10 Gigabit Ethernet.

Depending on the actual production use of the network, these settings may not be optimal for your usage, however they will certainly make artificial benchmarks like iperf run faster.

Interrupt Balancing

Ensure the irqbalance service is installed and running.

Ring Buffer

It is recommended to make the NIC receive and transmit ring buffers as large as possible to prevent packet drops.

The procedure to increase the buffer size is described at:

• System dropping packets due to rx_fw_discards

The change can be persisted across reboots by following:

• How do I run a script or program immediately after my network interface goes up?

Socket Buffers

Increasing the maximum socket buffer size to 16MiB allows buffers to grow larger if required.

The following changes can be applied in /etc/sysctl.conf and applied immediately with sysctl -p:

Raw

# set min/default/max TCP read buffer
net.ipv4.tcp_rmem = 4096 87380 16777216
# set min/default/max TCP write buffer
net.ipv4.tcp_wmem = 4096 16384 16777216

# set maximum non-TCP receive socket buffer size
net.core.rmem_max = 16777216
# set maximum non-TCP write socket buffer size
net.core.wmem_max = 16777216

Increasing the default socket buffer sizes is not required as this basic level of tuning as it may harm efficiency of non-streaming transfers.

Network Budget

If the third column of /proc/net/softnet_stat is increasing, eg:

Raw

# cat /proc/net/softnet_stat
0073d76b 00000000 000049ae 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

Then there could be benefit to allowing the kernel packet receive handler to run for a slightly longer time.

The default value is 300 meaning 300 packets are received before the handler ends. This can be increased, though there is unlikely to be much improvement increasing this past several thousand.

The following change can be applied in /etc/sysctl.conf and applied immediately with sysctl -p:

Raw

# set number of packets softirq receives off NIC before exiting
net.core.netdev_budget = 1000

Further Reading

The following documents can assist in more precise tuning:

• Red Hat Enterprise Linux Network Performance Tuning Guide
• http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf