解决Dell R410 网卡在Centos5.5 上丢包  

 
 

现象是每几十次访问网页就会一次打不开,数据库,临控等才时常取不到数据.
好像r410的网卡和centos5.5不怎么兼容

uname -a

  1. Linux c1gserver 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

dmesg |grep bnx

  1. Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)
  2. bnx2: eth0: using MSIX
  3. bnx2: eth0 NIC Copper Link is Up, 100 Mbps full duplex

ifconfig
有大量的丢包;dropped:6620400

  1. eth0      Link encap:Ethernet  HWaddr 00:26:B9:3B:XX:C3 
  2.           inet addr:61.255.xx.xx  Bcast:61.255.xx.xx  Mask:255.255.255.192
  3.           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  4.           RX packets:7330597243 errors:1 dropped:6620400 overruns:0 frame:1
  5.           TX packets:8116489614 errors:0 dropped:0 overruns:0 carrier:0
  6.           collisions:0 txqueuelen:1000
  7.           RX bytes:3435468718053 (3.1 TiB)  TX bytes:3656517601592 (3.3 TiB)
  8.           Interrupt:82 Memory:da000000-da012800

一.先尝试增加网卡缓存来解决问题
1.ethtool -g eth0

  1. Ring parameters for eth0:
  2. Pre-set maximums:
  3. RX:             1020
  4. RX Mini:        0
  5. RX Jumbo:       4080
  6. TX:             255
  7. Current hardware settings:
  8. RX:             255
  9. RX Mini:        0
  10. RX Jumbo:       0
  11. TX:             255

2.ethtool -G eth0 rx 1000
ethtool -g eth0

  1. Ring parameters for eth0:
  2. Pre-set maximums:
  3. RX:             1020
  4. RX Mini:        0
  5. RX Jumbo:       4080
  6. TX:             255
  7. Current hardware settings:
  8. RX:             1000
  9. RX Mini:        0
  10. RX Jumbo:       0
  11. TX:             255

ifconfig

  1. eth0      Link encap:Ethernet  HWaddr 00:26:B9:3B:XX:C3 
  2.           inet addr:61.255.xx.xx  Bcast:61.255.xx.xx  Mask:255.255.255.192
  3.           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  4.           RX packets:1348879 errors:0 dropped:2714 overruns:0 frame:0
  5.           TX packets:1449564 errors:0 dropped:0 overruns:0 carrier:0
  6.           collisions:0 txqueuelen:1000
  7.           RX bytes:646430926 (616.4 MiB)  TX bytes:576842481 (550.1 MiB)
  8.           Interrupt:82 Memory:da000000-da012800

ethtool -S eth0

  1. NIC statistics:
  2.      rx_bytes: 645417070
  3.      rx_error_bytes: 0
  4.      tx_bytes: 575096746
  5.      tx_error_bytes: 0
  6.      rx_ucast_packets: 1345249
  7.      rx_mcast_packets: 17
  8.      rx_bcast_packets: 826
  9.      tx_ucast_packets: 1446406
  10.      tx_mcast_packets: 0
  11.      tx_bcast_packets: 0
  12.      tx_mac_errors: 0
  13.      tx_carrier_errors: 0
  14.      rx_crc_errors: 0
  15.      rx_align_errors: 0
  16.      tx_single_collisions: 0
  17.      tx_multi_collisions: 0
  18.      tx_deferred: 0
  19.      tx_excess_collisions: 0
  20.      tx_late_collisions: 0
  21.      tx_total_collisions: 0
  22.      rx_fragments: 0
  23.      rx_jabbers: 0
  24.      rx_undersize_packets: 0
  25.      rx_oversize_packets: 0
  26.      rx_64_byte_packets: 620303
  27.      rx_65_to_127_byte_packets: 87492
  28.      rx_128_to_255_byte_packets: 36556
  29.      rx_256_to_511_byte_packets: 212099
  30.      rx_512_to_1023_byte_packets: 45961
  31.      rx_1024_to_1522_byte_packets: 343681
  32.      rx_1523_to_9022_byte_packets: 0
  33.      tx_64_byte_packets: 368814
  34.      tx_65_to_127_byte_packets: 714315
  35.      tx_128_to_255_byte_packets: 13048
  36.      tx_256_to_511_byte_packets: 18258
  37.      tx_512_to_1023_byte_packets: 39752
  38.      tx_1024_to_1522_byte_packets: 292219
  39.      tx_1523_to_9022_byte_packets: 0
  40.      rx_xon_frames: 0
  41.      rx_xoff_frames: 0
  42.      tx_xon_frames: 0
  43.      tx_xoff_frames: 0
  44.      rx_mac_ctrl_frames: 0
  45.      rx_filtered_packets: 0
  46.      rx_discards: 0
  47.      rx_fw_discards: 2704

rx_fw_discards: 2704
问题没有解决,依然丢包

二.更新网卡驱动
lsmod |grep bnx2

  1. bnx2                  209997  0

ethtool -i eth0

  1. driver: bnx2
  2. version: 1.9.3
  3. firmware-version: 5.0.6 NCSI 2.0.3
  4. bus-info: 0000:01:00.0

modinfo bnx2

  1. filename:       /lib/modules/2.6.18-164.el5/kernel/drivers/net/bnx2.ko
  2. version:        1.9.3
  3. license:        GPL
  4. description:    Broadcom NetXtreme II BCM5706/5708/5709/5716 Driver
  5. author:         Michael Chan <mchan@broadcom.com>
  6. srcversion:     D151EAED8C1037CA480DE9A
  7. alias:          pci:v000014E4d0000163Csv*sd*bc*sc*i*
  8. alias:          pci:v000014E4d0000163Bsv*sd*bc*sc*i*
  9. alias:          pci:v000014E4d0000163Asv*sd*bc*sc*i*
  10. alias:          pci:v000014E4d00001639sv*sd*bc*sc*i*
  11. alias:          pci:v000014E4d000016ACsv*sd*bc*sc*i*
  12. alias:          pci:v000014E4d000016AAsv*sd*bc*sc*i*
  13. alias:          pci:v000014E4d000016AAsv0000103Csd00003102bc*sc*i*
  14. alias:          pci:v000014E4d0000164Csv*sd*bc*sc*i*
  15. alias:          pci:v000014E4d0000164Asv*sd*bc*sc*i*
  16. alias:          pci:v000014E4d0000164Asv0000103Csd00003106bc*sc*i*
  17. alias:          pci:v000014E4d0000164Asv0000103Csd00003101bc*sc*i*
  18. depends:       
  19. vermagic:       2.6.18-164.el5 SMP mod_unload gcc-4.1
  20. parm:           disable_msi:Disable Message Signaled Interrupt (MSI) (int)
  21. parm:           enable_entropy:Allow bnx2 to populate the /dev/random entropy pool (int)
  22. module_sig:     883f3504a9f766557f09578a977b7e112ebf209f6aab61295466fb787675fec1378b254df8d186609f705f1f37e06d7e2958667d258f6afe6ac1621228

1.从官网下载驱动
http://zh-cn.broadcom.com/support/ethernet_nic/netxtremeii.php

Linux
iSCSI HBA only supported on RHEL 5.4, SUSE SLES 11 SP1 or newer versions of these distributions
6.2.23 03/18/11
24MB

2.安装包
unzip linux-6.2.23.zip
cd Server/Linux/Driver/
rpm -ivh netxtreme2-6.2.23-1.src.rpm

  1. 1:netxtreme2                                                       ########################################### [100%]

3.编译
cd /usr/src/redhat/
rpmbuild -bb SPECS/netxtreme2.spec

  1. Wrote: /usr/src/redhat/RPMS/x86_64/netxtreme2-6.2.23-1.x86_64.rpm
  2. Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.43778
  3. + umask 022
  4. + cd /usr/src/redhat/BUILD
  5. + cd netxtreme2-6.2.23
  6. + rm -rf /var/tmp/netxtreme2-buildroot /usr/src/redhat/BUILD/file.list.netxtreme2
  7. + exit 0

exit 0表示成功(1-255是失败)

4.安装生成的rpm包
cd RPMS/x86_64/
rpm -ivh netxtreme2-6.2.23-1.x86_64.rpm

  1. Preparing...                                                          ########################################### [100%]
  2.    1:netxtreme2                                                       ########################################### [100%]

新的驱动会生成在/lib/modules/2.6.18-164.el5/updates/

5.加载驱动
depmod -a
modprobe bnx2
service network restart
lsmod |grep bnx2

  1. bnx2                  209997  0

ethtool -i eth0

  1. driver: bnx2
  2. version: 1.9.3
  3. firmware-version: 5.0.6 NCSI 2.0.3
  4. bus-info: 0000:01:00.0

看着好像没区别,modinfo可以看到详细信息
modinfo bnx2

  1. filename:       /lib/modules/2.6.18-164.el5/updates/bnx2.ko
  2. version:        2.0.23b
  3. license:        GPL
  4. description:    Broadcom NetXtreme II BCM5706/5708/5709/5716 Driver
  5. author:         Michael Chan <mchan@broadcom.com>
  6. srcversion:     6E0DD070AB24C11F50B2712
  7. alias:          pci:v000014E4d0000163Csv*sd*bc*sc*i*
  8. alias:          pci:v000014E4d0000163Bsv*sd*bc*sc*i*
  9. alias:          pci:v000014E4d0000163Asv*sd*bc*sc*i*
  10. alias:          pci:v000014E4d00001639sv*sd*bc*sc*i*
  11. alias:          pci:v000014E4d000016ACsv*sd*bc*sc*i*
  12. alias:          pci:v000014E4d000016AAsv*sd*bc*sc*i*
  13. alias:          pci:v000014E4d000016AAsv0000103Csd00003102bc*sc*i*
  14. alias:          pci:v000014E4d0000164Csv*sd*bc*sc*i*
  15. alias:          pci:v000014E4d0000164Asv*sd*bc*sc*i*
  16. alias:          pci:v000014E4d0000164Asv0000103Csd00003106bc*sc*i*
  17. alias:          pci:v000014E4d0000164Asv0000103Csd00003101bc*sc*i*
  18. depends:       
  19. vermagic:       2.6.18-164.el5 SMP mod_unload gcc-4.1
  20. parm:           disable_msi:Disable Message Signaled Interrupt (MSI) (int)
  21. parm:           stop_on_tx_timeout:For debugging purposes, prevent a chip  reset when a tx timeout occurs (int)

观察一段时间没有问题

20110607更新
附加进一步解决方法

在/etc/modprobe.conf中加入一行

  1. options bnx2 disable_msi=1

创建一个脚本来重启服务
vi net.sh

  1. /etc/init.d/network stop
  2. rmmod bnx2
  3. modprobe bnx2
  4. /etc/init.d/network start

chmod a+x net.sh
./net.sh

参考:
http://www.pcwind.net/dell-r410.html
http://bbs.linuxtone.org/thread-3813-1-1.html