netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}'
TIME_WAIT 79
ESTABLISHED 6
LISTEN 3
可以使用man netstat查看TCP的各种状态信息描述
ESTABLISHED socket已经建立连接
CLOSED socket没有被使用,无连接
CLOSING 服务器端和客户端都同时关闭连接
CLOSE_WAIT 等待关闭连接
TIME_WAIT 表示收到了对方的FIN报文,并发送出了ACK报文,等待2MSL后就可回到CLOSED状态
LAST_ACK 远端关闭,当前socket被动关闭后发送FIN报文,等待对方ACK报文
LISTEN 监听状态
SYN_RECV 接收到SYN报文
SYN_SENT 已经发送SYN报文
FIN_WAIT1 The socket is closed, and the connection is shutting down
FIN_WAIT2 Connection is closed, and the socket is waiting for a shutdown from the remote end.
2.在需要被监控的zabbix-agent端添加脚本编写
创建文件夹
mkdir -p /usr/local/zabbix-agent/scripts/
mkdir -p /etc/zabbix/zabbix_agentd.d/
#!/bin/bash
#this script is used to get tcp and udp connetion status
#tcp status
metric=$1
tmp_file=/tmp/tcp_status.txt
/bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' > $tmp_file
case $metric in
closed)
output=$(awk '/CLOSED/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
listen)
output=$(awk '/LISTEN/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
synrecv)
output=$(awk '/SYN_RECV/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
synsent)
output=$(awk '/SYN_SENT/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
established)
output=$(awk '/ESTABLISHED/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
timewait)
output=$(awk '/TIME_WAIT/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
closing)
output=$(awk '/CLOSING/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
closewait)
output=$(awk '/CLOSE_WAIT/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
lastack)
output=$(awk '/LAST_ACK/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
finwait1)
output=$(awk '/FIN_WAIT1/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
finwait2)
output=$(awk '/FIN_WAIT2/{print $2}' $tmp_file)
if [ "$output" == "" ];then
echo 0
else
echo $output
fi
;;
*)
echo -e "\e[033mUsage: sh $0 [closed|closing|closewait|synrecv|synsent|finwait1|finwait2|listen|established|lastack|timewait]\e[0m"
esac
赋予脚本执行权限
chmod o+x /usr/local/zabbix-agent/scripts/tcp_conn_status.sh
3.agent的配置文件 /etc/zabbix/zabbix_agentd.conf 中定义了其他key的包含目录 Include=/etc/zabbix/zabbix_agentd.d/, 接着在 /etc/zabbix/zabbix_agentd.d/ 目录新建一个文件 tcp-status-params.conf, 内容如下
vim /etc/zabbix/zabbix_agentd.d/tcp-status-params.conf
UserParameter=tcp.status[*],/usr/local/zabbix-agent/scripts/tcp_conn_status.sh $1重启agent
service zabbix-agent restart
4.zabbix-master服务端测试
zabbix_get -s 192.168.3.86 -p 10050 -k "tcp.status[listen]"
13
5.zabbix web端配置:
登录Zabbix3.0 的web界面,一次选择 Configuration > Templates , 在主界面的右上角有个 Import 按钮,用来导入模板
zabbix-tcp-status.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <zabbix_export>
- <version>2.0</version>
- <date>2014-12-04T09:41:57Z</date>
- <groups>
- <group>
- <name>Templates</name>
- </group>
- </groups>
- <templates>
- <template>
- <template>Template TCP Connection Status</template>
- <name>Template TCP Connection Status</name>
- <groups>
- <group>
- <name>Templates</name>
- </group>
- </groups>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <items>
- <item>
- <name>CLOSED</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[closed]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>CLOSE_WAIT</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[closewait]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>CLOSING</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[closing]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>ESTABLISHED</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[established]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>FIN_WAIT1</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[finwait1]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>FIN_WAIT2</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[finwait2]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>LAST_ACK</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[lastack]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>LISTEN</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[listen]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>SYN_RECV</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[synrecv]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>SYN_SENT</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[synsent]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- <item>
- <name>TIME_WAIT</name>
- <type>0</type>
- <snmp_community/>
- <multiplier>0</multiplier>
- <snmp_oid/>
- <key>tcp.status[timewait]</key>
- <delay>60</delay>
- <history>90</history>
- <trends>365</trends>
- <status>0</status>
- <value_type>3</value_type>
- <allowed_hosts/>
- <units/>
- <delta>0</delta>
- <snmpv3_contextname/>
- <snmpv3_securityname/>
- <snmpv3_securitylevel>0</snmpv3_securitylevel>
- <snmpv3_authprotocol>0</snmpv3_authprotocol>
- <snmpv3_authpassphrase/>
- <snmpv3_privprotocol>0</snmpv3_privprotocol>
- <snmpv3_privpassphrase/>
- <formula>1</formula>
- <delay_flex/>
- <params/>
- <ipmi_sensor/>
- <data_type>0</data_type>
- <authtype>0</authtype>
- <username/>
- <password/>
- <publickey/>
- <privatekey/>
- <port/>
- <description/>
- <inventory_link>0</inventory_link>
- <applications>
- <application>
- <name>TCP Status</name>
- </application>
- </applications>
- <valuemap/>
- </item>
- </items>
- <discovery_rules/>
- <macros/>
- <templates/>
- <screens/>
- </template>
- </templates>
- <triggers>
- <trigger>
- <expression>{Template TCP Connection Status:tcp.status[timewait].last()}>10000</expression>
- <name>There are too many TCP TIME_WAIT status</name>
- <url/>
- <status>0</status>
- <priority>4</priority>
- <description/>
- <type>0</type>
- <dependencies/>
- </trigger>
- </triggers>
- <graphs>
- <graph>
- <name>TCP Status</name>
- <width>900</width>
- <height>200</height>
- <yaxismin>0.0000</yaxismin>
- <yaxismax>100.0000</yaxismax>
- <show_work_period>1</show_work_period>
- <show_triggers>1</show_triggers>
- <type>0</type>
- <show_legend>1</show_legend>
- <show_3d>0</show_3d>
- <percent_left>0.0000</percent_left>
- <percent_right>0.0000</percent_right>
- <ymin_type_1>0</ymin_type_1>
- <ymax_type_1>0</ymax_type_1>
- <ymin_item_1>0</ymin_item_1>
- <ymax_item_1>0</ymax_item_1>
- <graph_items>
- <graph_item>
- <sortorder>0</sortorder>
- <drawtype>0</drawtype>
- <color>C80000</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[closed]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>1</sortorder>
- <drawtype>0</drawtype>
- <color>00C800</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[closewait]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>2</sortorder>
- <drawtype>0</drawtype>
- <color>0000C8</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[closing]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>3</sortorder>
- <drawtype>0</drawtype>
- <color>C800C8</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[established]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>4</sortorder>
- <drawtype>0</drawtype>
- <color>00C8C8</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[finwait1]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>5</sortorder>
- <drawtype>0</drawtype>
- <color>C8C800</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[finwait2]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>6</sortorder>
- <drawtype>0</drawtype>
- <color>C8C8C8</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[lastack]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>7</sortorder>
- <drawtype>0</drawtype>
- <color>960000</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[listen]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>8</sortorder>
- <drawtype>0</drawtype>
- <color>009600</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[synrecv]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>9</sortorder>
- <drawtype>0</drawtype>
- <color>000096</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[synsent]</key>
- </item>
- </graph_item>
- <graph_item>
- <sortorder>10</sortorder>
- <drawtype>0</drawtype>
- <color>960096</color>
- <yaxisside>0</yaxisside>
- <calc_fnc>2</calc_fnc>
- <type>0</type>
- <item>
- <host>Template TCP Connection Status</host>
- <key>tcp.status[timewait]</key>
- </item>
- </graph_item>
- </graph_items>
- </graph>
- </graphs>
- </zabbix_export>
在对应主机上添加tcp的监控:
添加报警,当tcp连接数超过5W报警:
{Template TCP Connection Status:tcp.status[established].last()}>50000
最终效果图:
某些服务器的established连接很多(3W或更多),如果突然下降到一定的值(1000),这个可能也是问题(可能某个前端的服务出问题了,新的用户进不来),需要对这个值进行监控
{192.168.1.13:tcp_est_status.last()}<1000
后记:发现通过netstat监控服务器的tcp等连接数效率比较低,netstat统计占用大量cpu带来服务器额外的压力,通过ss命令会更加合适,详情请看:
http://blog.youkuaiyun.com/reblue520/article/details/52908966