failover 故障检测、报警、自动切换 程序 monitor_nginx.sh 用于对web server (Nginx) 监测与故障处理

 

前言

前几天，群里有人说用shell程序实现failover,所以今天自己写一个shell程序 monitor_nginx.sh

failover,fault detection and switching ,故障转移与切换程序(shell)

一.程序功能

用于检测web server (Nginx)是否宕掉，如果是用短信报警(sendmail139邮箱报警)并自动故障处理，浮动改热备机IP，实现故障转移与自动切换

二.系统架构

如图所示

172.29.141.112 上部署 Nginx 与 fastcgi(对外开放)

172.29.141.113 上部署 Nginx 与 fastcgi (作为热备)

172.29.141.115 上部署 MySQL

三. 程序源码

cat monitor_nginx.sh

[c-sharp] view plain copy print ?

1. #!/bin/bash
2. #author longxibendi
3. #blog http://blog.youkuaiyun.com/longxibendi
4. #function failover the web server and send message to SA
7. # send email to longxibendi@139.com
8. function_sendEmail ()
9. {
11. /usr/local/bin/sendEmail -f monitor_sys@163.com -t longxibendi@139.com -s smtp.163.com -u "web sever down" -xu monitor_sys -xp 123456789 -m "` date ` "
14. }
17. # change ip address
18. function_change_ip ()
19. {
20. /sbin/ifconfig eth0 172.29.141.112 broadcast 172.29.141.255 netmask 255.255.255.0 up
21. /sbin/route add -host 172.29.141.112 dev eth0
22. /sbin/arping -I eth0 -c 3 -s 172.29.141.112 172.29.141.115
23. ### /sbin/arping is to add arp list in 172.29.141.115
24. }
28. # to start local nginx web server
29. function_start_nginx_server ()
30. {
31. ulimit -SHn 65535
32. /usr/local/webserver/php/sbin/php-fpm start
33. /usr/local/webserver/nginx/sbin/nginx
35. }
38. # main function
39. function_main ()
40. {
42. while true
43. do
44. http_status_code=`curl -o /dev/null -s -w %{http_code} http://172.29.141.112/wordpress/`
46. # echo "$http_status_code";
47. if [ ${http_status_code} != 200 ] ;then
48. function_sendEmail ;
49. function_change_ip ;
50. function_start_nginx_server ;
51. fi
54. # execute curl per 3 seconds
56. sleep 3
58. done
60. }
62. function_main ;

#!/bin/bash#author longxibendi#blog http://blog.youkuaiyun.com/longxibendi#function failover the web server and send message to SA # send email to longxibendi@139.comfunction_sendEmail (){ /usr/local/bin/sendEmail -f monitor_sys@163.com -t longxibendi@139.com -s smtp.163.com -u "web sever down" -xu monitor_sys -xp 123456789 -m "` date ` " } # change ip address function_change_ip (){ /sbin/ifconfig eth0 172.29.141.112 broadcast 172.29.141.255 netmask 255.255.255.0 up /sbin/route add -host 172.29.141.112 dev eth0 /sbin/arping -I eth0 -c 3 -s 172.29.141.112 172.29.141.115 ### /sbin/arping is to add arp list in 172.29.141.115 } # to start local nginx web server function_start_nginx_server (){ ulimit -SHn 65535 /usr/local/webserver/php/sbin/php-fpm start /usr/local/webserver/nginx/sbin/nginx } # main functionfunction_main (){ while truedo http_status_code=`curl -o /dev/null -s -w %{http_code} http://172.29.141.112/wordpress/` # echo "$http_status_code"; if [ ${http_status_code} != 200 ] ;then function_sendEmail ; function_change_ip ; function_start_nginx_server ; fi # execute curl per 3 seconds sleep 3 done } function_main ;

四.程序说明

function_sendEmail 发送email到139邮箱，报警

function_change_ip 浮动更改ip，添加路由，修改db的arp表

function_start_nginx_server 启动本机nginx服务器，这一步也可以调整，如果本机的nginx已启动的话

function_main 主函数，通过curl 网址得到状态码

基本思想有两条

<!--[if !supportLists]-->1．<!--[endif]-->通过curl检测故障

<!--[if !supportLists]-->2．<!--[endif]-->通过浮动修改(ifconfig) ip 实现故障切换(failover)

五.如何部署

在172.29.141.113上部署。如果在112上部署，不需要function_change_ip这个函数。这是一种简单的解决方案，也可以用keepalive，如果PV很大，可能需要用lvs+keepalive或者硬件负载均衡器。

六.参考资料

1. man curl 这里给出重要的参数

-w/--write-out <format>

Defines what to display on stdout after a completed and success-

ful operation. The format is a string that may contain plain

text mixed with any number of variables. The string can be spec-

ified as "string", to get read from a particular file you spec-

ify it "@filename" and to tell curl to read the format from

stdin you write "@-".

The variables present in the output format will be substituted

by the value or text that curl thinks fit, as described below.

All variables are specified like %{variable_name} and to output

a normal % you just write them like %%. You can output a newline

by using /n, a carriage return with /r and a tab space with /t.

NOTE: The %-letter is a special letter in the win32-environment,

where all occurrences of % must be doubled when using this

option.

Available variables are at this point:

url_effective The URL that was fetched last. This is mostly

meaningful if you’ve told curl to follow loca-

tion: headers.

http_code The numerical code that was found in the last

retrieved HTTP(S) page.

http_connect The numerical code that was found in the last

response (from a proxy) to a curl CONNECT

request. (Added in 7.12.4)

time_total The total time, in seconds, that the full opera-

tion lasted. The time will be displayed with mil-

lisecond resolution.

time_namelookup

The time, in seconds, it took from the start

until the name resolving was completed.

time_connect The time, in seconds, it took from the start

until the connect to the remote host (or proxy)

was completed.

time_pretransfer

The time, in seconds, it took from the start

until the file transfer is just about to begin.

This includes all pre-transfer commands and nego-

tiations that are specific to the particular pro-

tocol(s) involved.

time_redirect The time, in seconds, it took for all redirection

steps include name lookup, connect, pretransfer

and transfer before final transaction was

started. time_redirect shows the complete execu-

tion time for multiple redirections. (Added in

7.12.3)

time_starttransfer

The time, in seconds, it took from the start

until the first byte is just about to be trans-

ferred. This includes time_pretransfer and also

the time the server needs to calculate the

result.

size_download The total amount of bytes that were downloaded.

size_upload The total amount of bytes that were uploaded.

size_header The total amount of bytes of the downloaded head-

ers.

size_request The total amount of bytes that were sent in the

HTTP request.

speed_download The average download speed that curl measured for

the complete download.

speed_upload The average upload speed that curl measured for

the complete upload.

content_type The Content-Type of the requested document, if

there was any.

num_connects Number of new connects made in the recent trans-

fer. (Added in 7.12.3)

num_redirects Number of redirects that were followed in the

request. (Added in 7.12.3)

ftp_entry_path The initial path libcurl ended up in when logging

on to the remote FTP server. (Added in 7.15.4)

2.HTTP状态码参考

http://baike.baidu.com/view/1790469.htm