HAProxy+Keepalived的配置（二）（及重要问题说明）

HAProxy+Keepalived的配置（二）（及重要问题说明）

2017年03月12日 23:19:52

阅读数：3607

在搭建HAProxy+Keepalived之前先单独测试HAProxy

一 HAProxy配置测试

   

   （注测试：HAProxy只设内网网卡，Client同样位于内网）

   1.测试web提前配置

1. yum install httpd -y #安装httpd
2. echo 192.168.1.188 >/var/www/html/index.html #输入网站显示各自IP
3. systemctl restart httpd
4. firewall-cmd --permanent --add-port=80/tcp #开启防火墙80端口
5. firewall-cmd --reload
6. #测试，浏览器登录IP查看，是否正常启动。

   2.HAProxy配置/etc/haproxy/haproxy.cfg（yum install haproxy -y）

1. global
2. #全区日志配置 使用rsyslog的local3设备
3. log 127.0.0.1 local3 info
4. #工作目录（安全）
5. chroot /var/lib/haproxy
6. #pid文件存储目录
7. pidfile /var/run/haproxy.pid
8. #后台进程数量
9. nbproc 1
10. #每个进程最大并发数
11. maxconn 40000
12. user haproxy
13. group haproxy
14. #后台程序模式工作
15. daemon
17. defaults
18. mode http
19. #后端连接重试次数，超出标识不可用
20. retries 3
21. #连接服务器最长等待时间
22. timeout connect 10s
23. #客户端发送请求最长等待时间
24. timeout client 30s
25. #服务器会复客户端最长等待时间
26. timeout server 30s
27. #对后端服务器的检测超时时间
28. timeout check 10s
30. #定义HAProxy监控页面
31. listen admin_stats
32. bind 0.0.0.0:9188
33. mode http
34. log 127.0.0.1 local3 err
35. #HAProxy监控页面统计自动刷新时间。
36. stats refresh 30s
37. #设置监控页面URL路径。 http：//IP:9188/haproxy-status可查看
38. stats uri /haproxy-status
39. #统计页面密码框提示信息
40. stats realm welcome login\ Haproxy
41. #登录统计页面用户和密码
42. stats auth admin:123456
43. #隐藏HAProxy版本信息
44. stats hide-version
45. #设置TURE后可在监控页面手工启动关闭后端真实服务器
46. stats admin if TRUE
48. #定义前端虚拟节点
49. frontend www
50. #监听端口
51. bind *:80
52. mode http
53. #启用日志记录HTTP请求。
54. option httplog
55. #启用后后端服务器可以获得客户端IP
56. option forwardfor
57. #客户端和服务器完成一次连接请求后，HAProxy主动关闭TCP链接（优化选项）
58. option httpclose
59. #使用全局日志配置
60. log global
61. #指定后端服务池（backend定义htmpool）
62. default_backend htmpool
64. #定义后端真实服务器
65. backend htmpool
66. mode http
67. #用于cookie保持环境。（如后端服务器故障，客户端cookie不会刷新，用此来把用户请求强制定向到正常服务器）
68. option redispatch
69. #负载均衡很高时，自动结束当前队列处理时间长的连接
70. option abortonclose
71. #负载均衡算法
72. balance roundrobin
73. #允许向cookie插入SERVERID.下面server可以使用cookie定义
74. cookie SERVERID
75. #启用HTTP服务状态检测功能 （后端服务器一定要存在此文件，不然haproxy认为其故障）
76. option httpchk GET /index.html
77. #后端服务设置
78. server web1 192.168.1.186:80 cookie server1 weight 6 check inter 2000 rise 2 fall 3
79. server web2 192.168.1.188:80 cookie server2 weight 6 check inter 2000 rise 2 fall 3

    开启日志（日志如果在主机性能较弱的情况下，最好关闭）

1. vim /etc/rsyslog.d/haproxy.conf
3. $ModLoad imudp
4. $UDPServerRun 514
5. local3.* /var/log/haproxy
7. systemctl restart rsyslog

    开启防火墙和安全策略

1. firewall-cmd --permanent --add-port=80/tcp
2. firewall-cmd --permanent --add-port=9188/tcp
3. firewall-cmd --reload
5. setsebool -P haproxy_connect_any=on
6. #getsebool -a | grep haproxy 查看

   启动HAProxy服务

1. systemctl start haproxy

    测试均衡能力

1. [root@test2 ~]# curl http://192.168.1.166
2. 192.168.1.188
3. [root@test2 ~]# curl http://192.168.1.166
4. 192.168.1.186
5. [root@test2 ~]# curl http://192.168.1.166
6. 192.168.1.188
7. [root@test2 ~]# curl http://192.168.1.166
8. 192.168.1.186

    测试HAProxy监控页面是否正常（http：//IP:9188/haproxy-status）

二 HAProxy+Keepalived配置 

   HAProxy配置

   (HAProxy两台Proxy主机配置相同，如上HAProxy的配置即可)

   Keepalived配置（/etc/keepalived/keepalived.conf）(yum install keepalived -y)

1. ! Configuration File for keepalived
2. global_defs {
3. #设置报警通知邮件地址，可以设置多个
4. notification_email {
5. msun1996@163.com
6. }
7. #设置邮件的发送地址
8. notification_email_from keepalived@msun.com
9. #设置smtp server的地址,该地址必须是存在的
10. smtp_server 127.0.0.1
11. #设置连接smtp server的超时时间
12. smtp_connect_timeout 30
13. #运行Keepalived服务器的标识，发邮件时显示在邮件标题中的信息
14. router_id HAProxy_msun
15. }
16. # 检测haproxy脚本
17. vrrp_script chk_haproxy {
18. script "/etc/keepalived/check_haproxy.sh"
19. interval 2
20. #下面方法相对更优
21. #script "killall -0 haproxy" #killall (安装 yum install psmisc -y)
22. #interval 2
23. #weghit 2 #权值脚本成功时（0）等于priority+weghit #否则为priority
24. }
25. #定义VRRP实例,实例名自定义
26. vrrp_instance haproxy_msun {
27. #指定Keepalived的角色，MASTER为主服务器，BACKUP为备用服务器
28. state MASTER #从设置为BACKUP
29. #指定HA监测的接口
30. interface eno16777736
31. #虚拟路由标识，这个标识是一个数字(1-255)，在一个VRRP实例中主备服务器ID必须一样
32. virtual_router_id 68
33. #优先级，数字越大优先级越高，在一个实例中主服务>器优先级要高于备服务器
34. priority 100 #从设置为99
35. #设置主备之间同步检查的时间间隔单位秒
36. advert_int 1
37. #设置验证类型和密码
38. authentication {
39. #验证类型有两种{PASS|HA}
40. auth_type PASS
41. #设置验证密码，在一个实例中主备密码保持一样
42. auth_pass 1689
43. }
44. track_script {
45. chk_haproxy # 执行监控的服务
46. }
47. #定义虚拟IP地址,可以有多个，每行一个
48. virtual_ipaddress {
49. 192.168.1.160
50. }
51. }

   添加HAProxy检测脚本到vim /etc/keepalived/check_haproxy.sh(执行权限)（这一步必须关闭SElinux,不然vrrp script 脚本是不可以执行的。下面会介绍开启selinux下的解决方法）

1. #!/bin/bash
2. #判断haproxy是否已经启动
3. if [ `ps -C haproxy --no-header |wc -l` -eq 0 ] ; then
4. #如果没有启动，则启动haproxy程序
5. systemctl start haproxy
6. #睡眠3秒钟以等待haproxy完全启动
7. sleep 3
8. if [ `ps -C haproxy --no-header |wc -l` -eq 0 ] ; then
9. systemctl stop keepalived
10. #如果haproxy还是没有启动，则将keepalived停掉，这样VIP会自动漂移到另外一台haproxy
11. fi
12. fi

   启动keepalived

1. systemctl start keepalived

   防火墙允许（开放vrrp组播）（不然会导致脑裂）

1. firewall-cmd --permanent --add-rich='rule family="ipv4" destination address="224.0.0.18" protocol value="vrrp" accept'
2. firewall-cmd --reload

   测试

1. [root@test5 ~]# curl 192.168.1.160
2. 192.168.1.186
3. [root@test5 ~]# curl 192.168.1.160
4. 192.168.1.188
5. [root@test5 ~]# curl 192.168.1.160
6. 192.168.1.186
7. [root@test5 ~]# curl 192.168.1.160
8. 192.168.1.188

   通过观察ip a输出VIP来判断是否存在脑裂，和主备故障时是否能正常切换。

三 部署HAProxy遇到的重点问题与解决

   1.部署单独的Haproxy时遇到问题

   配置完配置文件后，使用systemctl start haproxy,haproxy无法正常启动。（启动后查看状态没有对应进程显示）

  （防火墙80端口和9188端口已正常开启。所以不可能是防火墙问题，再说防火墙也不可能影响启动啊）

  /var/log/message有以下报错信息

1. [root@test5 ~]# tail /var/log/messages
2. Mar 12 14:59:29 test5 systemd: Starting HAProxy Load Balancer...
3. Mar 12 14:59:29 test5 haproxy-systemd-wrapper: [ALERT] 070/145929 (3131) : Starting proxy admin_stats: cannot bind socket [0.0.0.0:9188]
4. Mar 12 14:59:29 test5 haproxy-systemd-wrapper: haproxy-systemd-wrapper: exit, haproxy RC=1
5. Mar 12 14:59:29 test5 systemd: haproxy.service: main process exited, code=exited, status=1/FAILURE
6. Mar 12 14:59:29 test5 systemd: Unit haproxy.service entered failed state.
7. Mar 12 14:59:29 test5 systemd: haproxy.service failed.

   查看haproxy相关selinux政策，发现haproxy默认不允许连接

1. [root@test5 ~]# getsebool -a | grep haproxy
2. haproxy_connect_any --> off

   重新设置selinux政策后正常启动

1. [root@test5 ~]# setsebool -P haproxy_connect_any=on
2. [root@test5 ~]# systemctl start haproxy
3. [root@test5 ~]# netstat -tlunp
4. Active Internet connections (only servers)
5. Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
6. tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3171/haproxy
7. tcp 0 0 0.0.0.0:9188 0.0.0.0:* LISTEN 3171/haproxy

   2.搭建HAProxy+Keepalived遇到的问题

       1.服务器搭建完成后，出现脑裂现象。主备机多有VIP。

         解决：防火墙允许（开放vrrp组播）

1. firewall-cmd --permanent --add-rich='rule family="ipv4" destination address="224.0.0.18" protocol value="vrrp" accept'

       2.关于Keepalived中的vrrp script 设置的脚本不执行。

       因为检测HAproxy状态的脚本不能运行，导致就算主机HAProxy挂掉，但因为Keepalived还正常运行，VIP没有正常从主机正常漂移到从机，导致整个服务垮掉。

       直接快速解决方法：关闭SElinux.

       

       SElinux安全策略运行时解决

       1)尝试SElinux允许解决（解决失败）

       开启SElinux时的错误提示：

1. [root@test1 ~]# tail -f /var/log/messages
2. type=AVC msg=audit(1489338470.513:714): avc: denied { getattr } for pid=5174 comm="check_haproxy.s" path="/usr/bin/systemctl" dev="dm-0" ino=33947874 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
3. type=SYSCALL msg=audit(1489338470.513:714): arch=c000003e syscall=4 success=no exit=-13 a0=f72c00 a1=7ffd88bca560 a2=7ffd88bca560 a3=3 items=0 ppid=5173 pid=5174 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)

       安装selinux管理工具（根据报错显示可能解决方案）

1. yum install setroubleshoot -y

       查看settoubleshoot给出的解决方案  

1. [root@test1 ~]# tail -f /var/log/messages
2. Mar 13 02:30:27 test1 setroubleshoot: SELinux is preventing /usr/bin/bash from getattr access on the file /usr/bin/systemctl. For complete SELinux messages. run sealert -l 54416ee0-01c3-40e8-8198-675f6f86a7f7
3. Mar 13 02:30:27 test1 python: SELinux is preventing /usr/bin/bash from getattr access on the file /usr/bin/systemctl.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that bash should be allowed getattr access on the systemctl file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys#012# semodule -i my-checkhaproxys.pp#012

       尝试解决：

1. [root@test1 ~]# sealert -l 54416ee0-01c3-40e8-8198-675f6f86a7f7
2. SELinux is preventing /usr/bin/bash from getattr access on the file /usr/bin/systemctl.
4. ***** Plugin catchall (100. confidence) suggests **************************
6. If 确定应默认允许 bash getattr 访问 systemctl file。
7. Then 应该将这个情况作为 bug 报告。
8. 可以生成本地策略模块以允许此访问。
9. Do
10. allow this access for now by executing:
11. # ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys
12. # semodule -i my-checkhaproxys.pp        

1. [root@test1 ~]# sealert -l 54416ee0-01c3-40e8-8198-675f6f86a7f7
2. [root@test1 ~]# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys
3. [root@test1 ~]# semodule -i my-checkhaproxys.pp

        尝试后日志提示：

1. [root@test1 ~]# tail -f /var/log/messages
2. Mar 13 02:36:08 test1 setroubleshoot: SELinux is preventing /usr/bin/bash from open access on the file /usr/bin/systemctl. For complete SELinux messages. run sealert -l 487ebbb3-fefe-4018-8c4c-5be6a185e64b
3. Mar 13 02:36:08 test1 python: SELinux is preventing /usr/bin/bash from open access on the file /usr/bin/systemctl.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that bash should be allowed open access on the systemctl file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys#012# semodule -i my-checkhaproxys.pp#012

1. [root@test1 ~]# tail -f /var/log/audit/audit.log
2. type=AVC msg=audit(1489352184.678:549): avc: denied { open } for pid=3990 comm="check_haproxy.s" path="/usr/bin/systemctl" dev="dm-0" ino=33724848 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
3. type=SYSCALL msg=audit(1489352184.678:549): arch=c000003e syscall=2 success=no exit=-13 a0=14bec50 a1=0 a2=43 a3=7ffe13e19190 items=0 ppid=3986 pid=3990 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)

        继续按提示输入

1. [root@test1 ~]# sealert -l 487ebbb3-fefe-4018-8c4c-
2. [root@test1 ~]# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys
3. [root@test1 ~]# semodule -i my-checkhaproxys.pp

        尝试后日志提示：

1. [root@test1 ~]# tail -f /var/log/messages
2. Mar 13 05:02:35 test1 setroubleshoot: SELinux is preventing /usr/bin/bash from execute_no_trans access on the file /usr/bin/systemctl. For complete SELinux messages. run sealert -l a3a942ad-2b0e-4b4b-bf1f-b521256f4405
3. Mar 13 05:02:35 test1 python: SELinux is preventing /usr/bin/bash from execute_no_trans access on the file /usr/bin/systemctl.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that bash should be allowed execute_no_trans access on the systemctl file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys#012# semodule -i my-checkhaproxys.pp#012

1. [root@test1 ~]# tail -f /var/log/audit/audit.log
2. type=AVC msg=audit(1489352298.756:764): avc: denied { execute_no_trans } for pid=5507 comm="check_haproxy.s" path="/usr/bin/systemctl" dev="dm-0" ino=33724848 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
3. type=SYSCALL msg=audit(1489352298.756:764): arch=c000003e syscall=59 success=no exit=-13 a0=162cc50 a1=162ccc0 a2=162d360 a3=7ffddf4d6190 items=0 ppid=5503 pid=5507 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)

        继续按提示输入

1. [root@test1 ~]# sealert -l a3a942ad-2b0e-4b4b-bf1f-b521256f4405
2. [root@test1 ~]# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys
3. [root@test1 ~]# semodule -i my-checkhaproxys.pp

        尝试后日志提示：

1. [root@test1 ~]# tail -f /var/log/messages
2. Mar 13 05:06:42 test1 setroubleshoot: failed to retrieve rpm info for /run/dbus/system_bus_socket
3. Mar 13 05:06:42 test1 setroubleshoot: SELinux is preventing /usr/bin/systemctl from connectto access on the unix_stream_socket /run/dbus/system_bus_socket. For complete SELinux messages. run sealert -l e1afcda9-a674-4d76-8aa0-7787404c515e
4. Mar 13 05:06:42 test1 python: SELinux is preventing /usr/bin/systemctl from connectto access on the unix_stream_socket /run/dbus/system_bus_socket.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that systemctl should be allowed connectto access on the system_bus_socket unix_stream_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'systemctl' --raw | audit2allow -M my-systemctl#012# semodule -i my-systemctl.pp#012

1. [root@test1 ~]# tail -f /var/log/audit/audit.log
2. type=AVC msg=audit(1489352885.238:1505): avc: denied { connectto } for pid=11711 comm="systemctl" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket
3. type=SYSCALL msg=audit(1489352885.238:1505): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=7f42bfa16918 a2=21 a3=7ffe723af2b0 items=0 ppid=11707 pid=11711 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemctl" exe="/usr/bin/systemctl" subj=system_u:system_r:keepalived_t:s0 key=(null)

        继续按提示输入

1. [root@test1 ~]# sealert -l e1afcda9-a674-4d76-8aa0-7787404c515e
2. [root@test1 ~]# ausearch -c 'systemctl' --raw | audit2allow -M my-systemctl
3. [root@test1 ~]# semodule -i my-systemctl.pp

        尝试后日志：（最后没解决  ~。。~）

1. [root@test1 ~]# tail -f /var/log/messages
2. Mar 13 05:12:09 test1 Keepalived_vrrp[3090]: Process [14395] didn't respond to SIGTERM

1. [root@test1 ~]# tail -f /var/log/audit/audit.log
2. type=USER_AVC msg=audit(1489353197.502:2058): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { start } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/haproxy.service" cmdline="systemctl start haproxy" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:haproxy_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
3. type=USER_AVC msg=audit(1489353198.557:2059): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { stop } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/keepalived.service" cmdline="systemctl stop keepalived" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:keepalived_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
4. type=USER_AVC msg=audit(1489353198.557:2060): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { status } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/keepalived.service" cmdline="systemctl stop keepalived" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:keepalived_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'

1. type=AVC msg=audit(1489354986.919:1591): avc: denied { getattr } for pid=7566 comm="check_haproxy.s" path="/usr/sbin/haproxy" dev="dm-0" ino=68172487 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:haproxy_exec_t:s0 tclass=file
2. type=SYSCALL msg=audit(1489354986.919:1591): arch=c000003e syscall=4 success=no exit=-13 a0=19cbd50 a1=7ffde08b8720 a2=7ffde08b8720 a3=11 items=0 ppid=7565 pid=7566 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)

       观察：systemctl 相关命令还是因为selinux运行不了

       

       2）根据1）决定对vrrp script脚本修改如下

1. #!/bin/bash
2. #判断haproxy是否已经启动，去除haproxy试图启动部分
3. if [ `ps -C haproxy --no-header |wc -l` -eq 0 ] ; then
4. #如果haproxy还是没有启动，则将keepalived停掉，这样VIP会自动漂移到另外一台haproxy
5. pkill keepalived
6. fi

       （启动顺序必须是haproxy先启动，负责keepalived如果先启动会自杀）

       也可设置自启

1. systemctl enable haproxy
2. systemctl enable keepalived

   

      3）使用文件中注释的信息完成 vrrp_script （下面的脚本对主从服务器的priority差值有严格要求）(最优方法)

1. # 检测haproxy脚本
2. vrrp_script chk_haproxy{
3.    script "killall -0 haproxy" #killall (安装 yum install psmisc -y)

4.    interval 2

5.    weghit 2 #权值脚本成功时（0）等于priority+weghit #否则为priority

6. }