本文详细说明了如何使用HAProxy和Keepalived为MariaDB Galera集群实现高可用性和负载均衡，通过设置HAProxy作为负载均衡器和Keepalived作为状态检测和故障转移组件，确保集群在节点故障时能够自动切换，并且能够分离读写操作以提高性能。
Summary
This blog post explains how to setup HAProxy and Keepalived for use with for example the MariaDB Galera cluster setup described here.
The combination of HAProxy and Keepalived will make the MariaDB Galera cluster more resilient and high-available by adding load balancing and transparent failover when a MariaDB cluster node goes down. HAProxy also allows you to separate read and write actions to the MariaDB Galera cluster which is very useful if you want to loadbalance read actions across all MariaDB nodes but want to write to only one MariaDB node.
Since only one of anything is a Single Point of Failure (SPoF) I’ll be using two HAProxy nodes in a high-availability setup. Should the primary HAProxy node fail then Keepalived will initiate an automatic and transparent failover to the secondary HAProxy node.
Requirements
This tutorial requires the MariaDB Galera cluster setup mentioned above. You will also need 2 nodes (VMs or bare metal) called haproxy1 and haproxy2 each with one IP address. In addition you will also need a third IP address that will be used as the floating Virtual IP (VIP) address between the two HAProxy nodes. The HAProxy nodes don’t need a lot of power, memory or storage. One CPU with 512MB to 1GB memory and an 8GB disk are fine.
Let’s get started.
Install HAProxy and Keepalived on both HAProxy nodes
On haproxy1 install HAProxy and Keepalived.
[root@haproxy1 ~]# yum -y install haproxy keepalived Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package haproxy.x86_64 0:1.4.24-2.el6 will be installed ---> Package keepalived.x86_64 0:1.2.7-3.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: haproxy x86_64 1.4.24-2.el6 base-local 457 k keepalived x86_64 1.2.7-3.el6 base-local 174 k Transaction Summary ================================================================================ Install 2 Package(s) Total download size: 632 k Installed size: 2.0 M Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 632 k (1/2): haproxy-1.4.24-2.el6.x86_64.rpm | 457 kB 00:00 (2/2): keepalived-1.2.7-3.el6.x86_64.rpm | 174 kB 00:00 -------------------------------------------------------------------------------- Total 5.6 MB/s | 632 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : keepalived-1.2.7-3.el6.x86_64 1/2 Installing : haproxy-1.4.24-2.el6.x86_64 2/2 Verifying : haproxy-1.4.24-2.el6.x86_64 1/2 Verifying : keepalived-1.2.7-3.el6.x86_64 2/2 Installed: haproxy.x86_64 0:1.4.24-2.el6 keepalived.x86_64 0:1.2.7-3.el6 Complete!
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | [root@haproxy1 ~]# yum -y install haproxy keepalived Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package haproxy.x86_64 0:1.4.24-2.el6 will be installed ---> Package keepalived.x86_64 0:1.2.7-3.el6 will be installed --> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package Arch Version Repository Size ================================================================================ Installing: haproxy x86_64 1.4.24-2.el6 base-local 457 k keepalived x86_64 1.2.7-3.el6 base-local 174 k
Transaction Summary ================================================================================ Install 2 Package(s)
Total download size: 632 k Installed size: 2.0 M Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 632 k (1/2): haproxy-1.4.24-2.el6.x86_64.rpm | 457 kB 00:00 (2/2): keepalived-1.2.7-3.el6.x86_64.rpm | 174 kB 00:00 -------------------------------------------------------------------------------- Total 5.6 MB/s | 632 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : keepalived-1.2.7-3.el6.x86_64 1/2 Installing : haproxy-1.4.24-2.el6.x86_64 2/2 Verifying : haproxy-1.4.24-2.el6.x86_64 1/2 Verifying : keepalived-1.2.7-3.el6.x86_64 2/2
Installed: haproxy.x86_64 0:1.4.24-2.el6 keepalived.x86_64 0:1.2.7-3.el6
Complete! |
Now make sure both services start at boot:
[root@haproxy1 ~]# chkconfig haproxy on [root@haproxy1 ~]# chkconfig keepalived on
| 1 2 | [root@haproxy1 ~]# chkconfig haproxy on [root@haproxy1 ~]# chkconfig keepalived on |
Check if both services are properly activated:
[root@haproxy1 ~]# chkconfig | egrep 'haproxy|keepalived' haproxy 0:off 1:off 2:on 3:on 4:on 5:on 6:off keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off
| 1 2 3 | [root@haproxy1 ~]# chkconfig | egrep 'haproxy|keepalived' haproxy 0:off 1:off 2:on 3:on 4:on 5:on 6:off keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off |
That looks good so on to the next node.
Do the same on haproxy2:
[root@haproxy2 ~]# yum -y install haproxy keepalived Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package haproxy.x86_64 0:1.4.24-2.el6 will be installed ---> Package keepalived.x86_64 0:1.2.7-3.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: haproxy x86_64 1.4.24-2.el6 base-local 457 k keepalived x86_64 1.2.7-3.el6 base-local 174 k Transaction Summary ================================================================================ Install 2 Package(s) Total download size: 632 k Installed size: 2.0 M Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 632 k (1/2): haproxy-1.4.24-2.el6.x86_64.rpm | 457 kB 00:00 (2/2): keepalived-1.2.7-3.el6.x86_64.rpm | 174 kB 00:00 -------------------------------------------------------------------------------- Total 5.6 MB/s | 632 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : keepalived-1.2.7-3.el6.x86_64 1/2 Installing : haproxy-1.4.24-2.el6.x86_64 2/2 Verifying : haproxy-1.4.24-2.el6.x86_64 1/2 Verifying : keepalived-1.2.7-3.el6.x86_64 2/2 Installed: haproxy.x86_64 0:1.4.24-2.el6 keepalived.x86_64 0:1.2.7-3.el6 Complete!
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | [root@haproxy2 ~]# yum -y install haproxy keepalived Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package haproxy.x86_64 0:1.4.24-2.el6 will be installed ---> Package keepalived.x86_64 0:1.2.7-3.el6 will be installed --> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package Arch Version Repository Size ================================================================================ Installing: haproxy x86_64 1.4.24-2.el6 base-local 457 k keepalived x86_64 1.2.7-3.el6 base-local 174 k
Transaction Summary ================================================================================ Install 2 Package(s)
Total download size: 632 k Installed size: 2.0 M Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 632 k (1/2): haproxy-1.4.24-2.el6.x86_64.rpm | 457 kB 00:00 (2/2): keepalived-1.2.7-3.el6.x86_64.rpm | 174 kB 00:00 -------------------------------------------------------------------------------- Total 5.6 MB/s | 632 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : keepalived-1.2.7-3.el6.x86_64 1/2 Installing : haproxy-1.4.24-2.el6.x86_64 2/2 Verifying : haproxy-1.4.24-2.el6.x86_64 1/2 Verifying : keepalived-1.2.7-3.el6.x86_64 2/2
Installed: haproxy.x86_64 0:1.4.24-2.el6 keepalived.x86_64 0:1.2.7-3.el6
Complete! |
Now make sure both services start at boot:
[root@haproxy2 ~]# chkconfig haproxy on [root@haproxy2 ~]# chkconfig keepalived on
| 1 2 | [root@haproxy2 ~]# chkconfig haproxy on [root@haproxy2 ~]# chkconfig keepalived on |
Check if both services are properly activated:
[root@haproxy2 ~]# chkconfig | egrep 'haproxy|keepalived' haproxy 0:off 1:off 2:on 3:on 4:on 5:on 6:off keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off
| 1 2 3 | [root@haproxy2 ~]# chkconfig | egrep 'haproxy|keepalived' haproxy 0:off 1:off 2:on 3:on 4:on 5:on 6:off keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off |
That looks good so on to the next step.
Allow non-local Virtual IPs on all HAProxy nodes
Make binding to non-local Virtual IPs on all nodes:
[root@haproxy1 ~]# vi /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1 [root@haproxy1 ~]# sysctl -p
| 1 2 3 4 | [root@haproxy1 ~]# vi /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1
[root@haproxy1 ~]# sysctl -p |
[root@haproxy2 ~]# vi /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1 [root@haproxy2 ~]# sysctl -p
| 1 2 3 4 | [root@haproxy2 ~]# vi /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1
[root@haproxy2 ~]# sysctl -p |
Add HAProxy user to the MariaDB database
Make sure a HAProxy user exists on all MAriaDB nodes so HAProxy can access all the MariaDB nodes to check if they are still up. I’m using the user ‘haproxy’ and use the IPs of all the ethernet interfaces on the two HAProxy nodes (not the VIP address):
[patrick@db1 ~]$ mysql -u root -p Enter password: MariaDB [(none)]> CREATE USER 'haproxy'@'10.0.0.15'; MariaDB [(none)]> CREATE USER 'haproxy'@'10.0.0.17'; MariaDB [(none)]> FLUSH PRIVILEGES; MariaDB [(none)]> exit
| 1 2 3 4 5 6 | [patrick@db1 ~]$ mysql -u root -p Enter password: MariaDB [(none)]> CREATE USER 'haproxy'@'10.0.0.15'; MariaDB [(none)]> CREATE USER 'haproxy'@'10.0.0.17'; MariaDB [(none)]> FLUSH PRIVILEGES; MariaDB [(none)]> exit |
Since this information is automatically replicated to db2 and db3 we only
have to do this on db1. You can check if the information was replicated
with:
[patrick@db2 ~]$ mysql -u root -p -e "select * from mysql.user" | grep haproxy | cut -d$'\t' -f1,2 Enter password: 10.0.0.15 haproxy 10.0.0.17 haproxy
| 1 2 3 4 | [patrick@db2 ~]$ mysql -u root -p -e "select * from mysql.user" | grep haproxy | cut -d$'\t' -f1,2 Enter password: 10.0.0.15 haproxy 10.0.0.17 haproxy |
That looks good. On to the next step.
Setup rsyslog logging for HAProxy
By default HAProxy does not do any logging which obviously needs fixing. To enable HAProxy to log to rsyslog we need to add a rsyslog config file on both HAProxy nodes.
On node haproxy1:
[root@haproxy1 ~]# cat >> /etc/rsyslog.d/haproxy.conf << EOF $ModLoad imudp $UDPServerRun 514 $template Haproxy,"%msg%\n" local0.=info -/var/log/haproxy.log;Haproxy local0.notice -/var/log/haproxy-status.log;Haproxy local0.* ~ EOF
| 1 2 3 4 5 6 7 8 | [root@haproxy1 ~]# cat >> /etc/rsyslog.d/haproxy.conf << EOF $ModLoad imudp $UDPServerRun 514 $template Haproxy,"%msg%\n" local0.=info -/var/log/haproxy.log;Haproxy local0.notice -/var/log/haproxy-status.log;Haproxy local0.* ~ EOF |
Make sure the SELinux label is ok:
[root@haproxy1 ~]# /sbin/restorecon -v -F /etc/rsyslog.d/haproxy.conf
| 1 | [root@haproxy1 ~]# /sbin/restorecon -v -F /etc/rsyslog.d/haproxy.conf |
And restart the rsyslog service:
[root@haproxy1 ~]# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ]
| 1 2 3 | [root@haproxy1 ~]# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ] |
On node haproxy2:
[root@haproxy2 ~]# cat >> /etc/rsyslog.d/haproxy.conf << EOF $ModLoad imudp $UDPServerRun 514 $template Haproxy,"%msg%\n" local0.=info -/var/log/haproxy.log;Haproxy local0.notice -/var/log/haproxy-status.log;Haproxy local0.* ~ EOF
| 1 2 3 4 5 6 7 8 | [root@haproxy2 ~]# cat >> /etc/rsyslog.d/haproxy.conf << EOF $ModLoad imudp $UDPServerRun 514 $template Haproxy,"%msg%\n" local0.=info -/var/log/haproxy.log;Haproxy local0.notice -/var/log/haproxy-status.log;Haproxy local0.* ~ EOF |
Make sure the SELinux label is ok:
[root@haproxy1-1 ~]# /sbin/restorecon -v -F /etc/rsyslog.d/haproxy.conf
| 1 | [root@haproxy1-1 ~]# /sbin/restorecon -v -F /etc/rsyslog.d/haproxy.conf |
And restart the rsyslog service:
[root@haproxy2 ~]# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ]
| 1 2 3 | [root@haproxy2 ~]# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ] |
Add HAProxy config
HAProxy is extremely flexible and you can make it do a ton of cool things. Check out the HAProxy website for a taste what it’s capable of. The goal in this tutorial is to put the two HAProxy nodes in front of the MariaDB cluster nodes to loadbalance and when a MariaDB node fails to transparently redirect DB queries to the remaining operational MariaDB nodes.
On node haproxy1 first save the default configuration:
[root@haproxy1 ~]# cd /etc/haproxy/ [root@haproxy1 haproxy]# mv haproxy.cfg haproxy.cfg.orig
| 1 2 | [root@haproxy1 ~]# cd /etc/haproxy/ [root@haproxy1 haproxy]# mv haproxy.cfg haproxy.cfg.orig |
On node haproxy1 create the HAProxy configuration file:
[root@haproxy1 ~]# cat >> /etc/haproxy/haproxy.cfg << EOF global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 user haproxy group haproxy nbproc 1 pidfile /var/run/haproxy.pid defaults log global option tcplog option dontlognull retries 3 maxconn 4096 option redispatch timeout connect 50000ms timeout client 50000ms timeout server 50000ms listen mariadb-galera-writes bind 0.0.0.0:3307 mode tcp option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check backup server db3 10.0.0.13:3306 check backup listen mariadb-galera-reads bind 0.0.0.0:3306 mode tcp balance leastconn option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check server db3 10.0.0.13:3306 check # HAProxy web ui listen stats 0.0.0.0:9000 mode http stats enable stats uri /haproxy stats realm HAProxy\ Statistics stats auth haproxy:haproxy stats admin if TRUE EOF
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | [root@haproxy1 ~]# cat >> /etc/haproxy/haproxy.cfg << EOF global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 user haproxy group haproxy nbproc 1 pidfile /var/run/haproxy.pid
defaults log global option tcplog option dontlognull retries 3 maxconn 4096 option redispatch timeout connect 50000ms timeout client 50000ms timeout server 50000ms
listen mariadb-galera-writes bind 0.0.0.0:3307 mode tcp option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check backup server db3 10.0.0.13:3306 check backup
listen mariadb-galera-reads bind 0.0.0.0:3306 mode tcp balance leastconn option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check server db3 10.0.0.13:3306 check
# HAProxy web ui listen stats 0.0.0.0:9000 mode http stats enable stats uri /haproxy stats realm HAProxy\ Statistics stats auth haproxy:haproxy stats admin if TRUE EOF |
Make sure the SELinux label is ok:
[root@haproxy1 ~]# /sbin/restorecon -v -F /etc/haproxy/haproxy.cfg
| 1 | [root@haproxy1 ~]# /sbin/restorecon -v -F /etc/haproxy/haproxy.cfg |
Now let’s do the same for the second haproxy node.
On node haproxy2 first save the default configuration:
[root@haproxy2 ~]# cd /etc/haproxy/ [root@haproxy2 haproxy]# mv haproxy.cfg haproxy.cfg.orig
| 1 2 | [root@haproxy2 ~]# cd /etc/haproxy/ [root@haproxy2 haproxy]# mv haproxy.cfg haproxy.cfg.orig |
On node haproxy2 create the HAProxy configuration file again or copy it from node haproxy1 as they are the same.
[root@haproxy2 ~]# cat >> /etc/haproxy/haproxy.cfg << EOF global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 user haproxy group haproxy nbproc 1 pidfile /var/run/haproxy.pid defaults log global option tcplog option dontlognull retries 3 maxconn 4096 option redispatch timeout connect 50000ms timeout client 50000ms timeout server 50000ms listen mariadb-galera-writes bind 0.0.0.0:3307 mode tcp option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check backup server db3 10.0.0.13:3306 check backup listen mariadb-galera-reads bind 0.0.0.0:3306 mode tcp balance leastconn option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check server db3 10.0.0.13:3306 check # HAProxy web ui listen stats 0.0.0.0:9000 mode http stats enable stats uri /haproxy stats realm HAProxy\ Statistics stats auth haproxy:haproxy stats admin if TRUE EOF
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | [root@haproxy2 ~]# cat >> /etc/haproxy/haproxy.cfg << EOF global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 user haproxy group haproxy nbproc 1 pidfile /var/run/haproxy.pid
defaults log global option tcplog option dontlognull retries 3 maxconn 4096 option redispatch timeout connect 50000ms timeout client 50000ms timeout server 50000ms
listen mariadb-galera-writes bind 0.0.0.0:3307 mode tcp option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check backup server db3 10.0.0.13:3306 check backup
listen mariadb-galera-reads bind 0.0.0.0:3306 mode tcp balance leastconn option mysql-check user haproxy server db1 10.0.0.9:3306 check server db2 10.0.0.11:3306 check server db3 10.0.0.13:3306 check
# HAProxy web ui listen stats 0.0.0.0:9000 mode http stats enable stats uri /haproxy stats realm HAProxy\ Statistics stats auth haproxy:haproxy stats admin if TRUE EOF |
Make sure the SELinux label is ok:
[root@haproxy2 ~]# /sbin/restorecon -v -F /etc/haproxy/haproxy.cfg
| 1 | [root@haproxy2 ~]# /sbin/restorecon -v -F /etc/haproxy/haproxy.cfg |
Understanding the configuration of a HAProxy node is best done with the HAProxy docs. Look up each settings to get an understanding of what they mean.
The two sections called listen mariadb-galera-writes and listen mariadb-galera-reads are the importants ones.
The listen mariadb-galera-writes section has a check if the MariaDB nodes are still available and defines three MariaDB nodes with only one active and two as backup nodes. Should one MariaDB node fail then HAProxy will automatically ignore it and send write queries to the first backup node (db2 in this case). So this section limits writing to any database on the cluster to a single MariaDB node.
The listen mariadb-galera-reads section also has a check if the MariaDB nodes are still available. Since all three MariaDB nodes are active, all three MariaDB nodes can receive read queries. Should one MariaDB node fail then HAProxy will automatically ignore it and send queries to the other two MariaDB nodes where priority is given to the MariaDB node with the least connections. So this section allows reading from all three MariaDB nodes.
Make HAProxy high-available with Keepalived
What is Keepalived? Paraphrased from the Keepalive website: Keepalived provides simple and robust facilities for loadbalancing and high-availability to Linux system and Linux based infrastructures. Loadbalancing framework relies on the well-known and widely used Linux Virtual Server (IPVS) kernel module providing Layer4 loadbalancing while high-availability is achieved by the VRRP protocol.
Add Keepalived config
On node haproxy1 first save the default Keepalived configuration:
[root@haproxy1 ~]# cd /etc/keepalived/ [root@haproxy1 keepalived]# mv keepalived.cfg keepalived.cfg.orig
| 1 2 | [root@haproxy1 ~]# cd /etc/keepalived/ [root@haproxy1 keepalived]# mv keepalived.cfg keepalived.cfg.orig |
On node haproxy1 create the Keepalived configuration file:
[root@haproxy1 ~]# cat >> /etc/keepalived/keepalived.cfg << EOF # config file for keepalived on haproxy1 global_defs { notification_email { cluster-admin@example.org } notification_email_from haproxy1-noreply@example.org smtp_server smtp.example.org smtp_connect_timeout 30 router_id haproxy1 } vrrp_script haproxy { script "killall -0 haproxy" interval 2 weight 2 } vrrp_instance VI_1 { state MASTER interface eth0 smtp_alert virtual_router_id 10 priority 101 advert_int 1 authentication { auth_type PASS auth_pass <somesecret> # use 8 chars & something better } virtual_ipaddress { 10.0.0.25 } track_script { haproxy } } EOF
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | [root@haproxy1 ~]# cat >> /etc/keepalived/keepalived.cfg << EOF
# config file for keepalived on haproxy1
global_defs { notification_email { cluster-admin@example.org } notification_email_from haproxy1-noreply@example.org smtp_server smtp.example.org smtp_connect_timeout 30 router_id haproxy1 }
vrrp_script haproxy { script "killall -0 haproxy" interval 2 weight 2 }
vrrp_instance VI_1 { state MASTER interface eth0 smtp_alert virtual_router_id 10 priority 101 advert_int 1 authentication { auth_type PASS auth_pass <somesecret> # use 8 chars & something better } virtual_ipaddress { 10.0.0.25 } track_script { haproxy } } EOF |
Important: change the following parameters (if required) for your environment:
notification_email:
Change the ‘cluster-admin@example.org’ email address to the email address where you want the notifications to be sent.
notification_email_from:
Change the ‘haproxy-noreply@example.org’ email address to the your preferred ‘From:’ value.
smtp_server:
Change smtp.example.org to the DNS name or IP address of the MTA that your haproxy nodes use.
interface:
If the Virtual IP address (VIP) is not attached to eth0 then change it to the interface where it is attached.
auth_pass:
Change the ‘‘ password to a proper password of 8 characters.
virtual_ipaddress:
Change the VIP from ‘10.0.0.25’ to the Virtual IP address reserved for HAProxy.
Make sure the SELinux label is ok:
[root@haproxy1 ~]# /sbin/restorecon -v -F /etc/keepalived/keepalived.cfg
| 1 | [root@haproxy1 ~]# /sbin/restorecon -v -F /etc/keepalived/keepalived.cfg |
Let’s setup the Keepalived config on the second haproxy node (haproxy2):
On node haproxy2 first save the default Keepalived configuration:
[root@haproxy2 ~]# cd /etc/keepalived/ [root@haproxy2 keepalived]# mv keepalived.cfg keepalived.cfg.orig
| 1 2 | [root@haproxy2 ~]# cd /etc/keepalived/ [root@haproxy2 keepalived]# mv keepalived.cfg keepalived.cfg.orig |
On node haproxy2 create the Keepalived configuration file:
[root@haproxy2 ~]# cat >> /etc/keepalived/keepalived.cfg << EOF # config file for keepalived on haproxy2 global_defs { notification_email { cluster-admin@example.org } notification_email_from haproxy2-noreply@example.org smtp_server smtp.example.org smtp_connect_timeout 30 router_id haproxy2 } vrrp_script haproxy { script "killall -0 haproxy" interval 2 weight 2 } vrrp_instance VI_1 { state MASTER interface eth0 smtp_alert virtual_router_id 10 priority 100 advert_int 1 authentication { auth_type PASS auth_pass <somesecret> # use 8 chars & something better } virtual_ipaddress { 10.0.0.25 } track_script { haproxy } } EOF
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | [root@haproxy2 ~]# cat >> /etc/keepalived/keepalived.cfg << EOF
# config file for keepalived on haproxy2
global_defs { notification_email { cluster-admin@example.org } notification_email_from haproxy2-noreply@example.org smtp_server smtp.example.org smtp_connect_timeout 30 router_id haproxy2 }
vrrp_script haproxy { script "killall -0 haproxy" interval 2 weight 2 }
vrrp_instance VI_1 { state MASTER interface eth0 smtp_alert virtual_router_id 10 priority 100 advert_int 1 authentication { auth_type PASS auth_pass <somesecret> # use 8 chars & something better } virtual_ipaddress { 10.0.0.25 } track_script { haproxy } } EOF |
Make sure the SELinux label is ok:
[root@haproxy2 ~]# /sbin/restorecon -v -F /etc/keepalived/keepalived.cfg
| 1 | [root@haproxy2 ~]# /sbin/restorecon -v -F /etc/keepalived/keepalived.cfg |
Important – change the following parameters for your environment (if required) as you previously did for node haproxy1:
– notification_email
– notification_email_from
– smtp_server
– interface
– virtual_router_id (which should be the same on both nodes)
– auth_pass (which should be the same on both nodes)
– virtual_ipaddress (which should be the same on both nodes)
Important – note the differences and similarities between the Keepalived config file on haproxy1 and haproxy2:
Nodes haproxy1 | haproxy2 ========================================= router_id | haproxy1 | haproxy2 ----------------------------------------- virtual_router_id | 10 | 10 ----------------------------------------- priority | 101 | 100 ----------------------------------------- weight | 2 | 2 -----------------------------------------
| 1 2 3 4 5 6 7 8 9 10 | Nodes haproxy1 | haproxy2 ========================================= router_id | haproxy1 | haproxy2 ----------------------------------------- virtual_router_id | 10 | 10 ----------------------------------------- priority | 101 | 100 ----------------------------------------- weight | 2 | 2 ----------------------------------------- |
Keepalived Priorities and Weight or How Failover Works
haproxy1 has a base priority of 101 + a weight of 2 when HAProxy is running = 103
haproxy2 has a base priority of 100 + a weight of 2 when HAProxy is running = 102
Since haproxy1’s 103 is bigger than haproxy2’s 102 haproxy2 is not allowed to initiate a failover and grab the VIP.
Here’s what happens when HAProxy goes down on node haproxy1?
haproxy1 has a base priority of 101 + 0 because HAProxy is no longer running = 101
haproxy2 has a base priority of 100 + a weight of 2 because HAProxy is running = 102
Since haproxy1’s 101 is smaller than haproxy2’s 102 haproxy2 is allowed to initiate a failover and grab the VIP.
ProTip: check the priority that the ACTIVE node is broadcasting by sniffing the traffic with the following tcpdump command:
[root@haproxy2 ~]# tcpdump -s0 -v -pni eth0 host 224.0.0.18
| 1 | [root@haproxy2 ~]# tcpdump -s0 -v -pni eth0 host 224.0.0.18 |
Firewall Setup: Enable Multicast in IPtables
By default Keepalived uses multicast to send out its VRRP advertisements. You can configure Keepalived to use Unicast too if you prefer that. Here is an example multicast rule to allow multicast packets between the two haproxy nodes:
-A INPUT -m pkttype --pkt-type multicast -s <source_network /cidr> -d 224.0.0.0/8 -j ACCEPT</source_network>
| 1 | -A INPUT -m pkttype --pkt-type multicast -s <source_network /cidr> -d 224.0.0.0/8 -j ACCEPT</source_network> |
Since Keepalived is using vrrp.mcast.net (224.0.0.18) you could further tighten the rule above with:
-A INPUT -m pkttype --pkt-type multicast -s <source_network /cidr> -d 224.0.0.18/24 -j ACCEPT</source_network>
| 1 | -A INPUT -m pkttype --pkt-type multicast -s <source_network /cidr> -d 224.0.0.18/24 -j ACCEPT</source_network> |
As an example here is the first rule applied to the default CentOS 6 firewall:
# Firewall configuration written by system-config-firewall # Manual customization of this file is not recommended. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -m pkttype --pkt-type multicast -s 10.0.0.0/24 -d 224.0.0.0/8 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # Firewall configuration written by system-config-firewall # Manual customization of this file is not recommended. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -m pkttype --pkt-type multicast -s 10.0.0.0/24 -d 224.0.0.0/8 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT |
And if you want to manually add those rules and add a rule for protocol 112 (VRRP), here are the commands that you could use on both nodes:
[root@haproxy1,2 ~]# /sbin/iptables -I INPUT -i eth0 -d 224.0.0.0/8 -j ACCEPT [root@haproxy1,2 ~]# /sbin/iptables -A INPUT -p 112 -i eth0 -j ACCEPT [root@haproxy1,2 ~]# /sbin/iptables -A OUTPUT -p 112 -o eth0 -j ACCEPT [root@haproxy1,2 ~]# /sbin/service iptables save
| 1 2 3 4 | [root@haproxy1,2 ~]# /sbin/iptables -I INPUT -i eth0 -d 224.0.0.0/8 -j ACCEPT [root@haproxy1,2 ~]# /sbin/iptables -A INPUT -p 112 -i eth0 -j ACCEPT [root@haproxy1,2 ~]# /sbin/iptables -A OUTPUT -p 112 -o eth0 -j ACCEPT [root@haproxy1,2 ~]# /sbin/service iptables save |
Change your interface and tighten the source and/or destination networks to your requirements.
Troubleshooting Multicast
Here are some commands which show you the status of various interfaces and multicast groups those interfaces are member of:
[root@haproxy2-1 ~]# ip maddr show 1: lo inet 224.0.0.1 inet6 ff02::1 2: eth0 link 01:00:5e:00:00:12 link 33:33:00:00:02:02 link 33:33:ff:00:64:53 link 01:00:5e:00:00:01 link 33:33:00:00:00:01 inet 224.0.0.18 inet 224.0.0.1 inet6 ff02::202 inet6 ff02::1:ff00:6453 inet6 ff02::1 3: eth1 link 33:33:00:00:00:01 inet6 ff02::1
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | [root@haproxy2-1 ~]# ip maddr show 1: lo inet 224.0.0.1 inet6 ff02::1 2: eth0 link 01:00:5e:00:00:12 link 33:33:00:00:02:02 link 33:33:ff:00:64:53 link 01:00:5e:00:00:01 link 33:33:00:00:00:01 inet 224.0.0.18 inet 224.0.0.1 inet6 ff02::202 inet6 ff02::1:ff00:6453 inet6 ff02::1 3: eth1 link 33:33:00:00:00:01 inet6 ff02::1 |
[root@haproxy2 ~]# netstat -g IPv6/IPv4 Group Memberships Interface RefCnt Group --------------- ------ --------------------- lo 1 all-systems.mcast.net eth0 1 vrrp.mcast.net eth0 1 all-systems.mcast.net lo 1 ff02::1 eth0 1 ff02::202 eth0 1 ff02::1:ff00:6453 eth0 1 ff02::1 eth1 1 ff02::1
| 1 2 3 4 5 6 7 8 9 10 11 12 | [root@haproxy2 ~]# netstat -g IPv6/IPv4 Group Memberships Interface RefCnt Group --------------- ------ --------------------- lo 1 all-systems.mcast.net eth0 1 vrrp.mcast.net eth0 1 all-systems.mcast.net lo 1 ff02::1 eth0 1 ff02::202 eth0 1 ff02::1:ff00:6453 eth0 1 ff02::1 eth1 1 ff02::1 |
[root@haproxy2 ~]# cat /proc/net/igmp Idx Device : Count Querier Group Users Timer Reporter 1 lo : 1 V3 010000E0 1 0:00000000 0 2 eth0 : 2 V3 120000E0 1 0:00000000 0 010000E0 1 0:00000000 0
| 1 2 3 4 5 6 7 | [root@haproxy2 ~]# cat /proc/net/igmp Idx Device : Count Querier Group Users Timer Reporter 1 lo : 1 V3 010000E0 1 0:00000000 0 2 eth0 : 2 V3 120000E0 1 0:00000000 0 010000E0 1 0:00000000 0 |
ProTip: if multicast is disabled (but enabled in the kernel) you can manually enable it with the following command:
[root@haproxy2 ~]# ifconfig eth0 multicast
| 1 | [root@haproxy2 ~]# ifconfig eth0 multicast |
转载于:https://blog.51cto.com/756020191/1688725
扫一扫
272
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
