Keepalived+LVS+Httpd高可用实战笔记[单网段]
此次试验模拟外部client通过路由gateway访问内部网络的高可用Keepalived+LVS+Httpd服务集群,两套LVS-DR模式下负载均衡同时提供服务。
创建虚拟机
vmcentos[0]=CentOS7X64-client
vmcentos[1]=CentOS7X64-gateway
vmcentos[2]=CentOS7X64-keep01
vmcentos[3]=CentOS7X64-keep02
vmcentos[4]=CentOS7X64-httpd01
vmcentos[5]=CentOS7X64-httpd02
parentPath="/media/WNTime/本地磁盘/VirtualMachine.Spaces"
vmsour="$parentPath/CentOS7X64.Core/CentOS7X64.Core.vmx"
for item in ${vmcentos[*]};
do
vmdest="$parentPath/$item/$item.vmx"
echo "Clone: $vmdest"
vmrun -T ws clone $vmsour $vmdest full -cloneName=$item
sleep 5s;
done;
启动虚拟机
parentPath="/media/WNTime/本地磁盘/VirtualMachine.Spaces"
for item in ${vmcentos[*]};
do
vmdest="$parentPath/$item/$item.vmx"
vmshot=`date +%Y%m%d%H%M`
echo "Start: $vmdest"
vmrun -T ws start $vmdest
sleep 5s;
read -p "按任意键继续..."
done;
挂起虚拟机
parentPath="/media/WNTime/本地磁盘/VirtualMachine.Spaces"
for item in ${vmcentos[*]};
do
vmdest="$parentPath/$item/$item.vmx"
vmshot=`date +%Y%m%d%H%M`
echo "Suspend: $vmdest"
vmrun -T ws suspend $vmdest
sleep 5s;
done;
快照备份
parentPath="/media/WNTime/本地磁盘/VirtualMachine.Spaces"
for item in ${vmcentos[*]};
do
vmdest="$parentPath/$item/$item.vmx"
vmshot=`date +%Y%m%d`
echo "Snapshot: $vmdest"
vmrun -T ws snapshot $vmdest $vmshot
vmrun -T ws listSnapshots $vmdest
sleep 5s;
done;
还原虚拟机
parentPath="/media/WNTime/本地磁盘/VirtualMachine.Spaces"
vmshot=`date +%Y%m%d`
for item in ${vmcentos[*]};
do
vmdest="$parentPath/$item/$item.vmx"
vmshot=`date +%Y%m%d`
echo "Revert: $vmdest"
vmrun -T ws revertToSnapshot $vmdest $vmshot
sleep 5s;
done;
关闭虚拟机
parentPath="/media/WNTime/本地磁盘/VirtualMachine.Spaces"
for item in ${vmcentos[*]};
do
vmdest="$parentPath/$item/$item.vmx"
vmshot=`date +%Y%m%d%H%M`
echo "Stop: $vmdest"
vmrun -T ws stop $vmdest
sleep 5s;
read -p "按任意键继续..."
done;
实验网络规划
client | gateway | Keepalived | Keepalived | httpd | httpd | |
---|---|---|---|---|---|---|
网段 | client | gateway | keep-101 | keep-102 | httpd-111 | httpd-112 |
192.168.86.0 192.168.88.0 | 192.168.88.16 GW:192.168.88.100 | 192.168.88.100 192.168.86.200 | 192.168.86.101 GW:192.168.86.200 | 192.168.86.102 GW:192.168.86.200 | 192.168.86.111 GW:192.168.86.200 | 192.168.86.112 GW:192.168.86.200 |
VIP1 | 192.168.86.251(MASTER) | 192.168.86.251(BACKUP) | lo:192.168.86.251 | lo:192.168.86.251 | ||
VIP2 | 192.168.86.252(BACKUP) | 192.168.86.252(MASTER) | lo:192.168.86.252 | lo:192.168.86.252 |
配置hostname及ipaddress
client
## client
# vim /etc/sysconfig/network-script/ifcfg-en33
sudo hostnamectl set-hostname client
sudo sed -i -e 's/192.168.86.6/192.168.88.16/g' \
-e 's/^GATEWAY=192.168.86.2/GATEWAY=192.168.88.100/g' \
-e '/^UUID=/d' \
/etc/sysconfig/network-scripts/ifcfg-ens33
sudo systemctl restart network
gateway
## gateway
# vim /etc/sysconfig/network-script/ifcfg-en33
sudo hostnamectl set-hostname gateway
sudo cp /etc/sysconfig/network-scripts/ifcfg-ens33 /etc/sysconfig/network-scripts/ifcfg-ens36
sudo sed -i -e 's/192.168.86.6/192.168.86.200/g' \
-e '/^GATEWAY/d' \
-e '/^UUID=/d' \
/etc/sysconfig/network-scripts/ifcfg-ens33
sudo sed -i -e 's/192.168.86.6/192.168.88.100/g' \
-e '/^GATEWAY=/d' \
-e '/^UUID=/d' \
-e 's/ens33/ens36/g' \
/etc/sysconfig/network-scripts/ifcfg-ens36
sudo systemctl restart network
# 配置 另外一个Ip 为192.168.88.100
httpd-111
## httpd-111
# vim /etc/sysconfig/network-script/ifcfg-en33
sudo hostnamectl set-hostname httpd-111
sudo sed -i -e 's/192.168.86.6/192.168.86.111/g' \
-e 's/^GATEWAY=192.168.86.2/GATEWAY=192.168.86.200/g' \
-e '/^UUID=/d' \
/etc/sysconfig/network-scripts/ifcfg-ens33
sudo systemctl restart network
httpd-112
## httpd-112
# vim /etc/sysconfig/network-script/ifcfg-en33
sudo hostnamectl set-hostname httpd-112
sudo sed -i -e 's/192.168.86.6/192.168.86.112/g' \
-e 's/^GATEWAY=192.168.86.2/GATEWAY=192.168.86.200/g' \
-e '/^UUID=/d' \
/etc/sysconfig/network-scripts/ifcfg-ens33
sudo systemctl restart network
keep-101
## keep-101
# vim /etc/sysconfig/network-script/ifcfg-en33
sudo hostnamectl set-hostname keep-101
sudo sed -i -e 's/192.168.86.6/192.168.86.101/g' \
-e 's/^GATEWAY=192.168.86.2/GATEWAY=192.168.86.200/g' \
-e '/^UUID=/d' \
/etc/sysconfig/network-scripts/ifcfg-ens33
sudo systemctl restart network
keep-102
## keep-102
# vim /etc/sysconfig/network-script/ifcfg-en33
sudo hostnamectl set-hostname keep-102
sudo sed -i -e 's/192.168.86.6/192.168.86.102/g' \
-e 's/^GATEWAY=192.168.86.2/GATEWAY=192.168.86.200/g' \
-e '/^UUID=/d' \
/etc/sysconfig/network-scripts/ifcfg-ens33
sudo systemctl restart network
关闭防火墙
sudo setenforce 0
sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
sudo systemctl disable firewalld
sudo systemctl stop firewalld
sudo yum makecache
sudo yum install -y wget net-tools
sudo mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
sudo wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
sudo yum makecache
注意:一定要关闭防火墙,或是添加相关端口!
确认网关是否生效
route -n
安装 web服务测试
httpd-111, httpd-112
sudo yum install -y httpd
sudo systemctl stop firewalld
echo " sed -i 's/123/$HOSTNAME/' /usr/share/httpd/noindex/index.html" | sudo bash -
sudo systemctl restart httpd
安装keepalived ipvsadm
keep-101, keep-102
wget https://www.keepalived.org/software/keepalived-2.1.5.tar.gz --no-check-certificate
sudo yum install -y ipvsadm libnl libnl-devel libnl3-devel net-snmp-devel libnfnetlink-devel gcc make openssl-devel popt-devel curl
tar -xzf keepalived-2.1.5.tar.gz
cd keepalived-2.1.5/
./configure --prefix=/usr/local/keepalived --sysconf=/etc
make && sudo make install
# 测试安装
/usr/local/keepalived/sbin/keepalived -v
安装成功
ls -l /etc/keepalived/
配置系统服务
keep-101, keep-102
# ~/keepalived-2.1.5 是解压出来的文件夹,根据你的解压路径来,不是安装目录
sudo cp ~/keepalived-2.1.5/keepalived/etc/init.d/keepalived /etc/init.d/
sudo cp ~/keepalived-2.1.5/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
sudo systemctl daemon-reload
sudo systemctl enable keepalived
sudo systemctl status keepalived
高可用配置
keep-101
# 配置lvs
# 开启net.ipv4.ip_forward
sudo vim /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.ens33.send_redirects=0
# 加载并生效
sudo sysctl -p
配置keepalived
# sudo vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id keep_vip1
}
vrrp_instance VIP_251 {
state BACKUP
interface ens33
virtual_router_id 251
priority 100
advert_int 1
nopreempt
#preempt_delay 300
authentication {
auth_type PASS
auth_pass 12345678
}
virtual_ipaddress {
192.168.86.251
}
}
vrrp_instance VIP_252{
state BACKUP
interface ens33
virtual_router_id 252
priority 90
advert_int 1
nopreempt
#preempt_delay 300
authentication {
auth_type PASS
auth_pass 12345678
}
virtual_ipaddress {
192.168.86.252
}
}
# 检测语法
/usr/local/keepalived/sbin/keepalived -t
# 启动keepalived
sudo systemctl restart keepalived
sudo systemctl status keepalived
keep-102
# sudo vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id keep_vip2
}
vrrp_instance VIP_251 {
state BACKUP
interface ens33
virtual_router_id 251
priority 90
advert_int 1
nopreempt
#preempt_delay 300
authentication {
auth_type PASS
auth_pass 12345678
}
virtual_ipaddress {
192.168.86.251
}
}
vrrp_instance VIP_252{
state BACKUP
interface ens33
virtual_router_id 252
priority 100
advert_int 1
nopreempt
#preempt_delay 300
authentication {
auth_type PASS
auth_pass 12345678
}
virtual_ipaddress {
192.168.86.252
}
}
# 检测语法
/usr/local/keepalived/sbin/keepalived -t
# 启动keepalived
sudo systemctl restart keepalived
sudo systemctl status keepalived
keepalived 配置成功
配置LVS-DR模式
# httpd-111
su root
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
cp /etc/sysconfig/network-scripts/ifcfg-lo /etc/sysconfig/network-scripts/ifcfg-lo.bak
echo '' >> /etc/sysconfig/network-scripts/ifcfg-lo
echo IPADDR1=192.168.86.251 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo NETMASK1=255.255.255.255 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo PREFIX1=32 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo IPADDR2=192.168.86.252>> /etc/sysconfig/network-scripts/ifcfg-lo
echo NETMASK2=255.255.255.255 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo PREFIX2=32 >> /etc/sysconfig/network-scripts/ifcfg-lo
systemctl restart network
# httpd-112
su root
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
cp /etc/sysconfig/network-scripts/ifcfg-lo /etc/sysconfig/network-scripts/ifcfg-lo.bak
echo '' >> /etc/sysconfig/network-scripts/ifcfg-lo
echo IPADDR1=192.168.86.251 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo NETMASK1=255.255.255.255 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo PREFIX1=32 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo IPADDR2=192.168.86.252>> /etc/sysconfig/network-scripts/ifcfg-lo
echo NETMASK2=255.255.255.255 >> /etc/sysconfig/network-scripts/ifcfg-lo
echo PREFIX2=32 >> /etc/sysconfig/network-scripts/ifcfg-lo
systemctl restart network
# keep-101 keepalived.conf 追加配置
# sudo vim /etc/keepalived/keepalived.conf
virtual_server 192.168.86.251 80 {
delay_loop 1
lvs_sched wrr
lvs_method DR
#persistence_timeout 60
protocol TCP
real_server 192.168.86.111 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
real_server 192.168.86.112 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
}
virtual_server 192.168.86.252 80 {
delay_loop 1
lvs_sched wrr
lvs_method DR
#persistence_timeout 60
protocol TCP
real_server 192.168.86.111 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
real_server 192.168.86.112 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
}
# 检测语法
/usr/local/keepalived/sbin/keepalived -t
# 启动keepalived
sudo systemctl restart keepalived
sudo systemctl status keepalived
# keep-102 keepalived.conf 追加配置
# sudo vim /etc/keepalived/keepalived.conf
virtual_server 192.168.86.251 80 {
delay_loop 1
lvs_sched wrr
lvs_method DR
persistence_timeout 60
protocol TCP
real_server 192.168.86.111 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
real_server 192.168.86.112 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
}
virtual_server 192.168.86.252 80 {
delay_loop 1
lvs_sched wrr
lvs_method DR
persistence_timeout 60
protocol TCP
real_server 192.168.86.111 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
real_server 192.168.86.112 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
retry 3
delay_before_retry 4
}
}
}
# 检测语法
/usr/local/keepalived/sbin/keepalived -t
# 启动keepalived
sudo systemctl restart keepalived
sudo systemctl status keepalived
ipvsadm -Ln 规则查看
测试 lvs 负载均衡效果
gateway 配置
# 配置lvs
# 开启net.ipv4.ip_forward
sudo vim /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.ens33.send_redirects=0
# 加载并生效
sudo sysctl -p
client
# 配置 网关
route add -net 192.168.86.0/24 gw 192.168.88.100
while true; do
sleep 1s; date;
curl -s http://192.168.86.251|grep Testing;
curl -s http://192.168.86.251|grep Testing;
curl -s http://192.168.86.252|grep Testing;
curl -s http://192.168.86.252|grep Testing;
done;
初始测试效果
keep-101下线
keep-102下线
keep-102上线
keep-101 与 keep-102 对比
修改 keep-102 注释 persistence_timeout 60 ,然后下线 keep-101
在一定时间内使来自于同一个Client的所有TCP请求被负载到同一个RealServer上。
以下是 https://www.keepalived.org/manpage.html 给出的说明:
# LVS persistence timeout in seconds, default 6 minutes
persistence_timeout [INTEGER]