03-lvs-persistence
功能
无论ipvs使用何种scheduler,其都能够实现在 指定 时间范围内 始终将来自同一个ip地址的请求发往同一个RS,此功能是通过lvs持久链接模板实现的,其与调度算法无关;即即使使用rr算法,也是同一个rs
与SH算法的区别
sh始终记录源IP,故始终会调度到同一个RS,而lvs持久链接再超时之后会采用定义的调度算法来进行调度。
模型
每端口持久PPC
director处理端口分发,根据端口持久。即对于同一个端口的访问,lvs会定向到同一个RS;而对于不同端口的访问,lvs会重新定向RS
每客户端持久PCC
director不处理端口分发,即把director的端口设置为0,对于同一个客户端,lvs最开始定向到哪个RS,那么client以后所有的服务请求都将由这个RS去响应
每防火墙标记持久PFWMC
可以将两个服务绑定,client的请求将会定向到不同的RS,如http何https。
使用lvs persistence
使用lvs persistence很简单,只需要在开启lvs服务的时候添加一个-p选项就行了
ipvsadm -A|E -t|u|f service-address [-s scheduler]
[-p [timeout]] [-M netmask] [-b sched-flags]
timeout默认是360s
PPC
1 通过lvs-op-dr配置lvs服务
2 清空lvs服务
[root@localhost ~]# ipvsadm -C
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
3 添加lvs规则
[root@localhost ~]# ipvsadm -A -t 172.16.11.207:80 -s rr -p
[root@localhost ~]# ipvsadm -a -t 172.16.11.207:80 -r 172.16.11.101 -g
[root@localhost ~]# ipvsadm -a -t 172.16.11.207:80 -r 172.16.11.102 -g
[root@localhost ~]# ipvsadm -A -t 172.16.11.207:22 -s rr -p
[root@localhost ~]# ipvsadm -a -t 172.16.11.207:22 -r 172.16.11.101 -g
[root@localhost ~]# ipvsadm -a -t 172.16.11.207:22 -r 172.16.11.102 -g
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.11.207:22 rr persistent 360
-> 172.16.11.101:22 Route 1 0 0
-> 172.16.11.102:22 Route 1 0 0
TCP 172.16.11.207:80 rr persistent 360
-> 172.16.11.101:80 Route 1 0 0
-> 172.16.11.102:80 Route 1 0 0
添加了两种lvs,且都是用rr算法,其中一种为httpd服务,另一种是ssh服务
4 client请求验证
# client请求http服务
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
# client请求ssh服务
[root@husa ~]# ssh root@172.16.11.207
The authenticity of host '172.16.11.207 (172.16.11.207)' can't be established.
ECDSA key fingerprint is 25:93:d5:1e:fc:c2:3e:e6:bb:bb:9f:d1:f7:8a:bb:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.11.207' (ECDSA) to the list of known hosts.
root@172.16.11.207's password:
Last login: Sat Jan 30 16:28:41 2016 from 172.16.250.35
[root@localhost ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.16.11.207/32 brd 172.16.11.207 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno16777728: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:94:41:36 brd ff:ff:ff:ff:ff:ff
inet 172.16.11.102/16 brd 172.16.255.255 scope global eno16777728
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe94:4136/64 scope link
valid_lft forever preferred_lft forever
5: eno33554968: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:94:41:40 brd ff:ff:ff:ff:ff:ff
可以看到两次都定向到了RS2,换一台client试试^_^
经过多次试验才得出了RS1的http服务
可以发现,即使使用rr算法,对于同一个服务,那么RS就会是最开始提供服务的RS而不会发生改变
PCC
1 通过lvs-op-dr配置lvs服务
2 清空lvs服务
[root@localhost ~]# ipvsadm -C
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
3 添加lvs规则
[root@localhost ~]# ipvsadm -A -t 172.16.11.207:0 -s rr -p
[root@localhost ~]# ipvsadm -a -t 172.16.11.207:0 -r 172.16.11.101 -g
[root@localhost ~]# ipvsadm -a -t 172.16.11.207:0 -r 172.16.11.102 -g
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.11.207:0 rr persistent 360
-> 172.16.11.101:0 Route 1 0 0
-> 172.16.11.102:0 Route 1 0 0
把director的端口设置为0
4 client测试验证
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# curl http://172.16.11.207
<h1>REAL.Server2</h1>
[root@husa ~]# ssh root@172.16.11.207
root@172.16.11.207's password:
Last login: Sat Jan 30 20:02:22 2016 from 172.16.250.35
[root@localhost ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.16.11.207/32 brd 172.16.11.207 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno16777728: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:94:41:36 brd ff:ff:ff:ff:ff:ff
inet 172.16.11.102/16 brd 172.16.255.255 scope global eno16777728
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe94:4136/64 scope link
valid_lft forever preferred_lft forever
5: eno33554968: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:94:41:40 brd ff:ff:ff:ff:ff:ff
可以发现client一旦连接到一个RS上,那么后续的所有请求都将发送到这个RS上
PFWMC
1 以director为CA服务器,给RS1/2签证书
1.1 在director上生成私钥和自签证书
[root@localhost ~]# cd /etc/pki/CA
# 生成私钥
[root@localhost CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus
...........................................+++
...............................................+++
e is 65537 (0x10001)
# 生成自签证书
[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Hubei
Locality Name (eg, city) [Default City]:Wuhan
Organization Name (eg, company) [Default Company Ltd]:hazu
Organizational Unit Name (eg, section) []:optimize
Common Name (eg, your name or your server's hostname) []:www.withshirley.com
Email Address []:huaxiongcool@126.com
[root@localhost CA]# echo 01 > serial
[root@localhost CA]# touch index.txt
1.2 RS1/2生成私钥及证书签署请求
# RS1
[root@husa httpd]# cd /etc/httpd
[root@husa httpd]# mkdir ssl
[root@husa httpd]# cd ssl
[root@husa ssl]# (umask 077;openssl genrsa -out httpd.key 1024)
Generating RSA private key, 1024 bit long modulus
..............................++++++
.++++++
e is 65537 (0x10001)
[root@husa ssl]# openssl req -new -key httpd.key -out httpd.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Hubei
Locality Name (eg, city) [Default City]:Wuhan
Organization Name (eg, company) [Default Company Ltd]:hzau
Organizational Unit Name (eg, section) []:optimize
Common Name (eg, your name or your server's hostname) []:www.withshirley.com
Email Address []:huaxiongcool@126.com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
1.3 把RS1生成的证书签署请求发送到director
[root@husa ssl]# scp httpd.csr root@172.16.11.207:/tmp
The authenticity of host '172.16.11.207 (172.16.11.207)' can't be established.
ECDSA key fingerprint is f3:61:9f:e3:7c:e3:51:88:19:62:b8:6e:83:e6:c1:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.11.207' (ECDSA) to the list of known hosts.
root@172.16.11.207's password:
httpd.csr 100% 708 0.7KB/s 00:00
1.4 director 签署RS1的证书
[root@localhost CA]# openssl ca -in /tmp/httpd.csr -out /etc/pki/CA/certs/httpd.crt
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Dec 26 10:05:38 2015 GMT
Not After : Dec 25 10:05:38 2016 GMT
Subject:
countryName = CN
stateOrProvinceName = Hubei
organizationName = hzau
organizationalUnitName = optimize
commonName = www.withshirley.com
emailAddress = huaxiongcool@126.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
29:A5:91:C2:4C:9E:5A:25:F2:E3:C7:01:13:06:AB:84:DC:9D:0B:09
X509v3 Authority Key Identifier:
keyid:A1:F4:F7:6D:46:64:30:19:9C:9E:8C:2A:55:65:08:F8:00:7A:9F:4A
Certificate is to be certified until Dec 25 10:05:38 2016 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
1.5 把director生成的证书发送给RS1
[root@localhost CA]# scp certs/httpd.crt root@172.16.11.101:/etc/httpd/ssl/
1.6 为了避免1.4-1.5之间的重复操作,直接把RS1的ssl目录复制给RS2
[root@husa ssl]# scp -rp /etc/httpd/ssl root@172.16.11.102:/etc/httpd/
The authenticity of host '172.16.11.102 (172.16.11.102)' can't be established.
ECDSA key fingerprint is 25:93:d5:1e:fc:c2:3e:e6:bb:bb:9f:d1:f7:8a:bb:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.11.102' (ECDSA) to the list of known hosts.
root@172.16.11.102's password:
httpd.key 100% 891 0.9KB/s 00:00
httpd.csr 100% 708 0.7KB/s 00:00
httpd.crt 100% 3899 3.8KB/s 00:00
2 RS1/2配置https服务
2.1 安装mod_ssl
# RS1
[root@husa httpd]# yum install mod_ssl
# RS2
[root@localhost httpd]# yum install mod_ssl
2.2 编辑/etc/httpd/conf.d/ssl.conf
# RS1 \ 2
DocumentRoot "/var/www/html"
ServerName www.withshirley.com:443
SSLCertificateFile /etc/httpd/ssl/httpd.crt
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
3 client图形界面测试
3.1 client从CA server获取证书
# Arch
[husa@ArchLinux-husa ~]$ sudo scp root@172.16.11.207:/etc/pki/CA/cacert.pem /tmp
[sudo] password for husa:
The authenticity of host '172.16.11.207 (172.16.11.207)' can't be established.
RSA key fingerprint is SHA256:MUwnEiW2IT6byCgRMQ5lTejhuEVR1mCEVY9GPQuGZeo.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.11.207' (RSA) to the list of known hosts.
root@172.16.11.207's password:
cacert.pem 100% 1440 1.4KB/s 00:00
[husa@ArchLinux-husa ~]$ ls /tmp
0d38376c904e6e548802ff5fa1da774b.scel.gz
cacert.pem
fcitx-socket-:0
kde-husa
sddm-:0-qTEEMZ
sddm-auth5af2da41-e4ff-4187-b7df-ccc0aebb5637
sogou-qimpanel:0.pid
sogou-qimpanel-cell
sogou-qimpanelhusa
systemd-private-edf79725d3024643b122b894a8e4ff46-ntpd.service-lSqyNH
xauth-1000-_0
yaourt-tmp-husa
3.2 修改/etc/hosts文件
[root@husa ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdo
main4
::1 localhost localhost.localdomain localhost6 localhost6.localdo
main6
172.16.11.101 www.withshirley.com
#172.16.11.102 www.withshirley.com
3.3 RS1/2开启httpd服务
[root@localhost conf.d]# systemctl restart httpd.service
[root@localhost conf.d]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 128 :::443 :::*
3.2 client的GUI 浏览器导入证书
3.3 client的GUI 浏览器访问站点
4 director添加基于fwm的lvs服务
[root@localhost CA]# iptables -t mangle -A PREROUTING -d 172.16.11.207 -p tcp --dport 80 -j MARK --set-mark 222
[root@localhost CA]# iptables -t mangle -A PREROUTING -d 172.16.11.207 -p tcp --dport 443 -j MARK --set-mark 222
[root@localhost CA]# ipvsadm -A -f 222 -s rr -p
[root@localhost CA]# ipvsadm -a -f 222 -r 172.16.11.101 -g
[root@localhost CA]# ipvsadm -a -f 222 -r 172.16.11.102 -g
5 client再次访问
可以细微的发现,上图中的http和https都指向了Real Server2
通过上面的配置,client对于RS的不管是http还是https访问都将会重定向到同一台RS