12-高可用Kubernetes集群(测试环境)-多master集群-Nginx+keepalived(LB)

1. 安装nginx

参照网址https://nginx.org/en/linux_packages.html

1.1 安装先决条件

sudo yum install yum-utils

1.2 设置yum存储库

创建/etc/yum.repos.d/nginx.repo 使用以下内容命名的文件

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key

默认情况下，使用稳定nginx包的存储库。如果要使用主线nginx包，请运行以下命令：

sudo yum-config-manager --enable nginx-mainline

1.3 安装nginx

sudo yum install nginx

1.4 nginx.conf

master配置文件

user  nginx;
worker_processes  4;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

stream{
    log_format main "$remote_addr $upstream_addr $time_local $status";
    access_log /var/log/nginx/k8s-access.log main;
    upstream k8s-apiserver {
        server 192.168.1.72:6443;
        server 192.168.1.98:6443;
    }
    server {
        listen 0.0.0.0:6443;
        proxy_pass k8s-apiserver; 
    }
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

backup节点也做类似配置

启动和自启动

systemctl enable nginx
systemctl start nginx

2. 安装keepalived

yum install keepalived -y

master上的配置文件keepalived.conf

! Configuration File for keepalived 
 
global_defs { 
   # 接收邮件地址 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   # 邮件发送地址 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER 
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface eth0
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 100    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.7.43/24 
    } 
    track_script {
        check_nginx
    } 
}

注意: vrrp_instance VI_1 -> interface为当前机器网卡的接口名称
backup上的配置配置文件主要修改一下信息

state BACKUP 
priority 90

keepalived检查脚本/etc/keepalived/check_nginx.sh

count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

启动和自启动

systemctl enable keepalived
systemctl start keepalived

3. 遇到的坑

部署keepalved时，发现vip无法ping通，即使在相同网络也无法ping通.
部署keepalved的主机是在openstack上创建的, 查询得到openstack的port所在的宿主机上iptables 对 MAC地址和IP进行了限制.具体操作参见:在openstack 中部署keepalived 高可用集群