内网环境基于 k8s 的大型网站电商解决方案(一）

一、环境说明

1、所有系统为rockylinux8.6最小化安装，所有服务器均为内网，只有manager为双网卡（可访问互联网），提供内网yum源、DNS解析、时间同步等
2、 k8s搭建高可用集群版本为1.24.6(基于containerd部署) 3台控制节点，2台工作节点
3、搭建rancher平台管理k8s集群(注：rancher为centos7.9最小化安装）
4、 mysql版本为8.0.31  搭建MGR
5、 ceph版本为quincy版，通过cephadm搭建
6、镜像存放在 harbor 仓库，版本为2.6.0
8、电商项目使用 LNMP 架构
9、PHP 和 Nginx 共享同一个 pvc：基于 cephfs 划分 pv
10、使用 Prometheus 监控电商平台，在 Grafana 可视化展示监控数据
11、搭建 efk+logstash+kafka 日志收集平台
12、K8S升级，将k8s升级至1.25.2，备份etcd

规划如下：

序号	系统名	IP地址	配置	作用	备注
1	master1	192.168.8.81	8G /sda 60G	K8S控制节点
2	master2	192.168.8.82	8G /sda 60G	K8S控制节点
3	master3	192.168.8.83	8G /sda 60G	K8S控制节点
4	node1	192.168.8.84	8G /sda 60G	K8S工作节点
5	node2	192.168.8.85	8G /sda 60G	K8S工作节点
6	master	192.168.8.88	8G /sda 60G	K8S  VIP
7	harbor1	192.168.8.91	8G /sda 60G	harbor 私有仓库
8	harbor2	192.168.8.92	8G /sda 60G	harbor 私有仓库
9	rancher	192.168.8.96	8G /sda 60G	rancher管理平台	centos7.9
10	mysqla	192.168.8.51	8G /sda 60G	mysql数据库
11	mysqlb	192.168.8.52	8G /sda 60G	mysql数据库
12	mysqlc	192.168.8.53	8G /sda 60G	mysql数据库
13	mysql	192.168.8.55	8G /sda 60G	mysql数据库 vip
14	cepha	192.168.8.61	8G /sda 60G，/sdb 20G，/sdc 20G	ceph集群
15	cephb	192.168.8.62	8G /sda 60G，/sdb 20G，/sdc 20G	ceph集群
16	cephc	192.168.8.63	8G /sda 60G，/sdb 20G，/sdc 20G	ceph集群
17	nfs	192.168.8.100	8G /sda 60G	nfs共享
18	manager	192.168.8.80	8G /sda 60G	yum源、dns、ntp等

二、基础环境搭建

1、安装系统rockylinux8.6最小化

网卡模式为仅主机

手动分区

所有服务器配置DNS为192.168.8.80,gateway 192.168.8.1

 关闭selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

2、安装基础软件包及chrony（manager)

设置防火墙

firewall-cmd --add-service=http --add-service=ntp --add-service=dns --permanent
firewall-cmd --reload

安装基础软件包

yum install vim net-tools bash-completion wget -y

安装chronyc

yum install chrony -y
sed -i 's/2.pool.ntp.org/ntp.aliyun.com/g' /etc/chrony.conf
echo 'allow 192.168.8.0/24' >> /etc/chrony.conf
systemctl enable --now chronyd
systemctl status chronyd
chronyc sources

 3、配置manager服务器yum源

yum install httpd -y
systemctl enable  --now  httpd
mkdir /var/www/html/k8s
mkdir /var/www/html/ceph
mkdir /var/www/html/epel 
mkdir /var/www/html/docker
dnf install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo

cat << EOF  > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

cat  >  /etc/yum.repos.d/ceph.repo  << EOF
[ceph-norch]
name=ceph-norch
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/noarch/
enable=1
gpgcheck=0

[ceph-x86_64]
name=ceph-x86_64
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/x86_64/
enable=1
gpgcheck=0

[ceph-source]
name=ceph-source
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el8/SRPMS/
enable=1
gpgcheck=0

EOF

4、搭建yum源服务器（manager）

mount /dev/sr0 /mnt/
cp -r  /mnt/*  /var/www/html/
mv AppStream appstream
mv BaseOS  baseos

dnf install  -y kubelet-1.24.6 kubeadm-1.24.6 kubectl-1.24.6 --downloadonly --destdir /var/www/html/k8s/
dnf install  -y kubelet kubeadm kubectl --downloadonly --destdir /var/www/html/k8s/
dnf install  -y docker-ce --downloadonly --destdir /var/www/html/docker/
dnf install  -y cephadm   --downloadonly --destdir /var/www/html/ceph/
dnf install  -y ceph-common --downloadonly --destdir /var/www/html/ceph/
dnf install  -y perl  --downloadonly --destdir /var/www/html/epel/

yum install createrepo
createrepo /var/www/html/k8s
createrepo /var/www/html/docker
createrepo /var/www/html/ceph
createrepo /var/www/html/epel
systemctl restart httpd

http://192.168.8.80/appstream/

5、安装dnsmasq(manager)

yum install dnsmasq -y
echo 'listen-address=192.168.8.80' >> /etc/dnsmasq.conf
cat >> /etc/hosts << EOF
192.168.8.80 manager
192.168.8.81 master1
192.168.8.82 master2
192.168.8.83 master3
192.168.8.84 node1
192.168.8.85 node2
192.168.8.88 master
192.168.8.91 harbor1
192.168.8.92 harbor2
192.168.8.96 rancher
192.168.8.51 mysqla
192.168.8.52 mysqlb
192.168.8.53 mysqlc
192.168.8.55 mysql
192.168.8.61 cepha
192.168.8.62 cephb
192.168.8.63 cephc
192.168.8.100 nfs

EOF

systemctl enable --now dnsmasq

6、安装docker

 7、配置内网服务器yum源及NTP配置

除80外所有服务器上执行

rm -rf /etc/yum.repos.d/*
cat > /etc/yum.repos.d/base.repo << EOF
[appstream]
name=appstream
baseurl=http://manager/appstream
enable=1
gpgcheck=0

[baseos]
name=baseos
baseurl=http://manager/baseos
enable=1
gpgcheck=0

[k8s]
name=k8s
baseurl=http://manager/k8s
enable=1
gpgcheck=0

[docker]
name=k8s
baseurl=http://manager/docker
enable=1
gpgcheck=0

[ceph]
name=ceph
baseurl=http://manager/ceph
enable=1
gpgcheck=0

[epel]
name=epel
baseurl=http://manager/epel
enable=1
gpgcheck=0

EOF

yum install -y wget bash-completion vim net-tools  chrony
sed -i 's/2.pool.ntp.org/manager/g' /etc/chrony.conf
systemctl enable --now chronyd
chronyc sources

 所有服务器关机，打快照

三、搭建harbor私有仓库(harbor1、harbor2）

 1、安装docker

yum install -y  docker-ce
systemctl start docker && systemctl enable docker

2、修改内核参数

modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/profile
cat > /etc/sysctl.d/docker.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/docker.conf
systemctl restart docker

3、安装docker-compose

  上传docker-compose-linux-x86_64至/root

wget https://github.com/goharbor/harbor/releases/download/v2.6.0/harbor-offline-installer-v2.6.0.tgz
wget https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-x86_64
mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

4、生成ca证书

mkdir /data/ssl -p
cd /data/ssl/
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.pem

Country Name (2 letter code) [XX]: CN
State or Province Name (full name) []:xinjiang
Locality Name (eg, city) [Default City]:urumqi      
Organization Name (eg, company) [Default Company Ltd]:myhub
Organizational Unit Name (eg, section) []:CA
Common Name (eg, your name or your server's hostname) []:harbor1
Email Address []:23939296@qq.com

5、生成域名证书

openssl genrsa -out myhub.key  2048
openssl req -new -key myhub.key -out myhub.csr

Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:xinjiang
Locality Name (eg, city) [Default City]:urumqi
Organization Name (eg, company) [Default Company Ltd]:myhub
Organizational Unit Name (eg, section) []:CA
Common Name (eg, your name or your server's hostname) []:myhub
Email Address []:23939296@qq.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

openssl x509 -req -in myhub.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out myhub.pem -days 365
openssl x509 -noout -text -in myhub.pem

6、安装harbor

mkdir /data/install -p
ll /data/ssl

 

cd 
mv harbor-offline-installer-v2.6.0.tgz /data/install/
cd /data/install/
tar -xvf harbor-offline-installer-v2.6.0.tgz
cd harbor
cp harbor.yml.tmpl harbor.yml
vim harbor.yml

hostname: harbor1  ( harbor2)

certificate: /data/ssl/myhub.pem

private_key: /data/ssl/myhub.key

harbor_admin_password: password

docker load -i harbor.v2.6.0.tar.gz
./install.sh

7、 停止harbor

cd /data/install/harbor
docker-compose stop

8、启动harbor

cd /data/install/harbor
docker-compose start
firewall-cmd --add-service=http --add-service=https --permanent;firewall-cmd --reload

http://192.168.8.91     http://192.168.8.92

9、配置镜像自动同步

新建项目myhub    http://192.168.8.91

 配置仓库

 新建复制规则

10、在192.168.8.80上测试

cat > /etc/docker/daemon.json  << EOF
{
"registry-mirrors": ["http://hub-mirror.c.163.com","https://0x3urqgf.mirror.aliyuncs.com"],
"insecure-registries": [ "192.168.8.91","harbor1" ]
}

EOF

systemctl daemon-reload
systemctl restart docker

docker login 192.168.8.91

 上传镜像至仓库

docker pull nginx
docker pull busybox
docker tag busybox:latest 192.168.8.91/myhub/busybox:latest
docker tag nginx:latest 192.168.8.91/myhub/nginx:latest
docker push 192.168.8.91/myhub/busybox:latest
docker push 192.168.8.91/myhub/nginx:latest

http://192.168.8.91     http://192.168.8.92登录验证

 

镜像已自动同步

四、安装k8s高可用集群（master1-3，node1-2)

1、修改内核参数（五台设备上执行）

modprobe br_netfilter
lsmod | grep br_netfilter

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables =