1.前言
本次搭建一主两从的k8s集群,k8s使用的版本为1.26.0版本,选用docker作为容器引擎,因为k8s1.24.0版本之后不适配docker的原因,需要用到cri-docker去让k8s适配docker,也可以选用containerd作为容器引擎,使用containerd作为容器引擎就不会有不适配的问题,系统使用centos7.9版本
2.集群信息
主机名称 | IP | SERVICE |
k8s-master01 | 10.1.60.119 | kubectl、kubelet、kubeadm、docker、cri-docker |
k8s-node01 | 10.1.60.120 | kubectl、kubelet、kubeadm、docker、cri-docker |
k8s-node02 | 10.1.60.121 | kubectl、kubelet、kubeadm、docker、cri-docker |
3.系统基础环境配置(所有主机均要配置)
3.1关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
3.2关闭selinux服务
setenforce 0 #临时关闭
sed -i 's/enforcing/disabled/' /etc/selinux/config #永久关闭
3.3关闭系统交换分区
swapoff -a #临时关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab #永久关闭
3.4设置主机名称(三台主机名称不一样按照表格上的主机名填)
hostnamectl set-hostname k8s-master01
3.5编辑hosts文件
cat > /etc/hosts << EOF
10.1.60.119 k8s-master01
10.1.60.120 k8s-node01
10.1.60.121 k8s-node02
EOF
3.6配置时间同步
yum -y install chrony
systemctl strat chronyd && systemctl enbale chronyd
3.7配置ipvs
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
#授权并生效ipvs配置
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules
yum install -y ipset ipvsadm
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter
3.8将桥接的IPv4流量传递到iptables
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system #配置生效
4.安装docker服务
4.1下载docker yum源
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
4.2安装docker服务
yum -y install docker-ce
4.3启用docker并配置开机启动
systemctl start docker && systemctl enable docker
4.4配置docker镜像加速
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://sudzwtcw.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
4.5重启docker服务
systemctl restart docker
4.6下载cri-dockerd rpm包
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd-0.3.1-3.el7.x86_64.rpm
4.7安装rpm包
rpm -ivh cri-dockerd-0.3.1-3.el7.x86_64.rpm
4.8配置cri-docker服务依赖镜像地址
vi /usr/lib/systemd/system/cri-docker.service
注释以下配置项
#ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://
新增以下配置项
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
4.9加载配置项服务,并配置cri-docker开机自启
systemctl daemon-reload
systemctl start cri-docker && systemctl enable cri-docker
5.安装k8s管理软件
5.1配置k8s yum源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
5.2安装k8s管理软件
yum -y install kubelet-1.26.0 kubeadm-1.26.0 kubectl-1.26.0
systemctl enable kubelet
至此k8s基础环境已经配置完成,以上所有操作每台主机都要执行,也可以配置好一台主机后将环境复制,再更改一下即可
6.初始化k8s集群(只在master上执行)
kubeadm init \
--apiserver-advertise-address=10.1.60.119 \ #填master地址,配置集群控制节点的地址
--image-repository registry.aliyuncs.com/google_containers \ #配置镜像源的地址
--kubernetes-version v1.26.0 \ #配置k8s版本
--service-cidr=10.96.0.0/12 \ #集群内部主机的地址
--pod-network-cidr=10.244.0.0/16 \ #配置pod的网段,要与下面网络插件的地址一致
--cri-socket=unix:///var/run/cri-dockerd.sock \ #配置cri-dockerd接口
集群初始化完成后会输出一部分内容需要按照提示执行,我这里忘记截图了,执行的内容为以下步骤,需要执行完后节点才生效
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
执行完后查看节点状态
kubectl get nodes
此时已经可以查看到master节点,但是状态还是notready,这是因为还没有安装网络插件
这里额外补充一点知识,如果master节点在没有其它操作的情况下,要重新初始化,就先执行以下步骤
systemctl stop kubelet
netstat -tlpn #通过此命令找到etcd服务和apiserver服务的pid
kill -9 pid号 #这里输入刚刚找到的etcd服务和apiserver服务的pid,停掉服务
rm -rf /etc/kubernetes/* #删除k8s目录下的所有内容
rm -rf /var/lib/etcd/* #删除etcd目录下的所有内容
然后再重新执行集群初始化命令即可
7.将工作节点加入集群
将刚刚集群初始化的命令直接复制下来粘贴到node节点执行即可
要是找不到了也可以执行以下命令生成token
kubeadm token create --print-join-command
将生成的tonken加上cri-socket=unix:///var/run/cri-dockerd.sock参数指定容器引擎,如果不加上会报错
kubeadm join 10.1.60.124:16443 --token qc87b9.nxys4zjgvmq2bza4 --discovery-token-ca-cert-hash sha256:9636d912ddb2a9b1bdae085906c11f6839bcf060f8b9924132f6d82b8aaefecd --cri-socket unix:///var/run/cri-dockerd.sock
在所有工作节点上执行完成后可以在master节点执行以下命令查看
kubectl get nodes
8.安装网络插件
k8s集群的网络插件有calico、flannel等,这里使用使用flannel作为k8s集群的网络插件
flannel的yml文件不用更改内容,因为里面指定pod的网络地址段默认就是10.244.0.0/16,如果初始化k8s集群时指定的是其它地址段,将yml文件里的地址更改即可
下载flannel的yaml文件
wget https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
使用此yaml文件
kubectl create -f kube-flannel.yml
等flannel容器都运行,可通过以下命令查看
kubectl get pods -n kube-flannel
flannel都启动后查看集群状态就可以看到都已经是ready了
kubectl get nodes
如果下载kube-flannel.yml文件不成功也可以通过复制以下的内容自己创建一个
vi kube-flannel.yml
---
kind: Namespace
apiVersion: v1
metadata:
name: kube-flannel
labels:
k8s-app: flannel
pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: flannel
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- networking.k8s.io
resources:
- clustercidrs
verbs:
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: flannel
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: flannel
name: flannel
namespace: kube-flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-flannel
labels:
tier: node
k8s-app: flannel
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-flannel
labels:
tier: node
app: flannel
k8s-app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni-plugin
image: docker.io/flannel/flannel-cni-plugin:v1.1.2
#image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.2
command:
- cp
args:
- -f
- /flannel
- /opt/cni/bin/flannel
volumeMounts:
- name: cni-plugin
mountPath: /opt/cni/bin
- name: install-cni
image: docker.io/flannel/flannel:v0.21.5
#image: docker.io/rancher/mirrored-flannelcni-flannel:v0.21.5
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: docker.io/flannel/flannel:v0.21.5
#image: docker.io/rancher/mirrored-flannelcni-flannel:v0.21.5
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: EVENT_QUEUE_DEPTH
value: "5000"
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: xtables-lock
mountPath: /run/xtables.lock
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni-plugin
hostPath:
path: /opt/cni/bin
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate