kubeadm搭建k8s1.26.0版本集群

1.前言

本次搭建一主两从的k8s集群，k8s使用的版本为1.26.0版本，选用docker作为容器引擎，因为k8s1.24.0版本之后不适配docker的原因，需要用到cri-docker去让k8s适配docker，也可以选用containerd作为容器引擎，使用containerd作为容器引擎就不会有不适配的问题，系统使用centos7.9版本

2.集群信息

主机名称	IP	SERVICE
k8s-master01	10.1.60.119	kubectl、kubelet、kubeadm、docker、cri-docker
k8s-node01	10.1.60.120	kubectl、kubelet、kubeadm、docker、cri-docker
k8s-node02	10.1.60.121	kubectl、kubelet、kubeadm、docker、cri-docker

3.系统基础环境配置(所有主机均要配置)

3.1关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

3.2关闭selinux服务

setenforce 0  #临时关闭

sed -i 's/enforcing/disabled/' /etc/selinux/config #永久关闭

3.3关闭系统交换分区

swapoff -a #临时关闭

sed -ri 's/.*swap.*/#&/' /etc/fstab  #永久关闭

3.4设置主机名称(三台主机名称不一样按照表格上的主机名填)

hostnamectl set-hostname k8s-master01

3.5编辑hosts文件

cat > /etc/hosts << EOF

10.1.60.119 k8s-master01

10.1.60.120 k8s-node01

10.1.60.121 k8s-node02

EOF

3.6配置时间同步

yum -y install chrony

systemctl strat chronyd && systemctl enbale chronyd

3.7配置ipvs

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack_ipv4

EOF

#授权并生效ipvs配置

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules

yum install -y ipset ipvsadm

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

 sudo modprobe overlay && sudo modprobe br_netfilter

3.8将桥接的IPv4流量传递到iptables

cat > /etc/sysctl.d/k8s.conf << EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_forward = 1

EOF

 sysctl --system  #配置生效

4.安装docker服务

4.1下载docker yum源

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

 4.2安装docker服务

yum -y install docker-ce

4.3启用docker并配置开机启动

systemctl start docker && systemctl enable docker

4.4配置docker镜像加速

cat > /etc/docker/daemon.json << EOF

{

  "registry-mirrors": ["https://sudzwtcw.mirror.aliyuncs.com"],

  "exec-opts": ["native.cgroupdriver=systemd"]

}

EOF

 4.5重启docker服务

systemctl restart docker

4.6下载cri-dockerd rpm包

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd-0.3.1-3.el7.x86_64.rpm

 4.7安装rpm包

rpm -ivh cri-dockerd-0.3.1-3.el7.x86_64.rpm

4.8配置cri-docker服务依赖镜像地址

vi /usr/lib/systemd/system/cri-docker.service

注释以下配置项
#ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://
新增以下配置项
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7

4.9加载配置项服务，并配置cri-docker开机自启

systemctl daemon-reload

systemctl start cri-docker && systemctl enable cri-docker

5.安装k8s管理软件

5.1配置k8s yum源

cat > /etc/yum.repos.d/kubernetes.repo << EOF

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

5.2安装k8s管理软件

yum -y install kubelet-1.26.0 kubeadm-1.26.0 kubectl-1.26.0

systemctl enable kubelet

至此k8s基础环境已经配置完成，以上所有操作每台主机都要执行，也可以配置好一台主机后将环境复制，再更改一下即可

6.初始化k8s集群(只在master上执行)

kubeadm init \

--apiserver-advertise-address=10.1.60.119 \   #填master地址，配置集群控制节点的地址

--image-repository registry.aliyuncs.com/google_containers \   #配置镜像源的地址

--kubernetes-version v1.26.0 \       #配置k8s版本

--service-cidr=10.96.0.0/12 \        #集群内部主机的地址

--pod-network-cidr=10.244.0.0/16 \     #配置pod的网段，要与下面网络插件的地址一致

--cri-socket=unix:///var/run/cri-dockerd.sock \   #配置cri-dockerd接口

 集群初始化完成后会输出一部分内容需要按照提示执行，我这里忘记截图了，执行的内容为以下步骤，需要执行完后节点才生效

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

export KUBECONFIG=/etc/kubernetes/admin.conf

执行完后查看节点状态

kubectl get nodes

此时已经可以查看到master节点，但是状态还是notready，这是因为还没有安装网络插件

这里额外补充一点知识，如果master节点在没有其它操作的情况下，要重新初始化，就先执行以下步骤

systemctl stop kubelet

netstat -tlpn     #通过此命令找到etcd服务和apiserver服务的pid

kill -9   pid号   #这里输入刚刚找到的etcd服务和apiserver服务的pid，停掉服务

rm -rf /etc/kubernetes/*   #删除k8s目录下的所有内容

rm -rf /var/lib/etcd/*     #删除etcd目录下的所有内容

然后再重新执行集群初始化命令即可

7.将工作节点加入集群

将刚刚集群初始化的命令直接复制下来粘贴到node节点执行即可

要是找不到了也可以执行以下命令生成token

kubeadm token create --print-join-command

将生成的tonken加上cri-socket=unix:///var/run/cri-dockerd.sock参数指定容器引擎，如果不加上会报错

kubeadm join 10.1.60.124:16443 --token qc87b9.nxys4zjgvmq2bza4 --discovery-token-ca-cert-hash sha256:9636d912ddb2a9b1bdae085906c11f6839bcf060f8b9924132f6d82b8aaefecd  --cri-socket unix:///var/run/cri-dockerd.sock

在所有工作节点上执行完成后可以在master节点执行以下命令查看

kubectl get nodes

8.安装网络插件

k8s集群的网络插件有calico、flannel等，这里使用使用flannel作为k8s集群的网络插件

flannel的yml文件不用更改内容，因为里面指定pod的网络地址段默认就是10.244.0.0/16,如果初始化k8s集群时指定的是其它地址段，将yml文件里的地址更改即可

下载flannel的yaml文件

wget https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml

使用此yaml文件

kubectl create -f kube-flannel.yml

等flannel容器都运行，可通过以下命令查看

kubectl get pods -n kube-flannel

flannel都启动后查看集群状态就可以看到都已经是ready了

kubectl get nodes

如果下载kube-flannel.yml文件不成功也可以通过复制以下的内容自己创建一个

vi kube-flannel.yml

---
kind: Namespace
apiVersion: v1
metadata:
  name: kube-flannel
  labels:
    k8s-app: flannel
    pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: flannel
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
- apiGroups:
  - networking.k8s.io
  resources:
  - clustercidrs
  verbs:
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: flannel
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: flannel
  name: flannel
  namespace: kube-flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-flannel
  labels:
    tier: node
    k8s-app: flannel
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-flannel
  labels:
    tier: node
    app: flannel
    k8s-app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: docker.io/flannel/flannel-cni-plugin:v1.1.2
       #image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.2
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: docker.io/flannel/flannel:v0.21.5
       #image: docker.io/rancher/mirrored-flannelcni-flannel:v0.21.5
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: docker.io/flannel/flannel:v0.21.5
       #image: docker.io/rancher/mirrored-flannelcni-flannel:v0.21.5
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: xtables-lock
          mountPath: /run/xtables.lock
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate