kubernetes 1.25.0 版本集群部署+containerd运行时

最新推荐文章于 2025-06-03 14:42:27 发布

fittec_vb

最新推荐文章于 2025-06-03 14:42:27 发布

阅读量6k

点赞数

CC 4.0 BY-SA版权

文章标签： kubernetes docker 容器

本文链接：https://blog.youkuaiyun.com/fittec_vb/article/details/126666681

kubernetes1.25.0 版本j集群部署+containerd运行时前言
kubeadm 是 Kubernetes 官方提供的用于快速安部署 Kubernetes 集群的工具，伴随 Kubernetes 每个版本的发布都会同步更新，kubeadm 会对集群配置方面的一些实践做调整，通过实验 kubeadm 可以学习到 Kubernetes 官方在集群配置上一些新的最佳实践。
kubernetes 1.24版本后正式弃用docker,开始使用containerd作为容器运行时。

一、准备
1.1、系统配置
在安装之前，需要先做好如下准备。3 台 CentOS 7.9 主机如下：

cat /etc/hosts
192.168.197.121 master
192.168.197.122 node01
192.168.197.123 node02
1.2升级内核（该步骤可以省略）导入elrepo gpg key

rpm -import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

安装 elrepo YUM源仓库

yum install -y https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本

yum --enablerepo=“elrepo-kernel” -y install kernel-ml.x86_64

设置grub2默认引导为0

grub2-set-default 0

重新生成grub2引导文件

grub2-mkconfig -o /boot/grub2/grub.cfg

更新后，需要重启，使升级的内核生效

reboot

重启后执行 uname -ar 查看kernel 版本

[root@master ~]# uname -ar
Linux master 5.19.6-1.el7.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 29 10:06:03 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

1.3 模块安装

在各个主机上完成下面的系统配置。如果各个主机启用了防火墙策略，需要开放 Kubernetes 各个组件所需要的端口，可以查看 Ports and Protocols 中的内容, 开放相关端口或者关闭主机的防火墙。
禁用SELINUX：
临时禁用：

setenforce 0

永久生效：

vi /etc/selinux/config
SELINUX=disabled

创建 /etc/modules-load.d/containerd.conf 配置文件:

cat << EOF > /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

执行以下命令使配置生效:

modprobe overlay
modprobe br_netfilter

创建 /etc/sysctl.d/99-kubernetes-cri.conf 配置文件：

cat << EOF > /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
user.max_user_namespaces=28633
EOF

执行以下命令使配置生效:

sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf1.4 配置服务器支持开启ipvs的前提条件
由于 ipvs 已经加入到了内核的主干，所以为 kube-proxy 开启 ipvs 的前提需要加载以下的内核模块：
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
在各个服务器节点上执行以下脚本:

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe – ip_vs
modprobe – ip_vs_rr
modprobe – ip_vs_wrr
modprobe – ip_vs_sh
modprobe – nf_conntrack_ipv4
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
上面脚本创建了的 /etc/sysconfig/modules/ipvs.modules 文件，保证在节点重启后能自动加载所需模块。使用 lsmod | grep -e ip_vs -e nf_conntrack_ipv4 命令查看是否已经正确加载所需的内核模块。接下来还需要确保各个节点上已经安装了 ipset 软件包，为了便于查看 ipvs 的代理规则，最好安装一下管理工具 ipvsadm。

yum install -y ipset ipvsadm
如果不满足以上前提条件，则即使 kube-proxy 的配置开启了 ipvs 模式，也会退回到 iptables 模式。

1.5部署容器运行时 Containerd
在各个服务器节点上安装容器运行时 Containerd。下载Containerd的二进制包:

wget https://github.com/containerd/containerd/releases/download/v1.6.8/cri-containerd-cni-1.6.8-linux-amd64.tar.gz
cri-containerd-cni-1.6.8-linux-amd64.tar.gz 压缩包中已经按照官方二进制部署推荐的目录结构布局好。里面包含了 systemd 配置文件，containerd 以及 cni 的部署文件。将解压缩到系统的根目录 / 中:

tar -zxvf cri-containerd-cni-1.6.8-linux-amd64.tar.gz -C /
etc/
etc/systemd/
etc/systemd/system/
etc/systemd/system/containerd.service
etc/crictl.yaml
etc/cni/
etc/cni/net.d/
etc/cni/net.d/10-containerd-net.conflist
usr/
usr/local/
usr/local/sbin/
usr/local/sbin/runc
usr/local/bin/
usr/local/bin/critest
usr/local/bin/containerd-shim
usr/local/bin/containerd-shim-runc-v1
usr/local/bin/ctd-decoder
usr/local/bin/containerd
usr/local/bin/containerd-shim-runc-v2
usr/local/bin/containerd-stress
usr/local/bin/ctr
usr/local/bin/crictl
…
opt/cni/
opt/cni/bin/
opt/cni/bin/bridge
…
注意经测试 cri-containerd-cni-1.6.8-linux-amd64.tar.gz 包中包含的 runc 在 CentOS 7 下的动态链接有问题，这里从 runc 的 github 上单独下载 runc，并替换上面安装的 containerd 中的 runc:

wget https://github.com/opencontainers/runc/releases/download/v1.1.2/runc.amd64
cp runc.amd64 /usr/bin/runc
chmod a+x /usr/bin/runc
接下来生成 containerd 的配置文件:

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
根据文档 Container runtimes 中的内容，对于使用 systemd 作为 init system 的 Linux 的发行版，使用 systemd 作为容器的 cgroup driver 可以确保服务器节点在资源紧张的情况更加稳定，因此这里配置各个节点上 containerd 的 cgroup driver 为 systemd。修改前面生成的配置文件 /etc/containerd/config.toml：

[plugins.“io.containerd.grpc.v1.cri”.containerd.runtimes.runc]
…
[plugins.“io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]
SystemdCgroup = true
再修改 /etc/containerd/config.toml 中的

[plugins.“io.containerd.grpc.v1.cri”]
…

sandbox_image = “k8s.gcr.io/pause:3.6”

sandbox_image = “registry.aliyuncs.com/google_containers/pause:3.7”
配置 containerd 开机启动，并启动 containerd

systemctl enable containerd --now
使用 crictl 测试一下，确保可以打印出版本信息并且没有错误信息输出:

[root@master ~]# crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: v1.6.8
RuntimeApiVersion: v1
二、使用 kubeadm 部署 Kubernetes
2.1、安装 kubeadm 和 kubelet
下面在各节点安装 kubeadm 和 kubelet：

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum makecache fast
yum install kubelet kubeadm kubectl
运行 kubelet --help 可以看到原来 kubelet 的绝大多数命令行 flag 参数都被 DEPRECATED 了，官方推荐我们使用 --config 指定配置文件，并在配置文件中指定原来这些 flag 所配置的内容。具体内容可以查看这里 Set Kubelet parameters via a config file。最初 Kubernetes 这么做是为了支持动态 Kubelet 配置（Dynamic Kubelet Configuration），但动态 Kubelet 配置特性从 k8s 1.22 中已弃用，并在 1.24 中被移除。如果需要调整集群汇总所有节点 kubelet 的配置，还是推荐使用 ansible 等工具将配置分发到各个节点。kubelet 的配置文件必须是 json 或 yaml 格式，具体可查看这里。Kubernetes 1.8 开始要求关闭系统的 Swap，如果不关闭，默认配置下 kubelet 将无法启动。关闭系统的 Swap 方法如下:

swapoff -a
修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用 free -m 确认 swap 已经关闭。swappiness 参数调整，修改 /etc/sysctl.d/99-kubernetes-cri.conf 添加下面一行：

vm.swappiness=0
执行 sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf 使修改生效。

2.2、使用 kubeadm init 初始化集群

在各节点开机启动 kubelet 服务：

systemctl enable kubelet.service

使用 kubeadm config print init-defaults --component-configs KubeletConfiguration 可以打印集群初始化默认的使用的配置：

[root@master ~]# kubeadm config print init-defaults --component-configs KubeletConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:

groups:
- system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
- signing
- authentication
  kind: InitConfiguration
  localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
  nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null

apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.25.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:

10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: “0”
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
从默认的配置中可以看到，可以使用 imageRepository 定制在集群初始化时拉取 k8s 所需镜像的地址。基于默认配置定制出本次使用 kubeadm 初始化集群所需的配置文件 kubeadm.yaml：

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.197.121
bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
taints:

effect: PreferNoSchedule
key: node-role.kubernetes.io/master

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
imageRepository: registry.aliyuncs.com/google_containers
networking:
podSubnet: 10.244.0.0/16

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: false

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
这里定制了 imageRepository 为阿里云的 registry，避免因 gcr 被墙，无法直接拉取镜像。criSocket 设置了容器运行时为 containerd。同时设置 kubelet 的 cgroupDriver 为 systemd，设置 kube-proxy 代理模式为 ipvs。在开始初始化集群之前可以使用 kubeadm config images pull --config kubeadm.yaml 预先在各个服务器节点上拉取所 k8s 需要的容器镜像。

[root@master ~]# kubeadm config images pull --config kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.25.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.8
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.4-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.9.3

接下来使用 kubeadm 初始化集群，在 master 上执行下面的命令：

[root@master ~]# kubeadm init
–apiserver-advertise-address=192.168.197.121
–image-repository registry.cn-hangzhou.aliyuncs.com/google_containers
–kubernetes-version v1.25.0
–service-cidr=10.96.0.0/16
–pod-network-cidr=10.244.0.0/16 --service-dns-domain=cluster.local
[init] Using Kubernetes version: v1.25.0
[preflight] Running pre-flight checks
[WARNING Service-Kubelet]: kubelet service is not enabled, please run ‘systemctl enable kubelet.service’
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “ca” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.197.121]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-ca” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] Generating “etcd/ca” certificate and key
[certs] Generating “etcd/server” certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.197.121 127.0.0.1 ::1]
[certs] Generating “etcd/peer” certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.197.121 127.0.0.1 ::1]
[certs] Generating “etcd/healthcheck-client” certificate and key
[certs] Generating “apiserver-etcd-client” certificate and key
[certs] Generating “sa” key and public key
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “kubelet.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
[control-plane] Creating static Pod manifest for “kube-scheduler”
[etcd] Creating static Pod manifest for local etcd in “/etc/kubernetes/manifests”
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s
[apiclient] All control plane components are healthy after 10.506645 seconds
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[kubelet] Creating a ConfigMap “kubelet-config” in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: j7g45q.patwp0rg7372kwl7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the “cluster-info” ConfigMap in the “kube-public” namespace
[kubelet-finalize] Updating “/etc/kubernetes/kubelet.conf” to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(i d - u) :$ (id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.197.121:6443 --token j7g45q.patwp0rg7372kwl7
–discovery-token-ca-cert-hash sha256:97851417b6d607b9ca1ebe202691e1d5e0b909c5785fa4530c831c0b877bf9c1

上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个 Kubernetes 集群所需要的关键步骤。其中有以下关键内容：

[certs]生成相关的各种证书
[kubeconfig]生成相关的kubeconfig文件
[kubelet-start] 生成kubelet的配置文件"/var/lib/kubelet/config.yaml"
[control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod
[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
下面的命令是配置常规用户如何使用kubectl访问集群：mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(i d - u) :$ (id -g) $HOME/.kube/config
最后给出了将节点加入集群的命令kubeadm join 192.168.197.121:6443 --token j7g45q.patwp0rg7372kwl7
–discovery-token-ca-cert-hash sha256:97851417b6d607b9ca1ebe202691e1d5e0b909c5785fa4530c831c0b877bf9c1

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(i d - u) :$ (id -g) $HOME/.kube/config
查看一下集群状态，确认个组件都处于 healthy 状态，

[root@master ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {“health”:“true”,“reason”:“”}
node01，node02 节点加入集群：

kubeadm join 192.168.197.121:6443 --token j7g45q.patwp0rg7372kwl7
–discovery-token-ca-cert-hash sha256:97851417b6d607b9ca1ebe202691e1d5e0b909c5785fa4530c831c0b877bf9c1

安装网络插件

我们有两种选择，一个是Flannel，另一个是Calico

由CoreOS开发的项目Flannel，可能是最直接和最受欢迎的CNI插件。它是容器编排系统中最成熟的网络结构示例之一，旨在实现更好的容器间和主机间网络。许多常见的Kubernetes集群部署工具和许多Kubernetes发行版都可以默认安装Flannel。

Calico是Kubernetes生态系统中另一种流行的网络选择。虽然Flannel被公认为是最简单的选择，但Calico以其性能、灵活性而闻名。Calico的功能更为全面，不仅提供主机和pod之间的网络连接，还涉及网络安全和管理。Calico CNI插件在CNI框架内封装了Calico的功能。

此处我们使用Flannel插件。

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
下载后需要修改里面的配置，首先设置一下网卡

其次需要设置一下网段，这里的网段和 podSubnet的值保持一致，都是10.244.0.0/16

执行命令

[root@master ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

集群验证
通过kubectl命名可以看到个节点处于ready状态[root@master ~]# kubectl get nodeNAME STATUS ROLES AGE VERSIONmaster Ready control-plane 5m35s v1.25.0node01 Ready 17s v1.25.0node02 Ready 22s v1.25.0[root@master ~]# kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME STATUS MESSAGE ERRORscheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {“health”:“true”,“reason”:“”} 再执行一下如下命令，查看是否所有的pod都是运行正常的：kubectl get pod --all-namespaces -o wide 发现coredns没有启动成功。主要是CNI版本兼容性问题，通过日志查看：[root@master ~]# kubectl describe pods -n kube-system coredns-7f8cbcb969-r69wvName: coredns-7f8cbcb969-r69wvNamespace: kube-systemPriority: 2000000000Priority Class Name: system-cluster-criticalService Account: corednsNode: node02/192.168.197.123Start Time: Fri, 02 Sep 2022 09:30:06 +0800Labels: k8s-app=kube-dns pod-template-hash=7f8cbcb969Annotations: Status: PendingIP: IPs: Controlled By: ReplicaSet/coredns-7f8cbcb969Containers: coredns: Container ID: Image: registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.9.3 Image ID: Ports: 53/UDP, 53/TCP, 9153/TCP Host Ports: 0/UDP, 0/TCP, 0/TCP Args: -conf /etc/coredns/Corefile State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: Mounts: /etc/coredns from config-volume (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rrdrb (ro)Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled TrueVolumes: config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: coredns Optional: false kube-api-access-rrdrb: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: trueQoS Class: BurstableNode-Selectors: kubernetes.io/os=linuxTolerations: CriticalAddonsOnly op=Exists node-role.kubernetes.io/control-plane:NoSchedule node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300sEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 28m default-scheduler Successfully assigned kube-system/coredns-7f8cbcb969-r69wv to node02 Warning FailedCreatePodSandBox 28m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “5d41f76feb99a937579605a32400baac4888671d09c12f277aad8dbc7ba32129”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 28m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “63b6bbd98cf5ec935af33da68155e2740d338fd5573707b1508f5ab06e0ba824”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 28m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “c009bc7881c4190112b6d73aae3064810bbcfa2662512aff289837297bc11194”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 28m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “de3d98b4a06b2f74ea5f281cc6e93b85ed6d3b99df42adf23438d4e0e76d25db”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 28m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “af4cd679f594bc9c4a4d342aa1487f430ffda420f767b40bd33b80db46a86f2b”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 27m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “853d961d8d995dfbaccac0cab38a691c11ecfd25ce75868f129f9dc24cd1757c”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 27m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “a6e33d662249e7816b92b62fa6271076222907d3b2ec22ef2f52c7373efa270b”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 27m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “e58fc595d7847488497758cb2ca3136b3369616149c0715f3f3987e5584cfa0c”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 27m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “b67194953dbe57ba55048275f4215765111bfc65ab148731fc886ff027bbc1d0”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”] Warning FailedCreatePodSandBox 3m57s (x107 over 27m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “b7d02764459095c6611014c38b4897ed222ef9e65b4b625e0ae6554d21052393”: plugin type=“bridge” failed (add): incompatible CNI versions; config is “1.0.0”, plugin supports [“0.1.0” “0.2.0” “0.3.0” “0.3.1” “0.4.0”][root@master ~]#这里可以把containerd中的CNI去除掉，所有节点都操作。mv /etc/cni/net.d/10-containerd-net.conflist /etc/cni/net.d/10-containerd-net.conflist.baksystemctl daemon-reloadsystemctl restart containerd kubelet再次验证正常了：kubectl get pod --all-namespaces -o wide验证DNS解析创建ndsutils.yaml[root@master ~]# cat ndsutils.yamlapiVersion: v1kind: Podmetadata: name: dnsutilsspec: containers: - name: dnsutils image: mydlqclub/dnsutils:1.3 imagePullPolicy: IfNotPresent command: [“sleep”,“3600”]kubectl create -f ndsutils.yaml验证一下状态：[root@master ~]# kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESdnsutils 1/1 Running 2 (19m ago) 157m 10.244.1.3 node02 验证 k8s DNS 是否可用[root@master ~]# kubectl run curl --image=radial/busyboxplus:curl -itIf you don’t see a command prompt, try pressing enter.[ root@curl:/进入后执行 nslookup kubernetes.default 确认解析正常:[ root@curl:/ ]$ nslookup kubernetes.defaultServer: 10.96.0.10Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localName: kubernetes.defaultAddress 1: 10.96.0.1 kubernetes.default.svc.cluster.local

三、Kubernetes 常用组件部署
3.1、部署 ingress-nginx
为了便于将集群中的服务暴露到集群外部，需要使用 Ingress。

3.2部署dashboard
下载配置文件(需要翻墙)

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml
1修改配置文件
更改镜像地址和dashboard service暴露外网端口
vim dashboard.yaml

apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard

apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard

kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard

apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque

apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: “”

apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque

kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:

Allow Dashboard to get, update and delete Dashboard exclusive secrets.

apiGroups: [“”]
resources: [“secrets”]
resourceNames: [“kubernetes-dashboard-key-holder”, “kubernetes-dashboard-certs”, “kubernetes-dashboard-csrf”]
verbs: [“get”, “update”, “delete”]
Allow Dashboard to get and update ‘kubernetes-dashboard-settings’ config map.
apiGroups: [“”]
resources: [“configmaps”]
resourceNames: [“kubernetes-dashboard-settings”]
verbs: [“get”, “update”]
Allow Dashboard to get metrics.
apiGroups: [“”]
resources: [“services”]
resourceNames: [“heapster”, “dashboard-metrics-scraper”]
verbs: [“proxy”]
apiGroups: [“”]
resources: [“services/proxy”]
resourceNames: [“heapster”, “http:heapster:”, “https:heapster:”, “dashboard-metrics-scraper”, “http:dashboard-metrics-scraper”]
verbs: [“get”]

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:

Allow Metrics Scraper to get metrics from the Metrics server

apiGroups: [“metrics.k8s.io”]
resources: [“pods”, “nodes”]
verbs: [“get”, “list”, “watch”]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:

kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:

kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard

kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: kubernetes-dashboard
image: registry.cn-hangzhou.aliyuncs.com/wuxingge/dashboard:v2.5.1
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
“kubernetes.io/os”: linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule

kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper

kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: dashboard-metrics-scraper
image: registry.cn-hangzhou.aliyuncs.com/wuxingge/metrics-scraper:v1.0.7
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
“kubernetes.io/os”: linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}

创建dashboard访问token

#创建dashboard管理用户
[root@master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created

#绑定集群角色
[root@master ~]# kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-cluster-admin created

#查看sa是否创建成功
[root@master ~]# kubectl -n kube-system get serviceaccounts |grep dashboard
dashboard-admin 0 32s
#创建token
可以设置token过期时间 --duration 参数
[root@master ~]# kubectl -n kube-system create token dashboard-admin
eyJhbGciOiJSUzI1NiIsImtpZCI6IkJIazJ3NUpfMlN4V3lMOWxmMmR0UHFyNDhldUFFNlM0Wm9NR21ja015U00ifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjYyMTAzNTkyLCJpYXQiOjE2NjIwOTk5OTIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkYXNoYm9hcmQtYWRtaW4iLCJ1aWQiOiIyNjA3ODg3Yy0xY2FjLTQzYWYtYjUwOC1iOWI4YjliOWVkZDAifX0sIm5iZiI6MTY2MjA5OTk5Miwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.A4fIYX57k08G3O_VIm3mTGg1EazuBSPOSQS55qQhly6uunGs4yiy-QAeji8HQJvPCs1J0WMlIW890Qwj3w29NZB2nJZEy7BiJVSTcdQkSAsrjx4cdIMz9AQDqBXPrpOdGubpggAe5AP78F3St2TlnBbJUYfbyG34Q7gjByYvFl-BRhbkfPFl6PCQmP68YmrSKGwe-IlVbaNUdI5uxV99vVvIXM2lQSY9Tp4nGUA7sqou7Pgi5Xkx36sqCqnZpf-am_GwCJnH6Sh4dmcLWcy95cIxW7D4iF9jEI1DtcbB8eKjFGaXpZmSVWWdtFbCC-PlJlH0nUHsEX2RfYM8s0sdfg
登录验证：http://192.168.197.121:3001