centos7.9系统使用Kubeadm安装部署kubernetes1.23.1

搭建Kubernetes集群：从零开始到部署

最新推荐文章于 2025-09-19 16:22:37 发布

原创最新推荐文章于 2025-09-19 16:22:37 发布 · 1.7k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#linux

Docker 同时被 2 个专栏收录

75 篇文章

订阅专栏

K8S

48 篇文章

订阅专栏

一、机器情况

主机	ip	配置	操作系统
master	192.168.0.160	2c4g50G	centos7.8
node01	192.168.0.6	2c4g50G	centos7.8
node02	192.168.0.167	2c4g50G	centos7.8

二、机器设置

以下步骤需要在每个节点上执行

1、设置主机名

在各自节点上设置各自得主机名

hostnamectl set-hostname master

hostnamectl set-hostname node01

hostnamectl set-hostname node02

2、设置hosts

cat <<EOF >/etc/hosts

192.168.0.160 master

192.168.0.6 node01

192.168.0.167 node02

EOF

3、设置防火墙以及seliunx

关闭防火墙

systemctl stop firewalld

设置开机不启动

systemctl disable firewalld

关闭selinux

vi /etc/selinux/config

SELINUX=disabled

重启系统

reboot

4、关闭swap分区

vi /etc/fstab

#

# /etc/fstab

# Created by anaconda on Mon Jan 21 19:19:41 2019

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/centos-root / xfs defaults 0 0

UUID=214b916c-ad23-4762-b916-65b53fce1920 /boot xfs defaults 0 0

#/dev/mapper/centos-swap swap swap defaults 0 0

5、创建/etc/sysctl.d/k8s.conf文件，添加如下内容

cat <<EOF >/etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_forward = 1

vm.swappiness=0

EOF

#执行命令使修改生效

modprobe br_netfilter

sysctl -p /etc/sysctl.d/k8s.conf

6、kube-proxy开启ipvs的前置条件

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack_ipv4

EOF

加载模块

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

安装了ipset软件包

yum install ipset -y

安装管理工具ipvsadm

yum install ipvsadm -y

三、安装docker

以下步骤需要在每个节点上执行

1、设置阿里云docker yum源

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

如果yum-config-manager不能用，请安装yum-utils

yum -y install yum-utils

查看可安装Docker版本

yum list docker-ce.x86_64 --showduplicates |sort -r

2、安装docker

yum默认是安装最新版本，但是为了兼容性，这里就指定版本安装

1	`yum` `install` `-y --setopt=obsoletes=0 docker-ce-18.06.1.ce-3.el7`

3、设置docker的Cgroup Driver

cat <<EOF >/etc/docker/daemon.json

{

"exec-opts": ["native.cgroupdriver=systemd"]

}

EOF

4、启动docker设置开机启动

1	`systemctl start docker && systemctl` `enable` `docker`

四、使用kubeadm安装kubernetes

1-4需在所有节点执行，5-6在master节点上执行

1、配置yum源

vi /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

gpgcheck=1

enable=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg

https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

2、安装kubelet，kubeadm，kubectl

1	`yum makecache fast && yum` `install` `-y kubelet kubeadm kubectl`

3、修改kubelet的Cgroup Driver

cat <<EOF >/etc/sysconfig/kubelet

KUBELET_CGROUP_ARGS="--cgroup-driver=systemd"

KUBE_PROXY_MODE="ipvs"

EOF　

4、下载必要的镜像

[root@master ~]# kubeadm config images list

k8s.gcr.io/kube-apiserver:v1.23.1

k8s.gcr.io/kube-controller-manager:v1.23.1

k8s.gcr.io/kube-scheduler:v1.23.1

k8s.gcr.io/kube-proxy:v1.23.1

k8s.gcr.io/pause:3.6

k8s.gcr.io/etcd:3.5.1-0

k8s.gcr.io/coredns/coredns:v1.8.6 ##注意这里多了一级coredns

vi k8s.sh

docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.1 docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.1 docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.1 docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.23.1 docker pull registry.aliyuncs.com/google_containers/pause:3.6 docker pull registry.aliyuncs.com/google_containers/etcd:3.5.1-0 docker pull registry.aliyuncs.com/google_containers/coredns:v1.8.6

docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.1 k8s.gcr.io/kube-apiserver:v1.23.1 docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.1 k8s.gcr.io/kube-controller-manager:v1.23.1 docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.1 k8s.gcr.io/kube-scheduler:v1.23.1 docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.23.1 k8s.gcr.io/kube-proxy:v1.23.1 docker tag registry.aliyuncs.com/google_containers/pause:3.6 k8s.gcr.io/pause:3.6 docker tag registry.aliyuncs.com/google_containers/etcd:3.5.1-0 k8s.gcr.io/etcd:3.5.1-0 docker tag registry.aliyuncs.com/google_containers/coredns:v1.8.6 k8s.gcr.io/coredns:v1.8.6

docker rmi registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.1 docker rmi registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.1 docker rmi registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.1 docker rmi registry.aliyuncs.com/google_containers/kube-proxy:v1.23.1 docker rmi registry.aliyuncs.com/google_containers/pause:3.6 docker rmi registry.aliyuncs.com/google_containers/etcd:3.5.1-0 docker rmi registry.aliyuncs.com/google_containers/coredns:v1.8.6

5、初始化

kubeadm init \

--kubernetes-version=v1.23.1 \

--pod-network-cidr=10.244.0.0/16 \

--service-cidr=10.96.0.0/12 \

--apiserver-advertise-address=192.168.0.160 

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

 如果上面的参数填写错误，需要重新初始化需执行重置 kubeadm reset

kubernetes-version：修改要安装的版本

apiserver-advertise-address：master节点的ip地址

当出现如下信息表示成功

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

在nide01和node02节点上分别执行如下命令，加入集群

kubeadm join 192.168.0.160:6443 --token 57jle4.zbccddfk8d2su6pe \

--discovery-token-ca-cert-hash sha256:556eeec7a4d742155a785b90a6efaebd95c466ad939047d4ad90ccb55dc35418

6、部署

(1) Flannel

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml


kubectl apply -f  kube-flannel.yml 

#如需重新安装需要先删除所创建的网络配置
kubectl delete -f  kube-flannel.yml

(2) Calico

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

7、节点加入集群

在node01和node02节点上分别执行如下命令加入集群：

kubeadm join 192.168.0.160:6443 --token 57jle4.zbccddfk8d2su6pe \

--discovery-token-ca-cert-hash sha256:556eeec7a4d742155a785b90a6efaebd95c466ad939047d4ad90ccb55dc35418

8、查看状态

master节点上执行，如果都为Ready表示成功

kubectl get node

[root@master ~]# kubectl get node

NAME STATUS ROLES AGE VERSION

master Ready control-plane,master 27m v1.23.1

node01 Ready <none> 17m v1.23.1

node02 Ready <none> 17m v1.23.1

五、镜像地址

我已把相关镜像打包，放到云盘上了，有需要的自取

链接: https://pan.baidu.com/s/1XKN32WXiXmp6XKlsgw-xGw 提取码: q8hs

六、kubelet服务启动失败问题处理

1.问题重现

[root@leoheng-k8s ~]# systemctl enable kubelet && systemctl start kubelet
[root@leoheng-k8s ~]# systemctl status  kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-01-28 09:56:54 CST; 7s ago
     Docs: https://kubernetes.io/docs/
  Process: 2717 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 2717 (code=exited, status=255)

Jan 28 09:56:54 leoheng-k8s systemd[1]: Unit kubelet.service entered failed state.
Jan 28 09:56:54 leoheng-k8s systemd[1]: kubelet.service failed.

2.问题处理

1.关闭selinux、firewalld，并且把cgroup设置与docker一样

2.查看官方文档

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally.

3.执行k8s的初始化

[root@leoheng-k8s ~]# kubeadm init
[init] Using Kubernetes version: v1.20.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.2. Latest validated version: 19.03
	[WARNING Hostname]: hostname "leoheng-k8s" could not be reached
	[WARNING Hostname]: hostname "leoheng-k8s": lookup leoheng-k8s on 100.100.2.138:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local leoheng-k8s] and IPs [10.96.0.1 172.18.192.80]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [leoheng-k8s localhost] and IPs [172.18.192.80 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [leoheng-k8s localhost] and IPs [172.18.192.80 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.002755 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node leoheng-k8s as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node leoheng-k8s as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1mxqc0.zwbjh8m8g5d35foe
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.18.192.80:6443 --token 1mxqc0.zwbjh8m8g5d35foe \
    --discovery-token-ca-cert-hash sha256:09949a2800da77bd71d046080fe5a75662472a32d0070305ce1fc7457a642b2d

4.重新启动kubectl服务

[root@leoheng-k8s ~]# systemctl status  kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2021-01-28 09:58:49 CST; 2min 8s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 4088 (kubelet)
    Tasks: 13
   Memory: 37.8M
   CGroup: /system.slice/kubelet.service
           └─4088 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/...

Jan 28 10:00:34 leoheng-k8s kubelet[4088]: W0128 10:00:34.351824    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:36 leoheng-k8s kubelet[4088]: E0128 10:00:36.029903    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:39 leoheng-k8s kubelet[4088]: W0128 10:00:39.352002    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:41 leoheng-k8s kubelet[4088]: E0128 10:00:41.044753    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:44 leoheng-k8s kubelet[4088]: W0128 10:00:44.352167    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:46 leoheng-k8s kubelet[4088]: E0128 10:00:46.060000    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:49 leoheng-k8s kubelet[4088]: W0128 10:00:49.352345    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:51 leoheng-k8s kubelet[4088]: E0128 10:00:51.075055    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:54 leoheng-k8s kubelet[4088]: W0128 10:00:54.352524    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:56 leoheng-k8s kubelet[4088]: E0128 10:00:56.090169    4088 kubelet.go:2163] Container runtim...lized
Hint: Some lines were ellipsized, use -l to show in full.
[root@leoheng-k8s ~]#

七、添加节点失效couldn't validate the identity of the API Server 问题处理

报错信息如下

[root@k8s-node2 k8s]# kubeadm join 192.168.1.200:6443 --token ov6qse.lvw984yn30c96p9o     --discovery-token-ca-cert-hash sha256:ed7ea5ae0c06f4ace9013e663b223e8da72e4e94e4dc657cfb1db68d777f3984 
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.3. Latest validated version: 18.09
error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s

查看token

[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS

果然没有，生一个，再次查看，token生命周期为一天

[root@k8s-master ~]# kubeadm token create
wxvdun.vec7m9cu4ru3hngg
[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
wxvdun.vec7m9cu4ru3hngg   23h       2019-10-18T10:43:34+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
ed7ea5ae0c06f4ace9013e663b223e8da72e4e94e4dc657cfb1db68d777f3984

node节点上重新加入

[root@k8s-node2 ~]# kubeadm join 192.168.1.200:6443 --token wxvdun.vec7m9cu4ru3hngg     --discovery-token-ca-cert-hash sha256:ed7ea5ae0c06f4ace9013e663b223e8da72e4e94e4dc657cfb1db68d777f3984

指定两个地方，token名和sha256

八、conflicts with file from package问题处理

1、因为之前已经安装过旧版本的docker，在安装的时候报错如下：

Transaction check error:
  file /usr/bin/docker from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/bash-completion/completions/docker from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/fish/vendor_completions.d/docker.fish from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/man/man1/docker-attach.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/man/man1/docker-checkpoint-create.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/man/man1/docker-checkpoint-ls.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/man/man1/docker-checkpoint-rm.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64
  file /usr/share/man/man1/docker-checkpoint.1.gz from install of docker-ce-18.06.3.ce-3.el7.x86_64 conflicts with file from package docker-ce-cli-1:19.03.5-3.el7.x86_64

2、卸载旧版本的包。如上图所示，可知需要卸载 docker-ce-cli-1:19.03.5-3.el7.x86_64

sudo yum erase docker-ce-cli-1:19.03.5-3.el7.x86_64