kubernetes集群的服务器重启之后遇到的问题

最新推荐文章于 2024-12-28 22:11:37 发布

原创最新推荐文章于 2024-12-28 22:11:37 发布 · 4k 阅读

12 ·

CC 4.0 BY-SA版权

文章标签：

#kubernetes #java

技术博客专栏收录该内容

28 篇文章

订阅专栏

本文详细介绍了在Kubernetes集群服务器重启后遇到的一系列问题及解决方法，包括kubelet进程启动失败、交换空间未关闭导致的问题，以及如何正确创建容器实例等。通过关闭交换空间和使用正确的命令，成功恢复了集群的状态。

kubernetes集群部署见我的上一篇博客。部署kubernetes集群的服务器重启之后，执行kubectl get nodes报错：

[root@k8s-master ~]# kubectl get pods
The connection to the server [master_ip]:6443 was refused - did you specify the right host or port?

查询进程，发现没有kube的进程启动：

[root@k8s-master ~]# ps -aux | grep "kube"
root      2911  0.0  0.0 112728   972 pts/0    R+   21:40   0:00 grep --color=auto kube

启动kubelet进程：

 systemctl start kubelet

再去查看kubectl get pods，发现还是报上面那个错，用journalctl -xeu kubelet这个命令查看，发现里面有报错。

[root@k8s-master ~]# journalctl -xeu kubelet
5月 25 22:01:59 k8s-master kubelet[6066]: I0525 22:01:59.542680    6066 plugins.go:100] No cloud provider specified.
5月 25 22:01:59 k8s-master kubelet[6066]: I0525 22:01:59.542691    6066 server.go:837] Client rotation is on, will bootstrap in
5月 25 22:01:59 k8s-master kubelet[6066]: I0525 22:01:59.544221    6066 certificate_store.go:130] Loading cert/key pair from "/
5月 25 22:01:59 k8s-master kubelet[6066]: W0525 22:01:59.544845    6066 container_manager_linux.go:912] CPUAccounting not enabl
5月 25 22:01:59 k8s-master kubelet[6066]: W0525 22:01:59.544855    6066 container_manager_linux.go:915] MemoryAccounting not en
5月 25 22:01:59 k8s-master kubelet[6066]: W0525 22:01:59.544888    6066 container_manager_linux.go:912] CPUAccounting not enabl
5月 25 22:01:59 k8s-master kubelet[6066]: W0525 22:01:59.544892    6066 container_manager_linux.go:915] MemoryAccounting not en
5月 25 22:01:59 k8s-master kubelet[6066]: I0525 22:01:59.568642    6066 server.go:646] --cgroups-per-qos enabled, but --cgroup-
5月 25 22:01:59 k8s-master kubelet[6066]: F0525 22:01:59.568815    6066 server.go:274] failed to run Kubelet: running with swap
5月 25 22:01:59 k8s-master systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
5月 25 22:01:59 k8s-master systemd[1]: Unit kubelet.service entered failed state.
5月 25 22:01:59 k8s-master systemd[1]: kubelet.service failed.
5月 25 22:02:09 k8s-master systemd[1]: kubelet.service holdoff time over, scheduling restart.
5月 25 22:02:09 k8s-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished shutting down.
5月 25 22:02:09 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
5月 25 22:02:09 k8s-master kubelet[6089]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config
5月 25 22:02:09 k8s-master kubelet[6089]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config
5月 25 22:02:09 k8s-master kubelet[6089]: I0525 22:02:09.816728    6089 server.go:417] Version: v1.18.3
5月 25 22:02:09 k8s-master kubelet[6089]: I0525 22:02:09.817028    6089 plugins.go:100] No cloud provider specified.
5月 25 22:02:09 k8s-master kubelet[6089]: I0525 22:02:09.817041    6089 server.go:837] Client rotation is on, will bootstrap in
5月 25 22:02:09 k8s-master kubelet[6089]: I0525 22:02:09.820725    6089 certificate_store.go:130] Loading cert/key pair from "/
5月 25 22:02:09 k8s-master kubelet[6089]: W0525 22:02:09.821336    6089 container_manager_linux.go:912] CPUAccounting not enabl
5月 25 22:02:09 k8s-master kubelet[6089]: W0525 22:02:09.821346    6089 container_manager_linux.go:915] MemoryAccounting not en
5月 25 22:02:09 k8s-master kubelet[6089]: W0525 22:02:09.821381    6089 container_manager_linux.go:912] CPUAccounting not enabl
5月 25 22:02:09 k8s-master kubelet[6089]: W0525 22:02:09.821385    6089 container_manager_linux.go:915] MemoryAccounting not en
5月 25 22:02:09 k8s-master kubelet[6089]: I0525 22:02:09.844321    6089 server.go:646] --cgroups-per-qos enabled, but --cgroup-
5月 25 22:02:09 k8s-master kubelet[6089]: F0525 22:02:09.844491    6089 server.go:274] failed to run Kubelet: running with swap
5月 25 22:02:09 k8s-master systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
5月 25 22:02:09 k8s-master systemd[1]: Unit kubelet.service entered failed state.
5月 25 22:02:09 k8s-master systemd[1]: kubelet.service failed.

又是没有关闭交换空间的问题，关闭交换空间：swapoff -a.找个方法永久关闭交换空间，注释掉最后那一行即可。

#
# /etc/fstab
# Created by anaconda on Tue Oct 29 23:56:12 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=3231dabf-3131-4282-a18f-71f680af55d3 /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

现在执行命令systemctl start kubelet，可以看到master已经ok了：

[root@k8s-master ~]# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
k8s-master    Ready      master   34h   v1.18.3
k8s-node-01   NotReady   <none>   34h   v1.18.3
k8s-node-02   NotReady   <none>   34h   v1.18.3

对node节点执行一下命令：

swapoff -a
systemctl start kubelet

可以看到node节点也Ready了，最后别忘了永久关闭交换空间，以及设置kubelet开机时启动：

systemctl enable kubelet

运行第一个容器实例,发现报错：大概意思是replicas这个参数现在以及不支持了。

[root@k8s-master ~]# kubectl run my-first-nginx --image=nginx --replicas=2 --port=80
Flag --replicas has been deprecated, has no effect and will be removed in the future.
pod/my-first-nginx created

查看发现pod被创建了，但是deployment没有创建：

[root@k8s-master ~]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
my-first-nginx   1/1     Running   0          10m

[root@k8s-master ~]# kubectl get deployment
No resources found in default namespace.

解决方案就是用kubectl create代替这个kubectl run.
如何删除pod: