Kubernetes 主节点宕机恢复记录 MountVolume.SetUp failed for volume "kube-dns-config"

最新推荐文章于 2025-04-22 10:18:30 发布

shida_csdn

最新推荐文章于 2025-04-22 10:18:30 发布

阅读量2.1w

点赞数

CC 4.0 BY-SA版权

分类专栏：疑难杂症文章标签： kube-dns k8s

本文链接：https://blog.youkuaiyun.com/shida_csdn/article/details/79270985

疑难杂症专栏收录该内容

95 篇文章

订阅专栏

今天早上，发现原来运行的好好的Kubernetes集群不能正常工作了，dashboard界面打不开，主节点上 docker ps 不显示任何运行中容器，重启 kubelet 后，短暂恢复，之后再次陷入不可用状态，经过反复重启观察，发现是etcd不断重启，最后失败，导致其它组件相继失败。

etcd 日志如下：

2018-02-06 02:25:24.564269 I | etcdmain: etcd Version: 3.0.17
2018-02-06 02:25:24.564531 I | etcdmain: Git SHA: cc198e2
2018-02-06 02:25:24.564544 I | etcdmain: Go Version: go1.6.4
2018-02-06 02:25:24.564554 I | etcdmain: Go OS/Arch: linux/amd64
2018-02-06 02:25:24.564563 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2018-02-06 02:25:24.564636 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-02-06 02:25:24.565021 I | etcdmain: listening for peers on http://localhost:2380
2018-02-06 02:25:24.565114 I | etcdmain: listening for client requests on 127.0.0.1:2379
2018-02-06 02:25:24.568360 I | etcdserver: recovered store from snapshot at index 5900621
2018-02-06 02:25:24.568399 I | etcdserver: name = default
2018-02-06 02:25:24.568406 I | etcdserver: data dir = /var/lib/etcd
2018-02-06 02:25:24.568413 I | etcdserver: member dir = /var/lib/etcd/member
2018-02-06 02:25:24.568418 I | etcdserver: heartbeat = 100ms
2018-02-06 02:25:24.568423 I | etcdserver: election = 1000ms
2018-02-06 02:25:24.568428 I | etcdserver: snapshot count = 10000
2018-02-06 02:25:24.568464 I | etcdserver: advertise client URLs = http://127.0.0.1:2379
2018-02-06 02:25:24.760641 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 5904480
2018-02-06 02:25:24.760850 I | raft: 8e9e05c52164694d became follower at term 12
2018-02-06 02:25:24.760904 I | raft: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 12, commit: 5904480, applied: 5900621, 

lastindex: 5904480, lastterm: 12]
2018-02-06 02:25:24.761062 I | api: enabled capabilities for version 3.0
2018-02-06 02:25:24.761111 I | membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store
2018-02-06 02:25:24.761125 I | membership: set the cluster version to 3.0 from store
2018-02-06 02:25:24.829180 I | mvcc: restore compact to 2314830
2018-02-06 02:25:24.975979 I | etcdserver: starting server... [version: 3.0.17, cluster version: 3.0]
2018-02-06 02:25:25.661519 I | raft: 8e9e05c52164694d is starting a new election at term 12
2018-02-06 02:25:25.661581 I | raft: 8e9e05c52164694d became candidate at term 13
2018-02-06 02:25:25.661596 I | raft: 8e9e05c52164694d received vote from 8e9e05c52164694d at term 13
2018-02-06 02:25:25.661620 I | raft: 8e9e05c52164694d became leader at term 13
2018-02-06 02:25:25.661645 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 13
2018-02-06 02:25:25.663336 I | etcdserver: published {Name:default ClientURLs:[http://127.0.0.1:2379]} to cluster cdf818194e3a8c32
2018-02-06 02:25:25.663379 I | etcdmain: ready to serve client requests
2018-02-06 02:25:25.663797 N | etcdmain: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2018-02-06 02:25:37.403730 W | etcdserver: apply entries took too long [25.315155ms for 1 entries]
2018-02-06 02:25:37.403752 W | etcdserver: avoid queries with large range/delete range!
2018-02-06 02:25:49.930163 N | osutil: received terminated signal, shutting down...

查找好多资料，大多说是 etcd 版本问题，可是原来是好好的呀！！！明显不是这个问题。

费了半天劲，最终排查出的问题是：主节点机器磁盘空间不够了，/ 目录使用率达到了92%

删除部分文件后，磁盘使用率降到70%，重新启动 kubelet，发现etcd不再重启退出了，但是 kube-dns 一直启动不起来

提示错误：

Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  1h		35m		36	kubelet, inm-bj-vip-ms04			Warning		FailedMount		Unable to mount volumes for pod "kube-dns-2425271678-bgzzp_kube-system(d7381d67-fb52-11e7-a2f6-0050568b6632)": timeout expired waiting for volumes to attach/mount for pod "kube-system"/"kube-dns-2425271678-bgzzp". list of unattached/unmounted volumes=[kube-dns-config]
  1h		35m		36	kubelet, inm-bj-vip-ms04			Warning		FailedSync		Error syncing pod
  1h		34m		48	kubelet, inm-bj-vip-ms04			Warning		FailedMount		MountVolume.SetUp failed for volume "kube-dns-config" : stat /var/lib/kubelet/pods/d7381d67-fb52-11e7-a2f6-0050568b6632/volumes/kubernetes.io~configmap/kube-dns-config: no such file or directory
  30m		30m		1	kubelet, inm-bj-vip-ms04			Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "kube-dns-token-3hdnc" 
  28m		10m		9	kubelet, inm-bj-vip-ms04			Warning		FailedMount		Unable to mount volumes for pod "kube-dns-2425271678-bgzzp_kube-system(d7381d67-fb52-11e7-a2f6-0050568b6632)": timeout expired waiting for volumes to attach/mount for pod "kube-system"/"kube-dns-2425271678-bgzzp". list of unattached/unmounted volumes=[kube-dns-config]
  28m		10m		9	kubelet, inm-bj-vip-ms04			Warning		FailedSync		Error syncing pod
  30m		10m		18	kubelet, inm-bj-vip-ms04			Warning		FailedMount		MountVolume.SetUp failed for volume "kube-dns-config" : stat /var/lib/kubelet/pods/d7381d67-fb52-11e7-a2f6-0050568b6632/volumes/kubernetes.io~configmap/kube-dns-config: no such file or directory
  10m		10m		1	kubelet, inm-bj-vip-ms04			Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "kube-dns-token-3hdnc" 
  10m		9m		6	kubelet, inm-bj-vip-ms04			Warning		FailedMount		MountVolume.SetUp failed for volume "kube-dns-config" : stat /var/lib/kubelet/pods/d7381d67-fb52-11e7-a2f6-0050568b6632/volumes/kubernetes.io~configmap/kube-dns-config: no such file or directory
  9m		9m		1	kubelet, inm-bj-vip-ms04			Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "kube-dns-token-3hdnc" 
  7m		5m		2	kubelet, inm-bj-vip-ms04			Warning		FailedMount		Unable to mount volumes for pod "kube-dns-2425271678-bgzzp_kube-system(d7381d67-fb52-11e7-a2f6-0050568b6632)": timeout expired waiting for volumes to attach/mount for pod "kube-system"/"kube-dns-2425271678-bgzzp". list of unattached/unmounted volumes=[kube-dns-config]
  7m		5m		2	kubelet, inm-bj-vip-ms04			Warning		FailedSync		Error syncing pod
  9m		3m		11	kubelet, inm-bj-vip-ms04			Warning		FailedMount		MountVolume.SetUp failed for volume "kube-dns-config" : stat /var/lib/kubelet/pods/d7381d67-fb52-11e7-a2f6-0050568b6632/volumes/kubernetes.io~configmap/kube-dns-config: no such file or directory
  1m		1m		1	kubelet, inm-bj-vip-ms04			Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "kube-dns-token-3hdnc" 
  1m		50s		8	kubelet, inm-bj-vip-ms04			Warning		FailedMount		MountVolume.SetUp failed for volume "kube-dns-config" : stat /var/lib/kubelet/pods/d7381d67-fb52-11e7-a2f6-0050568b6632/volumes/kubernetes.io~configmap/kube-dns-config: no such file or directory

感觉是挂载卷问题，尝试重新安装网络都没有效果 ^~^|||

最后，发大招，关闭 kubelet 和 docker 后，删除所有容器，进入 /var/lib/kubelet/pods ，直接把 kube-dns 的pod 删掉（pod的名字在错误日志里有，这里就是d7381d67-fb52-11e7-a2f6-0050568b6632），再重启docker和 kubelet，dns服务恢复！！！

末尾附上 dns 恢复依次执行的命令：

# systemctl stop kubelet
# docker kill $(docker ps -a -q)
# docker rm $(docker ps -a -q)
# systemctl stop docker
# cd /var/lib/kubelet/pods
# mv d7381d67-fb52-11e7-a2f6-0050568b6632 /tmp
# systemctl start docker
# systemctl start kubelet