kubelet启动报错:kubelet.service: Failed with result 'exit

Kubernetes Kubelet启动失败
本文记录了在Kubernetes集群中,kubelet服务启动失败的问题排查与解决过程。问题出现于CgroupDriver配置不一致,kubelet与docker的CgroupDriver默认分别为cgroupfs与systemd。通过调整kubelet的CgroupDriver为cgroupfs,成功解决了启动失败的问题。
qyh@qyh-master:/home/kubernetes$ systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: 
   Active: failed (Result: exit-code) since Sun 2019-02-24 17:34:38 CST; 8s ago
  Process: 13392 ExecStart=/opt/kubernetes/bin/kubelet ${KUBE_LOGTOSTDERR} ${KUB
 Main PID: 13392 (code=exited, status=255)

2月 24 17:34:38 qyh-master kubelet[13392]:       --vmodule moduleSpec           
2月 24 17:34:38 qyh-master kubelet[13392]:       --volume-plugin-dir string     
2月 24 17:34:38 qyh-master kubelet[13392]:       --volume-stats-agg-period durat
2月 24 17:34:38 qyh-master kubelet[13392]: F0224 17:34:38.463222   13392 server.
2月 24 17:34:38 qyh-master systemd[1]: kubelet.service: Service hold-off time ov
2月 24 17:34:38 qyh-master systemd[1]: kubelet.service: Scheduled restart job, r
2月 24 17:34:38 qyh-master systemd[1]: Stopped Kubernetes Kubelet.
2月 24 17:34:38 qyh-master systemd[1]: kubelet.service: Start request repeated t
2月 24 17:34:38 qyh-master systemd[1]: kubelet.service: Failed with result 'exit
2月 24 17:34:38 qyh-master systemd[1]: Failed to start Kubernetes Kubelet.

------------------------------------------------------------------------------------------------------------------

查看docker和kublelt的Cgroup Driver是否一致。这个是不一致造成的。改成一致。可以看出报错的原因是kubelet与docker的cgroup driver 不一致导致

qyh@qyh-master:/home/kubernetes$ sudo docker info|grep Cgroup
Cgroup Driver: cgroupfs

----------------------------------------------------------------------------------------------------------------------

qyh@qyh-master:/home/kubernetes$ journalctl -f -u kubelet
-- Logs begin at Sun 2019-02-24 15:21:08 CST. --
2月 24 16:53:25 qyh-master kubelet[7850]:       --vmodule moduleSpec                                                                                        comma-separated list of pattern=N settings for file-filtered logging
2月 24 16:53:25 qyh-master kubelet[7850]:       --volume-plugin-dir string                                                                                  The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
2月 24 16:53:25 qyh-master kubelet[7850]:       --volume-stats-agg-period duration                                                                          Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
2月 24 16:53:25 qyh-master kubelet[7850]: F0224 16:53:25.446480    7850 server.go:154] unknown command:
2月 24 16:53:25 qyh-master systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
2月 24 16:53:25 qyh-master systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 5.
2月 24 16:53:25 qyh-master systemd[1]: Stopped Kubernetes Kubelet.
2月 24 16:53:25 qyh-master systemd[1]: kubelet.service: Start request repeated too quickly.
2月 24 16:53:25 qyh-master systemd[1]: kubelet.service: Failed with result 'exit-code'.
2月 24 16:53:25 qyh-master systemd[1]: Failed to start Kubernetes Kubelet.

--------------------------------------------------------

docker相比1.10增加了KernelMemory变量和CgroupDriver变量,KernelMemory变量表示是否设置linux内核内存限制,CgroupDriver变量表示使用哪个Cgroup驱动,有两种驱动,分别是cgroupfs和systemd,默认使用cgroupfs。 那么解决该问题的方法是:要么修改docker 的cgroup driver , 要么改kubelet的 cgroup driver.。 建议改kubelet的值, 因为docker本身有些镜像在使用的话, 直接改会破坏镜像

--------------------------------------------------------------------

直接更改kubelet.sh

# 其他参数
KUBELET_ARGS="--cgroup-driver=cgroupfs"
EOF

KUBE_PROXY_OPTS=" \${KUBE_LOGTOSTDERR}     \\
 \${KUBE_LOG_LEVEL}       \\
 \${NODE_ADDRESS}         \\
 \${NODE_PORT}            \\
 \${NODE_HOSTNAME}        \\
 \${KUBELET_KUBECONFIG}   \\
 \${KUBE_ALLOW_PRIV}      \\
 \${KUBELET_DNS_IP}       \\
 \${KUBELET_DNS_DOMAIN}   \\
 \${KUBELET_ARGS}"
8月 14 20:38:51 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:38:51 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:39:01 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 63. 8月 14 20:39:01 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:39:01 localhost.localdomain (kubelet)[35711]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:39:01 localhost.localdomain kubelet[35711]: E0814 20:39:01.509470 35711 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:39:01 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:39:01 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:39:11 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 64. 8月 14 20:39:11 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:39:11 localhost.localdomain (kubelet)[35780]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:39:11 localhost.localdomain kubelet[35780]: E0814 20:39:11.765548 35780 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:39:11 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:39:11 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:39:21 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 65. 8月 14 20:39:21 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:39:21 localhost.localdomain (kubelet)[35852]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:39:22 localhost.localdomain kubelet[35852]: E0814 20:39:22.005231 35852 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:39:22 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:39:22 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:39:32 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 66. 8月 14 20:39:32 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:39:32 localhost.localdomain (kubelet)[35923]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:39:32 localhost.localdomain kubelet[35923]: E0814 20:39:32.263622 35923 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:39:32 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:39:32 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:39:42 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 67. 8月 14 20:39:42 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:39:42 localhost.localdomain (kubelet)[35993]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:39:42 localhost.localdomain kubelet[35993]: E0814 20:39:42.510408 35993 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:39:42 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:39:42 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:39:52 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 68. 8月 14 20:39:52 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:39:52 localhost.localdomain (kubelet)[36068]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:39:52 localhost.localdomain kubelet[36068]: E0814 20:39:52.764578 36068 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:39:52 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:39:52 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:40:02 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 69. 8月 14 20:40:02 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:40:02 localhost.localdomain (kubelet)[36137]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:40:03 localhost.localdomain kubelet[36137]: E0814 20:40:03.018209 36137 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:40:03 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:40:03 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'. 8月 14 20:40:13 localhost.localdomain systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 70. 8月 14 20:40:13 localhost.localdomain systemd[1]: Started kubelet: The Kubernetes Node Agent. 8月 14 20:40:13 localhost.localdomain (kubelet)[36207]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_KUBEADM_ARGS 8月 14 20:40:13 localhost.localdomain kubelet[36207]: E0814 20:40:13.263321 36207 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" 8月 14 20:40:13 localhost.localdomain systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE 8月 14 20:40:13 localhost.localdomain systemd[1]: kubelet.service: Failed with result 'exit-code'.
08-15
[root@ex2 ~]# ls -l /etc/kubernetes/pki/ca.crt -rw-r--r-- 1 root root 1107 May 13 10:24 /etc/kubernetes/pki/ca.crt [root@ex2 ~]# clientCAFile: /etc/kubernetes/pki/ca.crt -bash: clientCAFile:: command not found [root@ex2 ~]# dnf install clientCAFile Last metadata expiration check: 0:57:32 ago on Tue 13 May 2025 09:38:54 AM CST. No match for argument: clientCAFile Error: Unable to find a match: clientCAFile [root@ex2 ~]# kubeadm init phase certs all --config /etc/kubernetes/kubeadm-config.yaml [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Using existing ca certificate authority [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node] and IPs [10.96.0.1 192.168.56.103] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost node] and IPs [192.168.56.103 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost node] and IPs [192.168.56.103 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [root@ex2 ~]# kubeadm init phase certs all --config /etc/kubernetes/kubeadm-config.yaml^C [root@ex2 ~]# vim /etc/kubernetes/kubeadm-config.yaml [root@ex2 ~]# kubeadm init phase certs ca # 仅生成CA证书 ^C [root@ex2 ~]# kubeadm init phase certs ca I0513 10:37:02.703232 7782 version.go:256] remote version is much newer: v1.33.0; falling back to: stable-1.29 [certs] Using existing ca certificate authority [root@ex2 ~]# systemctl restart kubelet [root@ex2 ~]# journalctl -f -u kubelet -- Logs begin at Tue 2025-05-06 11:12:05 CST. -- May 13 10:37:12 ex2.orange.com systemd[1]: Stopping kubelet: The Kubernetes Node Agent... May 13 10:37:12 ex2.orange.com systemd[1]: kubelet.service: Succeeded. May 13 10:37:12 ex2.orange.com systemd[1]: Stopped kubelet: The Kubernetes Node Agent. May 13 10:37:12 ex2.orange.com systemd[1]: Started kubelet: The Kubernetes Node Agent. May 13 10:37:12 ex2.orange.com kubelet[7806]: I0513 10:37:12.271425 7806 server.go:492] "Kubelet version" kubeletVersion="v1.29.15" May 13 10:37:12 ex2.orange.com kubelet[7806]: I0513 10:37:12.271448 7806 server.go:494] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 13 10:37:12 ex2.orange.com kubelet[7806]: I0513 10:37:12.271517 7806 server.go:924] "Client rotation is on, will bootstrap in background" May 13 10:37:12 ex2.orange.com kubelet[7806]: E0513 10:37:12.271533 7806 run.go:74] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory" May 13 10:37:12 ex2.orange.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE May 13 10:37:12 ex2.orange.com systemd[1]: kubelet.service: Failed with result 'exit-code'. qMay 13 10:37:22 ex2.orange.com systemd[1]: kubelet.service: Service RestartSec=10s expired, scheduling restart. May 13 10:37:22 ex2.orange.com systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1. May 13 10:37:22 ex2.orange.com systemd[1]: Stopped kubelet: The Kubernetes Node Agent. May 13 10:37:22 ex2.orange.com systemd[1]: Started kubelet: The Kubernetes Node Agent. May 13 10:37:22 ex2.orange.com kubelet[7819]: I0513 10:37:22.371705 7819 server.go:492] "Kubelet version" kubeletVersion="v1.29.15" May 13 10:37:22 ex2.orange.com kubelet[7819]: I0513 10:37:22.371727 7819 server.go:494] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 13 10:37:22 ex2.orange.com kubelet[7819]: I0513 10:37:22.371809 7819 server.go:924] "Client rotation is on, will bootstrap in background" May 13 10:37:22 ex2.orange.com kubelet[7819]: E0513 10:37:22.371824 7819 run.go:74] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory" May 13 10:37:22 ex2.orange.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE May 13 10:37:22 ex2.orange.com systemd[1]: kubelet.service: Failed with result 'exit-code'.
05-14
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值