Kubernetes证书过期问题

Kubernetes证书过期问题

一、问题现象

[root@k8s-master ~]# kubectl get nodes
E0410 15:14:59.821044  196079 memcache.go:238] couldn't get current server API group list: Get "https://10.6.225.200:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2024-04-10T15:14:59+08:00 is after 2024-03-07T02:23:05Z
E0410 15:14:59.828889  196079 memcache.go:238] couldn't get current server API group list: Get "https://10.6.225.200:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2024-04-10T15:14:59+08:00 is after 2024-03-07T02:23:05Z
E0410 15:14:59.837111  196079 memcache.go:238] couldn't get current server API group list: Get "https://10.6.225.200:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2024-04-10T15:14:59+08:00 is after 2024-03-07T02:23:05Z
E0410 15:14:59.845076  196079 memcache.go:238] couldn't get current server API group list: Get "https://10.6.225.200:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2024-04-10T15:14:59+08:00 is after 2024-03-07T02:23:05Z
E0410 15:14:59.852851  196079 memcache.go:238] couldn't get current server API group list: Get "https://10.6.225.200:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2024-04-10T15:14:59+08:00 is after 2024-03-07T02:23:05Z
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2024-04-10T15:14:59+08:00 is after 2024-03-07T02:23:05Z

二、检查证书有效期，显示已过期

[root@k8s-master ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 07, 2024 02:23 UTC   <invalid>       ca                      no
apiserver                  Mar 07, 2024 02:23 UTC   <invalid>       ca                      no
apiserver-etcd-client      Mar 07, 2024 02:23 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Mar 07, 2024 02:23 UTC   <invalid>       ca                      no
controller-manager.conf    Mar 07, 2024 02:23 UTC   <invalid>       ca                      no
etcd-healthcheck-client    Mar 07, 2024 02:23 UTC   <invalid>       etcd-ca                 no
etcd-peer                  Mar 07, 2024 02:23 UTC   <invalid>       etcd-ca                 no
etcd-server                Mar 07, 2024 02:23 UTC   <invalid>       etcd-ca                 no
front-proxy-client         Mar 07, 2024 02:23 UTC   <invalid>       front-proxy-ca          no
scheduler.conf             Mar 07, 2024 02:23 UTC   <invalid>       ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 05, 2033 02:23 UTC   8y              no
etcd-ca                 Mar 05, 2033 02:23 UTC   8y              no
front-proxy-ca          Mar 05, 2033 02:23 UTC   8y              no

• 或者用 openssl 方法，查看也是过期的

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep 'Not'
            Not Before: Sep 14 14:01:12 2021 GMT
            Not After : Sep 14 14:01:12 2022 GMT

三、处理过程
3.1 备份现有k8s凭证

mkdir -p /root/k8s-old-certs/pki
cp -rp /etc/kubernetes/pki/* /root/k8s-old-certs/pki

3.2 备份现有k8s配置

cp -p /etc/kubernetes/*.conf /root/k8s-old-certs

3.3 备份家目录设定，如果有

mkdir -p /root/k8s-old-certs/.kube
...

3.4 更新所有k8s凭证

[root@k8s-master k8s-old-certs]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.   #重启静态pod

3.5 验证证书时间

[root@k8s-master k8s-old-certs]# kubeadm certs check-expiration

3.6 重启静态 Pod

mkdir -p /root/k8s-old-manifests
mv /etc/kubernetes/manifests/* /root/k8s-old-manifests/      #执行此步骤等待20s后执行下面命令
mv /root/k8s-old-manifests/* /etc/kubernetes/manifests/