Kubernetes Service网络层故障排错总结

Kubernetes Service网络层出现故障总结

1、可能现象

curl -k https://10.96.0.1:443  # 连接失败

2、排查ku be-proxy

查看发现kube-proxy中有错误日志如下 failed to list *v1.EndpointSlice: Get "https://192.168.1.101:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=35533": dial tcp 192.168.1.101:6443: connect: connection refused E0626 05:42:21.880071 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.1.101:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=35533": dial tcp 192.168.1.101:6443: connect: connection refused W0626 05:42:30.906439 1 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.Service: Get "https://192.168.1.101:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=35560": dial tcp 192.168.1.101:6443: connect: connection refused E0626 05:42:30.906490 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.1.101:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=35560": dial tcp 192.168.1.101:6443: connect: connection refused

此刻pod处于running状态，但内部运行报错

3、修复策略

# 清理冲突规则
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -F
​
# 重置 Docker 网络 
sudo systemctl restart docker
​
# 添加 API Server 访问规则
sudo iptables -A INPUT -p tcp --dport 6443 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 6443 -j ACCEPT
​
# 删除现有代理让 K8s 重建
kubectl delete pod -n kube-system -l k8s-app=kube-proxy --force

4、重建kube-proxy成功后可能出现calico错误

root@k8s-master:/home/admin123/kubeedge-cloud-install-ubuntu20.04-amd64-focal# kubectl logs calico-node-cptt6 -n kube-system -c install-cni 2025-06-26 06:14:18.950 INFO cni-installer/<nil> <nil>: Running as a Kubernetes pod 2025-06-26 06:14:19.470 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/bandwidth" 2025-06-26 06:14:19.470 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/bandwidth 2025-06-26 06:14:19.511 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico" 2025-06-26 06:14:19.511 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico 2025-06-26 06:14:19.551 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico-ipam" 2025-06-26 06:14:19.551 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico-ipam 2025-06-26 06:14:19.552 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/flannel" 2025-06-26 06:14:19.552 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/flannel 2025-06-26 06:14:19.555 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/host-local" 2025-06-26 06:14:19.555 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/host-local 2025-06-26 06:14:19.595 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/install" 2025-06-26 06:14:19.595 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/install 2025-06-26 06:14:19.599 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/loopback" 2025-06-26 06:14:19.599 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/loopback 2025-06-26 06:14:19.602 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/portmap" 2025-06-26 06:14:19.602 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/portmap 2025-06-26 06:14:19.605 INFO cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/tuning" 2025-06-26 06:14:19.606 INFO cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/tuning 2025-06-26 06:14:19.606 INFO cni-installer/<nil> <nil>: Wrote Calico CNI binaries to /host/opt/cni/bin 2025-06-26 06:14:19.624 INFO cni-installer/<nil> <nil>: CNI plugin version: v3.26.0 2025-06-26 06:14:19.624 INFO cni-installer/<nil> <nil>: /host/secondary-bin-dir is not writeable, skipping W0626 06:14:19.625033 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. 2025-06-26 06:14:19.625 ERROR cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp 10.96.0.1:443: connect: network is unreachable 2025-06-26 06:14:19.625 FATAL cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp 10.96.0.1:443: connect: network is unreachable

5、继续修复

# 检查主机网络路由
# 在节点上执行
ip route show table all | grep kube
ip route get 10.96.0.1
​
#应有类似输出：10.96.0.1 via 192.168.1.101 dev eth0 
​
#若无手动添加路由
# 在所有节点执行（替换 eth0 为您的网卡名）
sudo ip route add 10.96.0.1/32 via 192.168.1.101 dev eth0
​
#再次查看calic应创建成功并无报错日志

6、该网络还可能会影响kubeedge的安装，会导致如下报错，其主要原因就是上面的路由缺失

TLSTunnelPrivateKeyFile does not exist in /etc/kubeedge/certs/server.key, will load from secret W0626 10:45:10.332634 1 validation.go:187] TLSTunnelCertFile does not exist in /etc/kubeedge/certs/server.crt, will load from secret W0626 10:45:10.332643 1 validation.go:190] TLSTunnelCAFile does not exist in /etc/kubeedge/ca/rootCA.crt, will load from secret I0626 10:45:10.332680 1 server.go:98] Version: v0.0.0-master+$Format:%h$ W0626 10:45:10.332696 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0626 10:45:10.332877 1 monitor.go:98] starting monitor server on addr: 127.0.0.1:9091 panic: failed to create system namespace: Post "https://10.96.0.1:443/api/v1/namespaces": dial tcp 10.96.0.1:443: connect: network is unreachable goroutine 1 [running]: github.com/kubeedge/kubeedge/cloud/cmd/cloudcore/app.NewCloudCoreCommand.func1(0xc0004a8700?, {0x2572083?, 0x4?, 0x257204f?}) /go/src/github.com/kubeedge/kubeedge/cloud/cmd/cloudcore/app/server.go:109 +0x825 github.com/kubeedge/kubeedge/vendor/github.com/spf13/cobra.(Command).execute(0xc000dd5800, {0xc0000500e0, 0x0, 0x0}) /go/src/github.com/kubeedge/kubeedge/vendor/github.com/spf13/cobra/command.go:944 +0x863 github.com/kubeedge/kubeedge/vendor/github.com/spf13/cobra.(Command).ExecuteC(0xc000dd5800) /go/src/github.com/kubeedge/kubeedge/vendor/github.com/spf13/cobra/command.go:1068 +0x3a5 github.com/kubeedge/kubeedge/vendor/github.com/spf13/cobra.(*Command).Execute(0xc0000c6058?) /go/src/github.com/kubeedge/kubeedge/vendor/github.com/spf13/cobra/command.go:992 +0x13 main.main() /go/src/github.com/kubeedge/kubeedge/cloud/cmd/cloudcore/cloudcore.go:32 +0x45