k8s1.24 FAQ

原创已于 2022-05-17 20:06:17 修改 · 1.7k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#docker #容器 #linux

于 2022-05-17 19:45:43 首次发布

DevOps专栏专栏收录该内容

2 篇文章

订阅专栏

在k8s1.24版本中遇到了runc相关的错误，表现为containerd创建任务失败，原因是无法找到oci runtime错误。解决方法包括：从官网下载最新runc替换原有文件并赋予执行权限；检查并修改containerd配置文件；处理cni插件初始化网络错误；解决kubelet认证问题；处理kube-proxy的授权问题；以及解决k8s拉取私有镜像仓库失败的问题，特别是遇到'permission denied'错误时。

k8s1.24版本关于runc的问题

问题报错：
May 12 13:48:30 k8snode2 containerd:
time=“2022-05-12T13:48:30.848749359+08:00” level=error msg=“RunPodSandbox for &PodSandboxMetadata{Name:kube-flannel-ds-8lvs6,Uid:1c0ac5b8-74eb-49b2-842c-bcc00355ddff,Namespace:kube-system,Attempt:0,} failed,
error” error=“failed to create containerd task: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open
/run/containerd/io.containerd.runtime.v2.task/k8s.io/029eabb43894ea21e4ac75a43a4f144031ed8946fe748006e8748d1d65843092/log.json: no such file or directory): runc did not terminate successfully: exit status 127: unknown”

解决方法

官网下载 runc 执行文件替换linux原有的 /user/local/sbin/runc, 赋予执行权限 chmod +x /user/local/sbin/runc

- containerd配置文件需要修改的地方

/etc/containerd/config.toml 

[plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
    .....
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
	......
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
                endpoint = ["https://registry-1.docker.io"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."myharbor.io"]
                endpoint = ["http://myharbor.io"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."myharbor.io".tls]
                insecure_skip_verify = true
        [plugins."io.containerd.grpc.v1.cri".registry.configs."myharbor.io".auth]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]

- cni插件未初始化网络错误，解决办法

mkdir -p /etc/cni/net.d/
cat >/etc/cni/net.d/10-flannel.conflist <<EOF
{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat >/run/flannel/subnet.env <<EOF
FLANNEL_NETWORK=172.16.0.0/12  # 这里的网段修改为kube-controller-manager组件启动参数--cluster-cidr指定的网段
FLANNEL_SUBNET=172.16.0.0/24 
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF

- 出现kubelet匿名还是集群角色无法认证的问题

May 12 15:58:22 k8snode1 kubelet: W0512 15:58:22.313065    3950 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope
May 12 15:58:22 k8snode1 kubelet: E0512 15:58:22.313093    3950 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope
May 12 15:58:22 k8snode1 kubelet: E0512 15:58:22.361994    3950 kubelet.go:2419] "Error getting node" err="node \"k8snode1\" not found"

解决办法

node节点关闭开启的kubelet
# systemctl stop kubelet

master节点
# kubectl delete clusterrolebinding kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" deleted

# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

#node节点开启kubelet
systemctl start kubelet

master节点
# kubectl get csr

- 问题:kube-proxy匿名授权

May 12 19:12:32 k8snode1 kube-proxy: E0512 19:12:32.438837   34504 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:anonymous" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

解决方法

kubectl config set-cluster kubernetes \
  --certificate-authority=/data/k8s/ssl/ca.pem \
  --embed-certs=true \
  --server="https://192.168.8.203:6443" \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=/data/k8s/ssl/kube-proxy.pem \
  --client-key=/data/k8s/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

systemctl restart kube-proxy

- 问题：k8s拉取私有镜像仓库失败处理方法

～/.docker/config.json
让k8s集群具有拉取私有仓库的权限
kubectl create secret generic harbor-registry --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson

问题：failed to set bridge addr: could not add IP address to \“cni0\”: permission denied"

报错信息：
May 14 14:35:27 k8snode4 kubelet: E0514 14:35:27.058422    5161 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"eip-nfs-nfsshare-556ccffc6d-jk8rk_kube-system(7c2fcb44-7df3-4804-a5bb-f5f7dc60b2b6)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"eip-nfs-nfsshare-556ccffc6d-jk8rk_kube-system(7c2fcb44-7df3-4804-a5bb-f5f7dc60b2b6)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"24743a13eb6ff7bf972c0f7a25bb6d5860ccafc1eed49b07d6ed6131edaa793d\\\": plugin type=\\\"bridge\\\" failed (add): failed to set bridge addr: could not add IP address to \\\"cni0\\\": permission denied\"" pod="kube-system/eip-nfs-nfsshare-556ccffc6d-jk8rk" podUID=7c2fcb44-7df3-4804-a5bb-f5f7dc60b2b6

处理方法

导致这个问题是由于cni0网段地址不一致导致网络加入没有权限
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat >/run/flannel/subnet.env <<EOF
FLANNEL_NETWORK=172.16.0.0/12
FLANNEL_SUBNET=172.16.0.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF

ifconfig cni0 down
ip link delete cni0
systemctl start kubelet