一次kubenetes的rook-ceph创建pv失败的故障排查

kubenetes基于rook-ceph创建pv失败的一次故障排除

1、本次问题出现，新创建statefulset的pod无法正常创建pv

Events:
  Type     Reason            Age        From               Message

----     ------            ----       ----               -------

  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "mysql-0": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "mysql-0": pod has unbound immediate PersistentVolumeClaims

出现pvc没法正常挂载pv，PV未正常挂载的情况；

排查问题过程：

1、首先检查rook-ceph状态

[root@master1 images]# kubectl exec -it -n rook-ceph rook-ceph-tools-6b4889fdfd-86dp5 /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
[root@rook-ceph-tools-6b4889fdfd-86dp5 /]# ceph -s
  cluster:
    id:     bb5107d5-d3f7-45df-9146-1148efa378b5
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum b,c,d (age 67m)
    mgr: a(active, since 7m)
    mds: myfs:1 {0=myfs-b=up:active} 1 up:standby-replay
    osd: 10 osds: 10 up (since 7h), 10 in (since 8d)

  task status:
    scrub status:
        mds.myfs-a: idle
        mds.myfs-b: idle

  data:
    pools:   4 pools, 97 pgs
    objects: 1.20k objects, 3.2 GiB
    usage:   19 GiB used, 1.9 TiB / 2.0 TiB avail
    pgs:     97 active+clean

  io:
    client:   852 B/s rd, 1 op/s rd, 0 op/s wr

检查发现ceph状态正常；

2、检查kube-system集群

[root@master1 images]# kubectl get po -n kube-system 
NAME                                         READY   STATUS    RESTARTS   AGE
calico-kube-controllers-578894d4cd-8wlg4     1/1     Running   0          8d
calico-node-5rnjk                            1/1     Running   0          8d
calico-node-7rvj2                            1/1     Running   0          8d
calico-node-p7hpq                            1/1     Running   0          8d
calico-node-vgrlg                            1/1     Running   0          8d
calico-node-zd2mn                            1/1     Running   0          8d
coredns-66bff467f8-fj7td                     1/1     Running   0          5d3h
coredns-66bff467f8-rmnzk                     1/1     Running   0          8d
dashboard-metrics-scraper-6b4884c9d5-8gtnl   1/1     Running   0          8d
etcd-master1                                 1/1     Running   0          20m
etcd-master2                                 1/1     Running   0          20m
etcd-master3                                 1/1     Running   0          20m
kube-apiserver-master1                       1/1     Running   0          8d
kube-apiserver-master2                       1/1     Running   0          8d
kube-apiserver-master3                       1/1     Running   0          8d
kube-controller-manager-master1              1/1     Running   63         8d
kube-controller-manager-master2              1/1     Running   64         8d
kube-controller-manager-master3              1/1     Running   64         8d
kube-proxy-6n7lz                             1/1     Running   0          8d
kube-proxy-7nstv                             1/1     Running   0          8d
kube-proxy-kxzhp                             1/1     Running   0          8d
kube-proxy-tw9j4                             1/1     Running   0          8d
kube-proxy-w4s47                             1/1     Running   0          8d
kube-scheduler-master1                       1/1     Running   63         32m
kube-schedu