running PreBind plugin “VolumeBinding“: binding volumes: timed out waiting for the condition

需求

在本地实现动态pv，创建一个stateful set，使用volumeClaimTemplates来申请存储
1、安装openebs，只安装本地的hostpath,默认的hostpath是/var/openebs/local

sudo mkdir -p /var/openebs/local
sudo chmod -R 777 /var/openebs/local
helm repo add openebs https://openebs.github.io/charts
helm install openebs openebs/openebs -n openebs --create-namespace \
	--set legacy.enabled=false \
	--set ndm.enabled=false \
	--set ndmOperator.enabled=false \
	--set localprovisioner.enableDeviceClass=false \
  --set localprovisioner.basePath=/var/openebs/local
# 查看pods
-> % kubectl get pods -n openebs
NAME                                           READY   STATUS    RESTARTS   AGE
openebs-localpv-provisioner-6f686f7697-dvjvb   1/1     Running   0          2m16s

2、创建StorageClass,默认的是openebs-hostpath

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sc-file-hdd
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: hostpath
      - name: BasePath
        value: /var/openebs/local
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

-> % kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-hostpath     openebs.io/local           Delete          WaitForFirstConsumer   false                  12m
sc-file-hdd          openebs.io/local           Delete          WaitForFirstConsumer   false                  68s
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate              false                  46h

3、创建PersistentVolumeClaim

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-hostpath-pvc
spec:
  storageClassName: sc-file-hdd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
#查看pvc处于Pending状态
-> % kubectl get pvc
NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
local-hostpath-pvc   Pending                                      sc-file-hdd    4s
accessModes: ReadWriteOnce

4、创建Pod消费pvc

apiVersion: v1
kind: Pod
metadata:
  name: hello-local-hostpath-pod2
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: local-hostpath-pvc
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage

# 查看pod
-> % kubectl get pod hello-local-hostpath-pod
NAME                       READY   STATUS    RESTARTS   AGE
hello-local-hostpath-pod   1/1     Running   0          26s
# 查看文件内容
-> % kubectl exec hello-local-hostpath-pod -- cat /mnt/store/greet.txt
Fri Oct 29 03:41:41 UTC 2021 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.
# 查看pvc
-> % kubectl get pvc local-hostpath-pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
local-hostpath-pvc   Bound    pvc-a24136a6-901f-425b-a671-533ce305eadd   1Gi        RWO            sc-file-hdd    9m6s

报错

如果将上面的pvc创建放到stateset中去做，就会有出现下面的报错了

running PreBind plugin “VolumeBinding”: binding volumes: timed out waiting for the condition

#查看kube-system调度的详情
kubectl get pods -n kube-system | tail -n +2 | awk '{print $1}' | while read pod;do  echo "-----------------$pod-----------------"; ;kubectl logs $pod --tail=20 -n  kube-system;done

-----------------kube-controller-manager-minikube-----------------
I1029 06:32:27.580961       1 event.go:291] "Event occurred" object="proxysql/proxysql-data-proxysql4406-0" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="ExternalProvisioning" message="waiting for a volume to be created, either by external provisioner \"openebs.io/local\" or manually created by system administrator"
E1029 06:32:32.792029       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
-----------------kube-scheduler-minikube-----------------
E1029 04:56:08.816642       1 framework.go:744] "Failed running PreBind plugin" err="binding volumes: timed out waiting for the condition" plugin="VolumeBinding" pod="proxysql/proxysql4406-0"
E1029 04:56:08.816808       1 factory.go:337] "Error scheduling pod; retrying" err="running PreBind plugin \"VolumeBinding\": binding volumes: timed out waiting for the condition" pod="proxysql/proxysql4406-0"

-----------------openebs-localpv-provisioner-----------------
E1029 07:02:37.040005       1 controller.go:956] error syncing claim "a93b3b2f-c369-404b-a97d-4fa2f7f58188": failed to provision volume with StorageClass "sc-file-hdd": Only support ReadWriteOnce access mode
I1029 07:02:37.039926       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"proxysql", Name:"proxysql-data-proxysql4406-0", UID:"a93b3b2f-c369-404b-a97d-4fa2f7f58188", APIVersion:"v1", ResourceVersion:"57647", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "proxysql/proxysql-data-proxysql4406-0"

pvc的annotations
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: openebs.io/local
    volume.kubernetes.io/selected-node: minikube

原因

看openebs-localpv-provisioner 和kube-scheduler-minikube 和kube-controller-manager-minikube的报错信息，就发现了问题

  volumeClaimTemplates:
    - metadata:
        name: proxysql-data
      spec:
        accessModes:
          - ReadWriteMany  ## 就是这里了 不能用ReadWriteMany只能用ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: sc-file-hdd
        volumeMode: Filesystem

解决

ReadWriteMany：volumn能被多个容器访问
ReadWriteOnce：volumn只能被一个容器访问
很好理解，statefulset本来就是有状态的数据，必须只能被每一个被一个容器访问

  volumeClaimTemplates:
    - metadata:
        name: proxysql-data
      spec:
        accessModes:
          - ReadWriteOnce  # 这里改成ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: sc-file-hdd
        volumeMode: Filesystem