Using Ceph RBD for dynamic provisioning

Creating a pool for dynamic volumes

1.Install the latest ceph-common package:

$ yum install -y ceph-common

Note:所有k8s master和被调度的node节点上均需要安装此包

2.From an administrator or MON node, create a new pool for dynamic volumes, for example:

$ ceph osd pool create kube 1024
$ ceph auth get-or-create client.kube mon 'allow r, allow command "osd blacklist"' osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring

Using an existing Ceph cluster for dynamic persistent storage

1.Generate the client.admin base64-encoded key:

$ ceph auth get client.admin

Ceph secret definition example

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: kube-system
data:
  key: QVFBOFF2SlZheUJQRVJBQWgvS2cwT1laQUhPQno3akZwekxxdGc9PQ== (1)
type: kubernetes.io/rbd (2)

(1).This base64 key is generated on one of the Ceph MON nodes using the ceph auth get-key client.admin | base64 command, then copying the output and pasting it as the secret key’s value.

(2).This value is required for Ceph RBD to work with dynamic provisioning.

2.Create the Ceph secret for the client.admin:

$ kubectl create -f ceph-secret.yaml
secret "ceph-secret" created

3.Create the storage class:

$ kubectl create -f ceph-storageclass.yaml
storageclass "dynamic" created

Ceph storage class example

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-dynamic
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd
volumeBindingMode: WaitForFirstConsumer
parameters:
  monitors: 192.168.1.1,192.168.1.2,192.168.1.3
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  fsType: xfs
  imageFeatures: layering
  imageFormat: "2"
  pool: k8s
  userId: k8s
  userSecretName: ceph-user-secret
  userSecretNamespace: kube-system

4. Create the PVC object definition:

PVC object definition example

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ceph-claim-dynamic
  finalizers:
  - kubernetes.io/pvc-protection
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: ceph-dynamic
  volumeMode: Filesystem

5.Create the PVC:

$ kubectl create -f ceph-pvc.yaml
persistentvolumeclaim "ceph-claim-dynamic" created

6.Verify that the PVC was created and bound to the expected PV:

$ kubectl get pvc
NAME        STATUS  VOLUME                                   CAPACITY ACCESSMODES  AGE
ceph-claim  Bound   pvc-f548d663-3cac-11e7-9937-0024e8650c7a 2Gi      RWO          1m

7.Create the pod object definition:

Pod object definition example

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod1 
spec:
  containers:
  - name: ceph-busybox
    image: busybox 
    command: ["sleep", "60000"]
    volumeMounts:
    - name: ceph-vol1 
      mountPath: /usr/share/busybox 
      readOnly: false
  volumes:
  - name: ceph-vol1
    persistentVolumeClaim:
      claimName: ceph-claim-dynamic 

8.Create the pod:

$ kubectl create -f ceph-pod1.yaml

9.Verify that the pod was created:

$kubectl get pod
NAME        READY     STATUS   RESTARTS   AGE
ceph-pod1   1/1       Running  0          2m