目的
kubenetes 后端使用 ceph rbd 作为 pod 目录存储
当 pod 存储空间不足，对 rbd 共享存储进行扩容
扩容做到用户无感知
参考
kubernetes 直接调用 ceph 作为后端存储
其他相关的 kubenetes 存储部分信息索引
说明
| 名称 | 信息 | 说明 |
|---|---|---|
| ceph | pool | kbeceph |
| ceph | user | terry |
| kubenetes | namespace | kubeceph |
简单步骤说明
| 说明 | 操作 | 备注 |
|---|---|---|
| ceph 搭建 | 创建集群，创建 pool 用于 k8s, 创建用户 | ceph 部署相关参考 |
| kubenetes 搭建 | 略 | 参考 kubenetes 部署方法 |
| ceph 用户权限 | mon “allow r” osd “allow rwx pool=kbeceph” | 最基本权限 |
| 获取 用户 key | ceph auth get-or-create client.terry | awk ‘/key/ {print $NF}’ | base64 | 必须转换 base64 |
| namespace | kubenetes 创建专用测试 namespace | |
| secert key | kubenetes 创建专用测试 secret key | |
| classstorage | 创建 storage class | 用于动态 PV |
| pv | 创建 pv | |
| pvc | 创建 pvc | |
| pod | 创建测试 pod | |
| 扩容测试 |
创建测试 namespace
apiVersion: v1
kind: Namespace
metadata:
name: kubeceph
获取 ceph key
必须把 ceph key 转换为 base64 才可以被 kubenetes 识别使用
# ceph auth get-or-create client.terry | awk '/key/ {print $NF}' | base64
QVFDQ2w5RxxxxxxxxxxxxxxxxxxR2szcWpVcXlRRkE9PQo=
创建 secert key
apiVersion: v1
kind: Secret
metadata:
name: ceph-terry-secret
namespace: kubeceph
type: "kubernetes.io/rbd"
data:
key: QVFDQ2xxxxxxxxxxxxxxxxxxxxxxR2szcWpVcXlRRkE9PQo=
创建 storageclass
为了保证数据一致性，并且可以在 pod 删除，重建依旧保留磁盘数据
定义磁盘策略为 Retain
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "ceph-kubernets"
namespace: kubeceph
provisioner: kubernetes.io/rbd
parameters:
monitors: 10.189.xx.xx:6789,10.189.xx.xx:6789,10.189.xx.xx:6789
adminId: terry
adminSecretName: ceph-terry-secret <- 对应你的 secret key name
adminSecretNamespace: kubeceph
pool: kbeceph
userId: terry
userSecretName: ceph-terry-secret
userSecretNamespace: kubeceph
fsType: xfs
imageFormat: "2" <- 使用 rbd Layer 2 格式
imageFeatures: "layering"
reclaimPolicy: Retain
创建 pv
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: webdata-pvc-01
namespace: kubeceph
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-kubernets
resources:
requests:
storage: 2Gi
查询 pvc 信息
# kubectl -n kubeceph get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
webdata-pvc-01 Bound pvc-10b946df-02d8-4f3d-8cb6-bf34567b76b9 2Gi RWO ceph-kubernets 54s
根据 pv 信息获取 ceph rbd 对应 volume 方法
# kubectl -n kubeceph describe pv pvc-10b946df-02d8-4f3d-8cb6-bf34567b76b9 | grep RBDImage
RBDImage: kubernetes-dynamic-pvc-0888014e-2c7f-409b-af94-3f70f9843b5d
查询具体 rbd 文件信息
# rbd -p kbeceph info kubernetes-dynamic-pvc-0888014e-2c7f-409b-af94-3f70f9843b5d
rbd image 'kubernetes-dynamic-pvc-0888014e-2c7f-409b-af94-3f70f9843b5d':
size 2048 MB in 512 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.31e9a2ae8944a
format: 2
features: layering
flags:
create_timestamp: Tue Dec 22 10:33:50 2020
创建 pod
kind: Pod
apiVersion: v1
metadata:
name: centos7-test-pvc-01
namespace: kubeceph
spec:
containers:
- name: cenotos7-test01
image: "xxxx/centos7.3-base-os7:201908238a1c867"
command: ["/bin/bash", "-c", "sleep 1000000000"]
volumeMounts:
- name: ceph-vol
mountPath: /media
readOnly: false
volumes:
- name: ceph-vol
persistentVolumeClaim:
claimName: webdata-pvc-01
填满测试磁盘
在 pod 中执行下面命令
# df -h | grep media
/dev/rbd1 2.0G 33M 2.0G 2% /media
# dd if=/dev/zero of=/media/1.img bs=1M count=2048
# df -h | grep /media
/dev/rbd1 2.0G 2.0G 688K 100% /media
扩容 pv 流程
扩容
从 2Gi 扩容至 11Gi
# kubectl -n kubeceph patch pvc webdata-pvc-01 -p '{ "spec": { "resources": { "requests": { "storage": "11Gi" }}}}'
persistentvolumeclaim/webdata-pvc-01 patched
检查 pvc
参考下面扩容成功显示
# kubectl -n kubeceph get pvc webdata-pvc-01
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
webdata-pvc-01 Bound pvc-10b946df-02d8-4f3d-8cb6-bf34567b76b9 11Gi RWO ceph-kubernets 19m
在 pod 中执行下面命令
对于pod 而言，整个扩容过程无感知
# df -h | grep /media
/dev/rbd1 11G 2.0G 9.0G 19% /media
# dd if=/dev/zero of=/media/2.img bs=1M count=2048
# df -h | grep /media
/dev/rbd1 11G 4.0G 7.0G 37% /media
可能遇到扩容中的错误
pv 扩容不成功
执行 describe pvc 可以看到下面信息
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ProvisioningSucceeded 18m persistentvolume-controller Successfully provisioned volume pvc-10b946df-02d8-4f3d-8cb6-bf34567b76b9 using kubernetes.io/rbd
Warning VolumeResizeFailed 12m volume_expand error expanding volume "kubeceph/webdata-pvc-01" of plugin "kubernetes.io/rbd": rbd info failed, error: parse rbd info output failed: 2020-12-22 10:39:48.542866 7fa907e90d40 -1 did not load config file, using default settings.
2020-12-22 10:39:48.549702 7fa907e90d40 -1 Errors while parsing config file!
2020-12-22 10:39:48.549710 7fa907e90d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-12-22 10:39:48.549711 7fa907e90d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-12-22 10:39:48.549712 7fa907e90d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-12-22 10:39:48.551631 7fa907e90d40 -1 Errors while parsing config file!
2020-12-22 10:39:48.551641 7fa907e90d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-12-22 10:39:48.551642 7fa907e90d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-12-22 10:39:48.551643 7fa907e90d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-12-22 10:39:48.585077 7fa907e90d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.terry.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
{"name":"kubernetes-dynamic-pvc-0888014e-2c7f-409b-af94-3f70f9843b5d","size":2147483648,"objects":512,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.31e9a2ae8944a","format":2,"features":["layering"],"flags":[],"create_timestamp":"Tue Dec 22 10:33:50 2020"}, invalid character '-' after top-level value
Warning VolumeResizeFailed 12m volume_expand error expanding volume "kubeceph/webdata-pvc-01" of plugin "kubernetes.io/rbd": rbd info failed, error: parse rbd info output failed: 2020-12-22 10:39:48.639662 7f6089825d40 -1 did not load config file, using default settings.
解决方法
需要获取 ceph.conf ceph.client.terry.keyring
把对应文件存放至管理节点中 /etc/ceph 目录下
用户 keyrnog 需要对应你的 secret 中定义的用户
扫一扫
978
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
