k8s集群中使用GlusterFS存储

最新推荐文章于 2025-05-29 20:16:56 发布

思维的深度

最新推荐文章于 2025-05-29 20:16:56 发布

阅读量1.6k

点赞数

CC 4.0 BY-SA版权

分类专栏： k8s 文章标签： k8s pv/pvc k8s存储 k8s glusterfs

本文链接：https://blog.youkuaiyun.com/skh2015java/article/details/108409654

k8s 专栏收录该内容

15 篇文章

订阅专栏

本文介绍如何在Kubernetes集群上快速部署GlusterFS分布式文件系统，包括节点标记、DaemonSet安装、RBAC权限设置、StorageClass创建及PVC配置，实现存储资源的高效管理和利用。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

glusterfs在k8s集群中快速搭建的方法：

https://github.com/kubernetes-retired/external-storage/tree/master/gluster/glusterfs

前提：需要k8s集群，至少需要2个node节点

本示例使用的k8s版本为1.18

安装使用

1. 给每个node节点添加标记

$ kubectl label nodes <storage-node-name> storagenode=glusterfs

2.安装GlusterFS DaemonSet

yaml地址：https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/glusterfs-daemonset.yaml

1.18版本需要将apiVersion改为apps/v1

spec加上selector便签

参考示例：glusterfs-daemonset.yaml

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: glusterfs
  labels:
    glusterfs: daemonset
  annotations:
    description: GlusterFS DaemonSet
    tags: glusterfs
spec:
selector:
   matchLabels:
    glusterfs: pod
  template:
    metadata:
      name: glusterfs
      labels:
        glusterfs-node: pod
    spec:
      nodeSelector:
        storagenode: glusterfs
      hostNetwork: true
      containers:
      - image: gluster/gluster-centos:latest
        imagePullPolicy: IfNotPresent
        name: glusterfs
        volumeMounts:
        - name: glusterfs-heketi
          mountPath: "/var/lib/heketi"
        - name: glusterfs-run
          mountPath: "/run"
        - name: glusterfs-lvm
          mountPath: "/run/lvm"
        - name: glusterfs-etc
          mountPath: "/etc/glusterfs"
        - name: glusterfs-logs
          mountPath: "/var/log/glusterfs"
        - name: glusterfs-config
          mountPath: "/var/lib/glusterd"
        - name: glusterfs-dev
          mountPath: "/dev"
        - name: glusterfs-misc
          mountPath: "/var/lib/misc/glusterfsd"
        - name: glusterfs-cgroup
          mountPath: "/sys/fs/cgroup"
          readOnly: true
        - name: glusterfs-ssl
          mountPath: "/etc/ssl"
          readOnly: true
        securityContext:
          capabilities: {}
          privileged: true
        readinessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - systemctl status glusterd.service
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 15
        livenessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - systemctl status glusterd.service
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 15
      volumes:
      - name: glusterfs-heketi
        hostPath:
          path: "/var/lib/heketi"
      - name: glusterfs-run
      - name: glusterfs-lvm
        hostPath:
          path: "/run/lvm"
      - name: glusterfs-etc
        hostPath:
          path: "/etc/glusterfs"
      - name: glusterfs-logs
        hostPath:
          path: "/var/log/glusterfs"
      - name: glusterfs-config
        hostPath:
          path: "/var/lib/glusterd"
      - name: glusterfs-dev
        hostPath:
          path: "/dev"
      - name: glusterfs-misc
        hostPath:
          path: "/var/lib/misc/glusterfsd"
      - name: glusterfs-cgroup
        hostPath:
          path: "/sys/fs/cgroup"
      - name: glusterfs-ssl
        hostPath:
          path: "/etc/ssl"

$ kubectl create -f glusterfs-daemonset.yaml

创建glusterfs的daemonset

$ kubectl get pods -l glusterfs=pod -o wide

3. 将节点添加到trusted pool

$ kubectl exec -ti glusterfs-drlgp gluster peer probe 192.168.20.165

$ kubectl exec -ti glusterfs-z5m4j gluster peer probe 192.168.20.45

4. RBAC权限(Kubernetes 1.8+)

Kubernetes 1.8+ provisioner访问k8s集群的API需要绑定一个ServiceAccount

rbac.yaml地址：

https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: glfs-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: glfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events", "pods/exec"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-glfs-provisioner
subjects:
  - kind: ServiceAccount
    name: glfs-provisioner
    namespace: default
# update namespace above to your namespace in order to make this work
roleRef:
  kind: ClusterRole
  name: glfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

kubectl create -f rbac.yaml

5.创建StorageClass

storageclass.yaml

地址：

https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: glusterfs-simple
provisioner: gluster.org/glusterfs-simple
parameters:
  forceCreate: "true"
  brickrootPaths: "192.168.20.165:/tmp/,192.168.20.45:/tmp"

$ kubectl create -f storageclass.yaml

6.创建pvc

pvc.yaml

pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gluster-simple-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "glusterfs-simple"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

$ kubectl create -f pvc.yaml

$ kubectl get pv,pvc