2019年01月21日 16:45:42 毛小威 阅读数：381 标签： GlusterFSHeketiStorage更多
个人分类： StoragesKubernetes
版权声明：本文为博主记录文章，可能没有普遍的适用性。 https://blog.youkuaiyun.com/oyym_mv/article/details/86573113
GlusterFS作为分布式共享文件系统也用了很多年了，简单好用，但是一直没有系统化记录下来，很是零散。这次趁着Kubernets集成GlusterFS作为共享存储，尽量记录些东西。
GlusterFS操作记录(1) GlusterFS简述
GlusterFS操作记录(2) GlusterFS存储卷类型介绍
GlusterFS操作记录(3) GlusterFS架构介绍
GlusterFS操作记录(4) GlusterFS快速安装部署配置
GlusterFS操作记录(5) GlusterFS+Heketi配置(独立部署)
GlusterFS操作记录(6) GlusterFS+Heketi在Kubernets上的集成配置
一、GlusterFS集群配置
1. 准备3个节点，并初始化
| 主机 | ip地址 | 角色 | 备注 |
|---|---|---|---|
| gluster-server-1 | 10.99.7.11 | gluster-server，heketi | |
| gluster-server-2 | 10.99.7.12 | gluster-server | |
| gluster-server-2 | 10.99.7.12 | gluster-server | |
| gluster-client-1 | 10.99.7.10 | gluster-server | 挂载共享卷 |
初始化包括设置主机名称，网络，DNS等等，这里直接省略了。
配置/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.99.7.11 gluster-server-1
10.99.7.12 gluster-server-2
10.99.7.13 gluster-server-3
10.99.7.10 gluster-client-1
- 1
- 2
- 3
- 4
- 5
- 6
2. 准备一块磁盘或者分区
每个虚拟机准备一块或者多块磁盘，并分别格式化，这里举例/vdb (50GB)
qemu-img create -f qcow2 gluster_server_1.disk 50G
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/data/gluster_server_1.disk'/>
<target dev='vdb' bus='virtio'/>
</disk>
virsh edit gluster_server_1
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
# fdisk -l
Disk /dev/vda: ... # 系统盘
Disk /dev/vdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
- 1
- 2
- 3
- 4
- 5
- 6
- 7
3. 安装glusterFS
gluster根据用户有很多个组件，这里简单安装有些组件就不配置了。
# 每个服务端主机上执行
yum install centos-release-gluster -y
# 我们选择5.2版本
yum list glusterfs --showduplicates|sort -r
glusterfs.x86_64 5.2-1.el7 centos-gluster5
glusterfs.x86_64 5.1-1.el7 centos-gluster5
glusterfs.x86_64 5.0-1.el7 centos-gluster5
glusterfs.x86_64 3.12.2-18.el7 base
yum -y install glusterfs-server glusterfs-fuse
systemctl enable glusterd.service
systemctl start glusterd.service
systemctl status glusterd.service
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
4. 配置防火墙
这里是简单安装测试，没有深入配置防火墙。可以全部放行，或者直接关闭防火墙
# 直接关闭防火墙
systemctl stop firewalld.service
systemctl disable irewalld.service
- 1
- 2
- 3
5. 配置授信池（将节点加入到集群）
# 在gluster-server-1上将 2,3加入进来
gluster peer probe gluster-server-2
gluster peer probe gluster-server-3
# 在gluster-server-2或者3上（新版本没有必要了，之前版本需要时因为主机名称解析的问题）
gluster peer probe gluster-server-1
#任意节点
[root@gluster-server-1 ~]# gluster peer status
Number of Peers: 2
Hostname: gluster-server-2
Uuid: 6f32f6d4-9cd7-4b40-b7b6-100054b187f7
State: Peer in Cluster (Connected)
Hostname: gluster-server-3
Uuid: 5669dbef-3f71-4e3c-8fb1-9c6e11c0c434
State: Peer in Cluster (Connected)
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
6. 创建一个存储卷(测试)
备注：测试卷直接使用CentOS7上的一个普通目录，没有使用准备的磁盘（vdb是为heketi方式管理glusterfs准备的）
[各节点创建目录]
mkdir -p /data/brick1
gluster volume create volume0 replica 3 gluster-server-1:/data/brick1/volume0 gluster-server-2:/data/brick1/volume0 gluster-server-3:/data//brick1/volume0 force
#备注 这里使用force参数是因为我没有使用挂载磁盘，而是使用的root分区下的一个目录，不然会报错
gluster volume start volume0
#volume start: volume0: success
gluster volume info
Volume Name: volume0
Type: Replicate
Volume ID: 8936cef1-125b-4689-963c-b558725925bc
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster-server-1:/data/brick1/volume0
Brick2: gluster-server-2:/data/brick1/volume0
Brick3: gluster-server-3:/data/brick1/volume0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
7. 配置客户端，测试存储卷
yum install centos-release-gluster -y
yum -y install glusterfs glusterfs-fuse
mkdir /volume0
mount -t glusterfs gluster-server-1:/volume0 /volume0
[root@gluster-client-1 glusterfs]# df -h
gluster-server-1:/volume0 100G 2.1G 98G 3% /volume0
[test]
for i in `seq -w 1 100`; do cp -rp /var/log/messages /volume0/copy-test-$i; done
ls -lA /volume0/copy* | wc -l
100
[任意server节点]
备注：因为我们是3副本模式，只有3个brick，所有每个节点上都能看到全部100个文案
ls -lA /data/brick1/volume0/copy*
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
到这里，glusterfs的服务端集群安装配置以及客户端测试已经完成，默认配置的安装还是很简单的。
8. 删除测试卷
# 客户端解除mount
umount /volume0
# 服务端
gluster volume stop volume0
gluster volume delete volume0
rm -rf /data/brick1/volume0 # 所有服务节点执行
- 1
- 2
- 3
- 4
- 5
- 6
- 7
二、Heketi配置
Heketi是一个第三方程序，提供了一个RESTful 管理API以及命令行工具，可以用来管理GlusterFS卷的生命周期。
- Heketi动态在集群内选择bricks构建指定的volumes，以确保副本会分散到集群不同的故障域内。
- Heketi还支持任意数量的ClusterFS集群，以保证接入的云服务器不局限于单个GlusterFS集群。
Heketi以及Heketi-cli的安装可以直接在github上下载安装，centos7也提供了rpm安装方式。本文采用yum进行安装
1. 安装Heketi以及Heketi-cli
yum install heketi heketi-client -y
rpm -aq |grep heketi
heketi-client-8.0.0-1.el7.x86_64
heketi-8.0.0-1.el7.x86_64
- 1
- 2
- 3
- 4
- 5
搜优glusterfs服务节点加载模块
modprobe dm_thin_pool
- 1
2. 修改配置Heketi
vi /etc/heketi/heketi.json
# 这里只展示需要修改的配置，其他默认配置参考原始文件（这里采用默认值）
{
# 服务端口，可以根据需要进行修改，防止端口使用冲突
"port": "8080",
# 启用认证
"use_auth": true,
# 配置admin用的key
"admin": {
"key": "admin_secret"
},
# executor有三种，mock，ssh，Kubernets，这里使用ssh
"executor": "ssh",
# ssh相关配置
"sshexec": {
"keyfile": "/var/lib/heketi/id_rsa",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
# heketi数据库文件位置(这里是默认路径)
"db": "/var/lib/heketi/heketi.db",
# 日志级别(none, critical, error, warning, info, debug)
"loglevel" : "warning"
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
3. executor使用ssh，需要配置ssh秘钥访问所有gluster节点。
ssh-keygen -t rsa -q -f /var/lib/heketi/id_rsa -N ''
chown heketi:heketi /var/lib/heketi/id_rsa*
ssh-copy-id -i /var/lib/heketi/id_rsa root@gluster-server-1
ssh-copy-id -i /var/lib/heketi/id_rsa root@gluster-server-2
ssh-copy-id -i /var/lib/heketi/id_rsa root@gluster-server-3
- 1
- 2
- 3
- 4
- 5
4. 启动heketi服务
systemctl start heketi.service
systemctl enable heketi.service
systemctl status heketi.service
● heketi.service - Heketi Server
Loaded: loaded (/usr/lib/systemd/system/heketi.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2019-01-21 15:40:58 CST; 2s ago
Main PID: 2967 (heketi)
CGroup: /system.slice/heketi.service
└─2967 /usr/bin/heketi --config=/etc/heketi/heketi.json
Jan 21 15:40:58 gluster-server-1 systemd[1]: Started Heketi Server.
Jan 21 15:40:58 gluster-server-1 systemd[1]: Starting Heketi Server...
Jan 21 15:40:58 gluster-server-1 heketi[2967]: Heketi 8.0.0
Jan 21 15:40:58 gluster-server-1 heketi[2967]: Authorization loaded
Jan 21 15:40:58 gluster-server-1 heketi[2967]: Listening on port 8080
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
5. 通过topology文件添加glusterfs集群
heketi可以通过命令行手动添加glusterfs集群信息，也可以通过topology文件添加，实际产线中使用文件的方式易于维护。
# 没有就创建该文件
cat /etc/heketi/topology-cluster-1.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"10.99.7.11"
],
"storage": [
"10.99.7.11"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"10.99.7.12"
],
"storage": [
"10.99.7.12"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"10.99.7.13"
],
"storage": [
"10.99.7.13"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
}
]
}
]
}
heketi-cli --user admin --secret admin_secret --server http://10.99.7.11:8080 topology load --json /etc/heketi/topology-cluster-1.json
Creating cluster ... ID: 77b24830f331f2e12ca064d7daab3e43
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 10.99.7.11 ... ID: 52a177be9da6b0e00bd3863cb2565e6a
Adding device /dev/vdb ... OK
Creating node 10.99.7.12 ... ID: 508df329b279b6f73027f3aa9d81e005
Adding device /dev/vdb ... OK
Creating node 10.99.7.13 ... ID: 80ed10d504d363c0ba099855f9431f69
Adding device /dev/vdb ... OK
# 因为启用了认证，每次命令行都需要输入认证信息，因为就有一个集群，可以配置如下
alias heketi-cli='heketi-cli --server "http://10.99.7.11:8080" --user admin --secret admin_secret'
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
6. 通过heketi-cli管理glusterfs卷声明周期
# 创建一个测试卷(大小10GB，默认3副本)
heketi-cli volume create --size=10
Name: vol_a0eef26f5873eb842dd65db819b2fa65
Size: 10
Volume Id: a0eef26f5873eb842dd65db819b2fa65
Cluster Id: 77b24830f331f2e12ca064d7daab3e43
Mount: 10.99.7.12:vol_a0eef26f5873eb842dd65db819b2fa65
Mount Options: backup-volfile-servers=10.99.7.11,10.99.7.13
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
heketi-cli volume list
Id:a0eef26f5873eb842dd65db819b2fa65 Cluster:77b24830f331f2e12ca064d7daab3e43 Name:vol_a0eef26f5873eb842dd65db819b2fa65
# 通过gluster命令查询下
# gluster volume list
vol_a0eef26f5873eb842dd65db819b2fa65
# gluster volume info vol_a0eef26f5873eb842dd65db819b2fa65
Volume Name: vol_a0eef26f5873eb842dd65db819b2fa65
Type: Replicate
Volume ID: 9ce4e4d6-48d7-4685-87ff-6de80eaf4166
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.99.7.11:/var/lib/heketi/mounts/vg_6233c772878f135485b9ef759e8faa7d/brick_d365d8eb8bd0979a829f667b45d17df4/brick
Brick2: 10.99.7.12:/var/lib/heketi/mounts/vg_1a3767125942101c0304a52ccef213b7/brick_0dbefac4edcdb43ff719f1e254ecc996/brick
Brick3: 10.99.7.13:/var/lib/heketi/mounts/vg_e2c7ef9056c649121eef4c03eaed4110/brick_77d18cb0021f4180930583c887ee2933/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
# gluster volume status vol_a0eef26f5873eb842dd65db819b2fa65
Status of volume: vol_a0eef26f5873eb842dd65db819b2fa65
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.99.7.11:/var/lib/heketi/mounts/vg_
6233c772878f135485b9ef759e8faa7d/brick_d365
d8eb8bd0979a829f667b45d17df4/brick 49152 0 Y 4160
Brick 10.99.7.12:/var/lib/heketi/mounts/vg_
1a3767125942101c0304a52ccef213b7/brick_0dbe
fac4edcdb43ff719f1e254ecc996/brick 49153 0 Y 3491
Brick 10.99.7.13:/var/lib/heketi/mounts/vg_
e2c7ef9056c649121eef4c03eaed4110/brick_77d1
8cb0021f4180930583c887ee2933/brick 49152 0 Y 3374
Self-heal Daemon on localhost N/A N/A Y 4183
Self-heal Daemon on gluster-server-2 N/A N/A Y 3514
Self-heal Daemon on gluster-server-3 N/A N/A Y 3397
Task Status of Volume vol_a0eef26f5873eb842dd65db819b2fa65
------------------------------------------------------------------------------
There are no active volume tasks
# cat /etc/fstab
...
/dev/mapper/vg_6233c772878f135485b9ef759e8faa7d-brick_d365d8eb8bd0979a829f667b45d17df4 /var/lib/heketi/mounts/vg_6233c772878f135485b9ef759e8faa7d/brick_d365d8eb8bd0979a829f667b45d17df4 xfs rw,inode64,noatime,nouuid 1 2
# pvdisplay,vgdisplay,lvdisplay
# pvdisplay
--- Physical volume ---
PV Name /dev/vdb
VG Name vg_6233c772878f135485b9ef759e8faa7d
PV Size 50.00 GiB / not usable 132.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 12767
Free PE 10181
Allocated PE 2586
PV UUID amSE30-3PyS-jYQJ-LzYd-MziU-oPzw-qesxjU
# vgdisplay
--- Volume group ---
VG Name vg_6233c772878f135485b9ef759e8faa7d
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 21
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 49.87 GiB
PE Size 4.00 MiB
Total PE 12767
Alloc PE / Size 2586 / 10.10 GiB
Free PE / Size 10181 / <39.77 GiB
VG UUID W9IgyL-ld7N-fTD6-RL6D-YNwj-y6dd-9SrROI
# lvdisplay
--- Logical volume ---
LV Name tp_040396bb756b39a9df9642ffa5848114
VG Name vg_6233c772878f135485b9ef759e8faa7d
LV UUID GhsMPO-hZkz-Sgtu-a8x5-BShv-f431-tHpAtN
LV Write Access read/write
LV Creation host, time gluster-server-1, 2019-01-21 16:07:29 +0800
LV Pool metadata tp_040396bb756b39a9df9642ffa5848114_tmeta
LV Pool data tp_040396bb756b39a9df9642ffa5848114_tdata
LV Status available
# open 2
LV Size 10.00 GiB
Allocated pool data 0.18%
Allocated metadata 0.09%
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 252:2
--- Logical volume ---
LV Path /dev/vg_6233c772878f135485b9ef759e8faa7d/brick_d365d8eb8bd0979a829f667b45d17df4
LV Name brick_d365d8eb8bd0979a829f667b45d17df4
VG Name vg_6233c772878f135485b9ef759e8faa7d
LV UUID qXbd1K-rQB1-aLuU-LOj9-Jobq-No6D-p1FNYE
LV Write Access read/write
LV Creation host, time gluster-server-1, 2019-01-21 16:07:29 +0800
LV Pool name tp_040396bb756b39a9df9642ffa5848114
LV Status available
# open 1
LV Size 10.00 GiB
Mapped size 0.18%
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 252:4
客户端挂载使用
mount -t glusterfs gluster-server-1:/vol_a0eef26f5873eb842dd65db819b2fa65 /volume0/
cd /volume0/
for i in `seq -w 1 100`; do cp -rp /var/log/messages /volume0/copy-test-$i; done
ls -lA /volume0/copy* | wc -l
100
- 1
- 2
- 3
- 4
- 5
- 6
475
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
