本文尝试在龙蜥Anolis OS8.6QU1上通过cephadm部署ceph17.2.0分布式块存储，实现国产化操作系统上的存储系统构建,本步骤为完成首节点mon节点和图形界面，完成其它节点的基础安装，完成添加其它主机和添加mgr节点,完成添加mgr节点，添加OSD,再进行的第六步操作-部署iscsi服务。
一、验证当前安装情况
[root@ceph1 opt]# ceph -s
cluster:
id: 58a31a00-bf04-11ed-a192-000e1e99b662
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 3h)
mgr: ceph1.biobao(active, since 3h), standbys: ceph3.hlsirv, ceph2.dojvls
osd: 88 osds: 88 up (since 98m), 88 in (since 98m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 577 KiB
usage: 3.8 GiB used, 72 TiB / 72 TiB avail
pgs: 1 active+clean
二、安装ceph iscsi gateway相关软件
分发软件
[root@ceph1 opt]# scp -r iscsiGW-rpmfiles root@ceph3:/opt
ceph-iscsi-tools-2.2-1.el8.noarch.rpm 100% 53KB 23.1MB/s 00:00
ceph-iscsi-3.5-1.el8.noarch.rpm 100% 211KB 90.2MB/s 00:00
libtcmu-1.5.2.31.gda0fb70-0.el8.x86_64.rpm 100% 47KB 45.6MB/s 00:00
libtcmu-devel-1.5.2.31.gda0fb70-0.el8.x86_64.rpm 100% 6844 9.5MB/s 00:00
tcmu-runner-1.5.2.31.gda0fb70-0.el8.x86_64.rpm 100% 126KB 80.9MB/s 00:00
[root@ceph1 opt]# scp -r iscsiGW-rpmfiles root@ceph4:/opt
ceph-iscsi-tools-2.2-1.el8.noarch.rpm 100% 53KB 21.2MB/s 00:00
ceph-iscsi-3.5-1.el8.noarch.rpm 100% 211KB 90.3MB/s 00:00
libtcmu-1.5.2.31.gda0fb70-0.el8.x86_64.rpm 100% 47KB 45.0MB/s 00:00
libtcmu-devel-1.5.2.31.gda0fb70-0.el8.x86_64.rpm 100% 6844 7.2MB/s 00:00
tcmu-runner-1.5.2.31.gda0fb70-0.el8.x86_64.rpm 100% 126KB 75.0MB/s 00:00 安装软件（以ceph-3为例）
[root@ceph1 opt]# ssh ceph3
Last login: Sun Mar 12 18:03:48 2023
[root@ceph3 ~]# cd /opt
[root@ceph3 opt]# ll
总用量 12
-rw-r--r-- 1 root root 271 3月 12 09:11 bat_add_osd.sh
-rw-r--r-- 1 root root 109 3月 12 09:01 bat_wipe_fs
-rw-r--r-- 1 root root 424 3月 10 17:10 bond.sh
drwxr-xr-x 4 root root 71 3月 11 19:36 ceph17.2.0
drwxr-xr-x 2 root root 250 3月 13 09:27 iscsiGW-rpmfiles
[root@ceph3 opt]# cd iscsiGW-rpmfiles/
[root@ceph3 iscsiGW-rpmfiles]# ls
ceph-iscsi-3.5-1.el8.noarch.rpm libtcmu-1.5.2.31.gda0fb70-0.el8.x86_64.rpm tcmu-runner-1.5.2.31.gda0fb70-0.el8.x86_64.rpm
ceph-iscsi-tools-2.2-1.el8.noarch.rpm libtcmu-devel-1.5.2.31.gda0fb70-0.el8.x86_64.rpm
[root@ceph3 iscsiGW-rpmfiles]# dnf install *
上次元数据过期检查：2:22:40 前，执行于 2023年03月13日 星期一 07时06分20秒。
依赖关系解决。
=========================================================================================================================================================
软件包 架构 版本 仓库 大小
=========================================================================================================================================================
安装:
ceph-iscsi noarch 3.5-1.el8 @commandline 211 k
ceph-iscsi-tools noarch 2.2-1.el8 @commandline 53 k
libtcmu x86_64 1.5.2.31.gda0fb70-0.el8 @commandline 47 k
libtcmu-devel x86_64 1.5.2.31.gda0fb70-0.el8 @commandline 6.7 k
tcmu-runner x86_64 1.5.2.31.gda0fb70-0.el8 @commandline 126 k
...
已安装:
avahi-libs-0.7-20.an8.x86_64 ceph-iscsi-3.5-1.el8.noarch ceph-iscsi-tools-2.2-1.el8.noarch
glusterfs-6.0-61.3.0.1.an8.x86_64 glusterfs-api-6.0-61.3.0.1.an8.x86_64 glusterfs-client-xlators-6.0-61.3.0.1.an8.x86_64
glusterfs-libs-6.0-61.3.0.1.an8.x86_64 libtcmu-1.5.2.31.gda0fb70-0.el8.x86_64 libtcmu-devel-1.5.2.31.gda0fb70-0.el8.x86_64
libuv-1:1.41.1-1.0.1.an8.x86_64 pcp-5.3.6-1.an8.x86_64 pcp-conf-5.3.6-1.an8.x86_64
pcp-libs-5.3.6-1.an8.x86_64 pcp-pmda-lio-5.3.6-1.an8.x86_64 pcp-selinux-5.3.6-1.an8.x86_64
python3-babel-2.5.1-7.0.1.an8.noarch python3-cffi-1.11.5-5.0.1.an8.x86_64 python3-chardet-3.0.4-7.el8.noarch
python3-click-6.7-8.el8.noarch python3-configshell-1:1.1.28-1.an8.noarch python3-cryptography-3.2.1-5.an8.x86_64
python3-flask-1:0.12.2-4.el8.noarch python3-idna-2.5-5.el8.noarch python3-itsdangerous-0.24-14.el8.noarch
python3-jinja2-2.10.1-3.an8.noarch python3-kmod-0.9-20.el8.x86_64 python3-markupsafe-0.23-19.el8.x86_64
python3-netifaces-0.10.6-4.el8.x86_64 python3-pcp-5.3.6-1.an8.x86_64 python3-ply-3.9-9.an8.noarch
python3-pyOpenSSL-19.0.0-1.an8.noarch python3-pycparser-2.14-14.el8.noarch python3-pyparsing-2.1.10-7.el8.noarch
python3-pysocks-1.6.8-3.el8.noarch python3-pytz-2017.2-9.el8.noarch python3-requests-2.20.0-2.1.el8.noarch
python3-rtslib-2.1.74-1.an8.noarch python3-urllib3-1.24.2-5.an8.noarch python3-urwid-1.3.1-4.el8.x86_64
python3-werkzeug-0.12.2-4.el8.noarch tcmu-runner-1.5.2.31.gda0fb70-0.el8.x86_64
完毕！
修改iscsi-gateway配置文件
[root@ceph3 iscsiGW-rpmfiles]# vi /etc/ceph/iscsi-gateway.cfg
[root@ceph3 iscsiGW-rpmfiles]# cat /etc/ceph/iscsi-gateway.cfg
[config]
cluster_name = ceph
pool = iscsi_pools
gateway_keyring = ceph.client.admin.keyring
api_secure = false
api_user = admin
api_password = 设置api的接口密码
api_port = 5000
trusted_ip_list = 192.168.188.1,192.168.188.2,192.168.188.3,192.168.188.4修改rbd-target-api配置文件，主要是修改StartLimitInterval项和增加RestartSec项，如下：
[root@ceph3 iscsiGW-rpmfiles]# cp /usr/lib/systemd/system/rbd-target-api.service{,.bak}
[root@ceph3 iscsiGW-rpmfiles]# vi /usr/lib/systemd/system/rbd-target-api.service
[root@ceph3 iscsiGW-rpmfiles]# diff /usr/lib/systemd/system/rbd-target-api.service /usr/lib/systemd/system/rbd-target-api.service.bak
22,23c22
< RestartSec=60s
< StartLimitInterval=30s
---
> StartLimitInterval=30min启动服务
[root@ceph3 iscsiGW-rpmfiles]# systemctl daemon-reload
[root@ceph3 iscsiGW-rpmfiles]# systemctl enable --now rbd-target-api
Created symlink /etc/systemd/system/multi-user.target.wants/rbd-target-api.service → /usr/lib/systemd/system/rbd-target-api.service.
[root@ceph3 iscsiGW-rpmfiles]# systemctl status rbd-target-api
● rbd-target-api.service - Ceph iscsi target configuration API
Loaded: loaded (/usr/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-03-13 09:34:14 CST; 7s ago
Main PID: 23935 (rbd-target-api)
Tasks: 52 (limit: 1649436)
Memory: 81.6M
CGroup: /system.slice/rbd-target-api.service
├─23935 /usr/bin/python3.6 -s /usr/bin/rbd-target-api
└─24054 /usr/libexec/platform-python -s /usr/bin/ceph -n client.admin --conf /etc/ceph/ceph.conf osd blocklist rm 192.168.188.3:6976/36352430>
3月 13 09:34:14 ceph3 systemd[1]: Started Ceph iscsi target configuration API.
3月 13 09:34:15 ceph3 rbd-target-api[23935]: Started the configuration object watcher
3月 13 09:34:15 ceph3 rbd-target-api[23935]: Checking for config object changes every 1s
3月 13 09:34:15 ceph3 rbd-target-api[23935]: Processing osd blocklist entries for this node
3月 13 09:34:16 ceph3 rbd-target-api[23935]: Removing blocklisted entry for this host : 192.168.188.3:0/959736569
3月 13 09:34:18 ceph3 rbd-target-api[23935]: Successfully removed blocklist entry
3月 13 09:34:18 ceph3 rbd-target-api[23935]: Removing blocklisted entry for this host : 192.168.188.3:6977/3635243000
3月 13 09:34:20 ceph3 rbd-target-api[23935]: Successfully removed blocklist entry
3月 13 09:34:20 ceph3 rbd-target-api[23935]: Removing blocklisted entry for this host : 192.168.188.3:6976/3635243000
[root@ceph3 iscsiGW-rpmfiles]# systemctl enable --now rbd-target-gw && systemctl status rbd-target-gw
Created symlink /etc/systemd/system/multi-user.target.wants/rbd-target-gw.service → /usr/lib/systemd/system/rbd-target-gw.service.
● rbd-target-gw.service - Setup system to export rbd images through LIO
Loaded: loaded (/usr/lib/systemd/system/rbd-target-gw.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-03-13 09:35:26 CST; 28ms ago
Main PID: 24559 ((arget-gw))
Tasks: 1 (limit: 1649436)
Memory: 640.0K
CGroup: /system.slice/rbd-target-gw.service
└─24559 (arget-gw)
3月 13 09:35:26 ceph3 systemd[1]: Started Setup system to export rbd images through LIO.对ceph4节点完成相同的安装和服务启动操作。
三、详细配置
在ceph3上操作即可：
[root@ceph3 iscsiGW-rpmfiles]# gwcli
Warning: Could not load preferences file /root/.gwcli/prefs.bin.
/> cd /iscsi-targets
/iscsi-targets> create iqn.2021-10.com.cqmc.iscsi-gw:iscsi-gw
ok
/iscsi-targets> cd iqn.2021-10.com.cqmc.iscsi-gw:iscsi-gw/gateways
/iscsi-target...i-gw/gateways> create skipchecks=true ceph3 192.168.188.3
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...i-gw/gateways> create skipchecks=true ceph4 192.168.188.4
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...i-gw/gateways> ls
o- gateways .................................................................................................. [Up: 2/2, Portals: 2]
o- ceph3 .................................................................................................... [10.227.17.205 (UP)]
o- ceph4 .................................................................................................... [10.227.17.206 (UP)]
/iscsi-target...i-gw/gateways> cd /disks
/disks> create pool=iscsi_pools image=disk001 size=6T
ok
/disks> ls
o- disks .......................................................................................................... [6.0T, Disks: 1]
o- iscsi_pools .............................................................................................. [iscsi_pools (6.0T)]
o- disk001 ............................................................................... [iscsi_pools/disk001 (Unknown, 6.0T)]
/disks> cd /iscsi-targets/iqn.2021-10.com.cqmc.iscsi-gw:iscsi-gw/hosts
/iscsi-target...scsi-gw/hosts> create iqn.1991-05.com.microsoft:citrix7-sas.cqx4a.net
ok
/iscsi-target...sas.cqx4a.net> auth username=定义client连接用户名 password=定义client连接密码
ok
/iscsi-target...sas.cqx4a.net> disk action=add disk=iscsi_pools/disk001
ok
/iscsi-target...sas.cqx4a.net> ls
o- iqn.1991-05.com.microsoft:citrix7-sas.cqx4a.net .................................................... [Auth: CHAP, Disks: 1(6.0T)]
o- lun 0 ............................................................................... [iscsi_pools/disk001(6.0T), Owner: ceph3]
/iscsi-target...sas.cqx4a.net> cd /
/> ls
o- / ......................................................................................................................... [...]
o- cluster ......................................................................................................... [Clusters: 1]
| o- ceph ............................................................................................................ [HEALTH_OK]
| o- pools .......................................................................................................... [Pools: 2]
| | o- .mgr ................................................................ [(x3), Commit: 0.00Y/23903414M (0%), Used: 852512b]
| | o- iscsi_pools ........................................................... [(x3), Commit: 6.0T/23903414M (26%), Used: 2322b]
| o- topology ............................................................................................... [OSDs: 88,MONs: 3]
o- disks ........................................................................................................ [6.0T, Disks: 1]
| o- iscsi_pools ............................................................................................ [iscsi_pools (6.0T)]
| o- disk001 .............................................................................. [iscsi_pools/disk001 (Online, 6.0T)]
o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
o- iqn.2021-10.com.cqmc.iscsi-gw:iscsi-gw ............................................................ [Auth: None, Gateways: 2]
o- disks .......................................................................................................... [Disks: 1]
| o- iscsi_pools/disk001 .............................................................................. [Owner: ceph3, Lun: 0]
o- gateways ............................................................................................ [Up: 2/2, Portals: 2]
| o- ceph3 .............................................................................................. [192.168.188.3 (UP)]
| o- ceph4 .............................................................................................. [192.168.188.4 (UP)]
o- host-groups .................................................................................................. [Groups : 0]
o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
o- iqn.1991-05.com.microsoft:citrix7-sas.cqx4a.net ............................................ [Auth: CHAP, Disks: 1(6.0T)]
o- lun 0 ....................................................................... [iscsi_pools/disk001(6.0T), Owner: ceph3]
/> exit本例在ceph3\ceph4上启用了iscsiGateway，在iscsi_pools上新建了一块6T的卷，并关联到iqn为“iqn.1991-05.com.microsoft:citrix7-sas.cqx4a.net”的windows客户端。
ceph4无需操作，验证信息：
[root@ceph4 ~]# gwcli info
HTTP mode : http
Rest API port : 5000
Local endpoint : http://localhost:5000/api
Local Ceph Cluster : ceph
2ndary API IP's : 192.168.188.1,192.168.188.2,192.168.188.3,192.168.188.4四、添加iscsiGateway信息到dashboard
在ceph1操作，编写api接口信息文件
[root@ceph1 opt]# vi /etc/ceph/iscsi1.api
[root@ceph1 opt]# cat /etc/ceph/iscsi1.api
http://admin:此处为前面iscsi-gateway文件中设置的api接口密码@192.168.188.3:5000
[root@ceph1 opt]# vi /etc/ceph/iscsi2.api
[root@ceph1 opt]# cat /etc/ceph/iscsi2.api
http://admin:此处为前面iscsi-gateway文件中设置的api接口密码@192.168.188.4:5000添加iscsiGateway到dashboard
[root@ceph1 opt]# ceph dashboard iscsi-gateway-add -i /etc/ceph/iscsi1.api iscsi1
Success
[root@ceph1 opt]# ceph dashboard iscsi-gateway-list
{"gateways": {"ceph3": {"service_url": "http://admin:*******@192.168.188.3:5000"}}}
[root@ceph1 opt]# ceph dashboard iscsi-gateway-add -i /etc/ceph/iscsi2.api iscsi2
Success
[root@ceph1 opt]# ceph dashboard iscsi-gateway-list
{"gateways": {"ceph3": {"service_url": "http://admin:*******@192.168.188.3:5000"}, "ceph4": {"service_url": "http://admin:*******@192.168.188.4:5000"}}}图形界面验证
页面总览
Block-iSCSI界面
Pools界面
至此，我们在国产化操作系统龙蜥Anolis OS8.6QU1上完成了裸容量72T的4节点存储系统，并向外提供iscsi存储服务。
扫一扫
786
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
