ceph 创建 rgw 服务 + centos6 编译 s3fs_ceph rgw安装-优快云博客

本文链接：https://blog.youkuaiyun.com/signmem/article/details/128340290

目的

创建 ceph-radosgw 服务
server 版本 15.2
client centos6
通过 s3fs 对 ceph rgw 进行 mount 操作

ceph rgw 创建

服务器角色

主机名	ip	角色	os
ns-storage-020100.vclound.com	10.189.20.100	rgw server	centos7
ns-storage-020101.vclound.com	10.189.20.101	rgw server	centos7
ns-storage-020102.vclound.com	10.189.20.102	rgw server	centos7
s3client	10.199.1.1	s3client	centos6

创建 keyring 文件

三个服务器分别执行

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring 
chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring

生成 key

三个服务器分别执行

ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.rgw.ns-storage-020100 --gen-key 
ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.rgw.ns-storage-020101 --gen-key 
ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.rgw.ns-storage-020102 --gen-key

授权

三个服务器分别执行

ceph-authtool -n client.rgw.ns-storage-020100 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring 
ceph-authtool -n client.rgw.ns-storage-020101 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring 
ceph-authtool -n client.rgw.ns-storage-020102 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

查询其中一台服务器，参考授权文件

[client.rgw.ns-storage-020100]
        key = AQDd45pjDdsBFBAA83J0gG6iep9UWoAmhKWjIA==
        caps mon = "allow rwx"
        caps osd = "allow rwx"

导入 key 到 ceph

三个服务器分别执行

ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.ns-storage-020100 -i /etc/ceph/ceph.client.radosgw.keyring 
ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.ns-storage-020101 -i /etc/ceph/ceph.client.radosgw.keyring 
ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.ns-storage-020102 -i /etc/ceph/ceph.client.radosgw.keyring

创建 ceph rgw 配置

三个服务器分别执行

mkdir /var/log/radosgw
chown ceph:ceph /var/log/radosgw   <- 放日志


vim /etc/ceph/ceph.conf    
--- 新增下面配置
[client.rgw.ns-storage-020100] 
host=ns-storage-020100 
keyring=/etc/ceph/ceph.client.radosgw.keyring 
log file=/var/log/radosgw/client.radosgw.gateway.log 
rgw_frontends = "civetweb port=80"   <- 指定 rgw http 端口

[client.rgw.ns-storage-020101] 
host=ns-storage-020100 
keyring=/etc/ceph/ceph.client.radosgw.keyring 
log file=/var/log/radosgw/client.radosgw.gateway.log 
rgw_frontends = "civetweb port=80"

[client.rgw.ns-storage-020102] 
host=ns-storage-020100 
keyring=/etc/ceph/ceph.client.radosgw.keyring 
log file=/var/log/radosgw/client.radosgw.gateway.log 
rgw_frontends = "civetweb port=80"

--- 新增上面配置

创建服务启动文件

三个服务器分别执行

 cp /usr/lib/systemd/system/ceph-radosgw@.service /usr/lib/systemd/system/ceph-radosgw@rgw.ns-storage-020100.service

 cp /usr/lib/systemd/system/ceph-radosgw@.service /usr/lib/systemd/system/ceph-radosgw@rgw.ns-storage-020101.service

 cp /usr/lib/systemd/system/ceph-radosgw@.service /usr/lib/systemd/system/ceph-radosgw@rgw.ns-storage-020102.service

启动 rgw

systemctl start  ceph-radosgw@rgw.ns-storage-020100
systemctl start  ceph-radosgw@rgw.ns-storage-020101
systemctl start  ceph-radosgw@rgw.ns-storage-020102

ceph rgw default pool

服务启动后会默认生成下面新 pool

# ceph df
--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED     RAW USED  %RAW USED
hdd    84 TiB  83 TiB  1.5 TiB   1.5 TiB       1.80
TOTAL  84 TiB  83 TiB  1.5 TiB   1.5 TiB       1.80

--- POOLS ---
POOL                   ID  PGS  STORED   OBJECTS  USED     %USED  MAX AVAIL
device_health_metrics  12   64      0 B       28      0 B      0     23 TiB
volumes                13   32      0 B        0      0 B      0     23 TiB
.rgw.root              14  495  1.3 KiB        4  1.3 KiB      0     23 TiB
default.rgw.log        15  512  3.4 KiB      175  3.4 KiB      0     23 TiB
default.rgw.control    16  512      0 B        8      0 B      0     23 TiB
default.rgw.meta       17  512      0 B        0      0 B      0     23 TiB

参考池创建过程

# ceph -s
  cluster:
    id:     aa0e0954-4abd-41d3-9fcf-b38a8159167a
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ns-storage-020100,ns-storage-020101,ns-storage-020102 (age 6h)
    mgr: ns-storage-020101(active, since 23h), standbys: ns-storage-020100, ns-storage-020102
    osd: 24 osds: 24 up (since 6h), 24 in (since 6h)
    rgw: 3 daemons active (ns-storage-020100, ns-storage-020101, ns-storage-020102)

  task status:

  data:
    pools:   6 pools, 1856 pgs
    objects: 215 objects, 4.7 KiB
    usage:   1.5 TiB used, 83 TiB / 84 TiB avail
    pgs:     0.054% pgs not active
             1855 active+clean
             1    peering

  progress:
    PG autoscaler decreasing pool 14 PGs from 512 to 32 (7m)
      [====........................] (remaining: 34m)
    PG autoscaler decreasing pool 17 PGs from 512 to 8 (6m)
      [===.........................] (remaining: 46m)
    PG autoscaler decreasing pool 15 PGs from 512 to 32 (6m)
      [====........................] (remaining: 31m)
    PG autoscaler decreasing pool 16 PGs from 512 to 32 (6m)
      [===.........................] (remaining: 37m)

rgw 接口测试

# curl  http://ns-storage-020100.vclound.com
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

创建 bucket 用户

# radosgw-admin user create --uid="dba" --display-name="backup db data user"

{
    "user_id": "dba",
    "display_name": "backup db data user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "dba",
            "access_key": "MZAX7JQS3V057P68J7KZ",
            "secret_key": "HpJGPMeBfLolR54p6cyZfY3UE7AKEFeAUMrTcgFz"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

# radosgw-admin user create --uid="restore" --display-name="read db data user"
{
    "user_id": "restore",
    "display_name": "read db data user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "restore",
            "access_key": "QMBIJK4MA9K0WQU9F3HM",
            "secret_key": "OAAh4YuXpZOtQKgWpDjnSAjtnD0BQ7H9elDC1WW7"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

记住 access_key， secret_key
假如忘记了，可以查询用户信息

# radosgw-admin user info --uid dba

创建 bucket

服务器上执行（centos7）
安装软件

# yum install -y s3fs-fuse  python-boto

创建秘钥

# echo MZAX7JQS3V057P68J7KZ:HpJGPMeBfLolR54p6cyZfY3UE7AKEFeAUMrTcgFz > ${HOME}/.passwd-s3fs

创建 bucket 脚本
创建 backupdata bucket

vi s3test.py
import boto.s3.connection

access_key = 'MZAX7JQS3V057P68J7KZ'
secret_key = 'HpJGPMeBfLolR54p6cyZfY3UE7AKEFeAUMrTcgFz'
conn = boto.connect_s3(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
        host='ns-storage-020100', port=80,
        is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
       )

bucket = conn.create_bucket('backupdata')
for bucket in conn.get_all_buckets():
    print "{name} {created}".format(
        name=bucket.name,
        created=bucket.creation_date,
    )

创建 restore buckey

import boto.s3.connection

access_key = 'QMBIJK4MA9K0WQU9F3HM'
secret_key = 'OAAh4YuXpZOtQKgWpDjnSAjtnD0BQ7H9elDC1WW7'
conn = boto.connect_s3(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
        host='ns-storage-020100', port=80,
        is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
       )

bucket = conn.create_bucket('restore')
for bucket in conn.get_all_buckets():
    print "{name} {created}".format(
        name=bucket.name,
        created=bucket.creation_date,
    )

执行脚本

# python s3test.py
backupdata 2022-12-15T12:41:46.100Z
# python /tmp/s3test2.py
restore 2022-12-16T03:35:26.020Z

参考创建后的 pool

# ceph df
--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED     RAW USED  %RAW USED
hdd    84 TiB  83 TiB  1.0 TiB   1.1 TiB       1.25
TOTAL  84 TiB  83 TiB  1.0 TiB   1.1 TiB       1.25

--- POOLS ---
POOL                        ID  PGS  STORED   OBJECTS  USED     %USED  MAX AVAIL
device_health_metrics       12   64      0 B       28      0 B      0     24 TiB
volumes                     13   32      0 B        0      0 B      0     24 TiB
.rgw.root                   14   32  1.3 KiB        4  1.3 KiB      0     24 TiB
default.rgw.log             15   32   19 KiB      207   19 KiB      0     24 TiB
default.rgw.control         16   32      0 B        8      0 B      0     24 TiB
default.rgw.meta            17    8  1.6 KiB       10  1.6 KiB      0     24 TiB
default.rgw.buckets.index   18    8      0 B       22      0 B      0     24 TiB
default.rgw.buckets.data    19   32  1.0 GiB    1.16k  1.0 GiB      0     24 TiB
default.rgw.buckets.non-ec  20   32      0 B        0      0 B      0     24 TiB

当前两个 bucket 都存放在相同的 pool 里面

挂载测试 (centos7)

分别创建秘钥文件 (对应两个用户)

# cat .passwd-s3fs-backupdata  （属性 0600）
MZAX7JQS3V057P68J7KZ:HpJGPMeBfLolR54p6cyZfY3UE7AKEFeAUMrTcgFz
# cat .passwd-s3fs-resotre（属性 0600）
QMBIJK4MA9K0WQU9F3HM:OAAh4YuXpZOtQKgWpDjnSAjtnD0BQ7H9elDC1WW7

创建对应目录

mkdir -p /mnt/backupdata  /mnt/restore

分别挂载不同 bucket

s3fs backupdata /mnt/backupdata  -o passwd_file=${HOME}/.passwd-s3fs-backupdata -o url=http://10.189.20.100 -o use_path_request_style

s3fs restore /mnt/restore -o passwd_file=${HOME}/.passwd-s3fs-resotre -o url=http://10.189.20.100 -o use_path_request_style

验证挂载

# df -h | grep s3fs
s3fs             16E     0   16E    0% /mnt/backupdata
s3fs             16E     0   16E    0% /mnt/restore

centos6 build s3fs

卸载软件，确保最新 fuse 可以使用

yum remove fuse fuse* fuse-devel

安装编译需要软件

yum install automake gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

编译新版本 fuse

wget https://github.com/libfuse/libfuse/releases/download/fuse-2.9.7/fuse-2.9.7.tar.gz
tar -xzvf fuse-2.9.7.tar.gz
rm -f fuse-2.9.7.tar.gz
cd fuse-2.9.7 
./configure --prefix=/usr
make
make install
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
ldconfig

验证 fuse 版本

pkg-config --modversion fuse
2.9.7    <- 正常会返回

编译新版本 s3fs-fuse

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

参考编译后版本

# uname -r
2.6.32-504.23.4.el6.x86_64
# s3fs --version
Amazon Simple Storage Service File System V1.91 (commit:14eb1a7) with OpenSSL