Ceph 分布式存储系统:从入门到精通

Ceph 是一款开源、可扩展、高可用的分布式存储系统,支持对象存储、块存储和文件存储三大功能,广泛应用于云计算、大数据和企业级存储场景。Ceph 具有无单点故障、高容错性和动态扩展性,适合作为私有云和公有云存储解决方案。本文将带你从 Ceph 安装配置开始,深入到集群运维和企业实战案例。


1. Ceph 核心概念

组件描述
MON(Monitor)负责维护集群状态、健康检查和集群映射
OSD对象存储守护进程,存储数据和副本
MDS元数据服务器,管理 Ceph 文件系统元数据(仅 CephFS 使用)
RADOS可靠自治分布式对象存储(Ceph 的底层存储引擎)
RGWRADOS Gateway,提供 S3 和 Swift 兼容 API


2. Ceph 安装与环境配置

2.1 环境要求

  • 操作系统:CentOS 7/8 或 Ubuntu 20.04+

  • 最少 3 台服务器

  • 推荐配置:4GB 内存 / 2 核 CPU / 100GB 磁盘


2.2 安装 Ceph(Octopus 版本示例)

1. 添加 Ceph 仓库

bash

复制编辑

sudo yum install -y https://download.ceph.com/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm sudo yum update -y

2. 安装 Ceph 部署工具

bash

复制编辑

sudo yum install -y ceph-deploy


2.3 部署 Ceph 集群

1. 初始化集群

bash

复制编辑

mkdir my-cluster && cd my-cluster ceph-deploy new mon1 mon2 mon3


2. 安装 Ceph 软件

bash

复制编辑

ceph-deploy install mon1 mon2 mon3 osd1 osd2 osd3


3. 创建 Monitor 节点

bash

复制编辑

ceph-deploy mon create-initial


4. 部署 OSD

bash

复制编辑

ceph-deploy osd create --data /dev/sdb osd1 ceph-deploy osd create --data /dev/sdb osd2 ceph-deploy osd create --data /dev/sdb osd3


5. 查看集群状态

bash

复制编辑

ceph -s

输出:


yaml

复制编辑

cluster: id: f92abf31-8f9d-4ebf-8bc3-7e34fa0d9d5d health: HEALTH_OK


3. Ceph 基础功能

3.1 对象存储(RADOS Gateway)

启动 RGW

bash

复制编辑

ceph-deploy rgw create mon1

访问 S3 API
  • Endpoint: http://mon1:7480

  • 使用 S3 客户端工具(如 s3cmd)上传下载对象。


3.2 块存储(RBD)

创建存储池

bash

复制编辑

ceph osd pool create rbd_pool 128

创建 RBD 镜像

http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E6%80%8E%E4%B9%88%E4%B9%B0%E7%90%83%E6%8A%95%E6%B3%A8%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E6%AD%A3%E8%A7%84%E4%B9%B0%E7%90%83%E7%BD%91%E7%AB%99%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E8%B6%B3%E7%90%83%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E5%B9%B3%E5%8F%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E5%B9%B3%E5%8F%B0%E5%85%AC%E5%8F%B8%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E5%8E%BB%E5%93%AA%E9%87%8C%E4%B9%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E8%B5%9B%E5%93%AA%E9%87%8C%E4%B9%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E8%B5%9B%E6%80%8E%E4%B9%88%E4%B9%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E6%8A%95%E6%B3%A8%E5%AE%98%E7%BD%91%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E5%A4%96%E5%9B%B4%E5%BC%80%E6%88%B7%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E5%9C%A8%E5%93%AA%E7%82%B9%E4%B9%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E5%9C%A8%E5%93%AA%E5%84%BF%E4%B9%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E7%90%83%E6%80%8E%E4%B9%88%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E4%B9%B0%E7%90%83%E6%8A%95%E6%B3%A8%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E5%93%AA%E4%B9%B0%E7%90%83%E5%A5%BD%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E5%93%AA%E8%83%BD%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E8%83%BD%E4%B9%B0%E7%90%83%E5%90%97%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E5%8E%BB%E5%93%AA%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E7%8E%B0%E9%87%91%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E4%BF%A1%E8%AA%89%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E5%9C%A8%E5%93%AA%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%8E%B0%E5%9C%A8%E5%8F%AF%E4%BB%A5%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8A%E7%9A%87%E5%86%A0%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8A%E4%B9%B0%E7%90%83%E5%85%85%E5%80%BC%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8A%E4%B9%B0%E7%90%83%E6%8A%95%E6%B3%A8%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8B%E5%93%AA%E9%87%8C%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8B%E5%93%AA%E8%83%BD%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8B%E6%80%8E%E4%B9%88%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E5%9C%A8%E5%93%AA%E4%B8%AA%E7%BD%91%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E5%9C%A8%E5%93%AA%E4%B9%B0%E7%90%83%E5%AE%89%E5%85%A8%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E6%80%8E%E4%B9%88%E6%8A%95%E6%B3%A8%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E6%AD%A3%E8%A7%84%E4%B9%B0%E7%90%83%E5%B9%B3%E5%8F%B0%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E5%9B%BD%E9%99%85%E4%B9%B0%E7%90%83%E7%BD%91%E5%9D%80%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E8%B5%9B%E4%BA%8B%E4%B9%B0%E7%90%83%E5%9C%B0%E5%9D%80%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E8%A7%84%E5%BE%8B%E6%8A%80%E5%B7%A7%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BD%91%E4%B8%8A%E4%B9%B0%E7%90%83%E8%BF%9E%E6%8E%A5%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E9%9D%A0%E8%B0%B1%E7%BD%91%E5%9D%80%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E5%AE%98%E6%96%B9%E7%BD%91%E7%AB%99%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E4%B9%B0%E7%90%83%E7%BD%91%E7%AB%99%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%BA%BF%E4%B8%8A%E4%B9%B0%E7%90%83%E7%BD%91%E7%AB%99%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99
http://jform2.baidu.com/s?wd=%E4%B8%96%E7%95%8C%E6%9D%AF%E7%AB%9E%E5%BD%A9%E6%8A%95%E6%B3%A8%E4%B9%B0%E7%90%83%E2%8B%98%E7%BD%91%3A%F0%9D%94%BE%F0%9D%95%84%EF%BD%A1%F0%9D%94%B8%F0%9D%94%BE%E2%8B%99

bash

复制编辑

rbd create myimage --size 10G --pool rbd_pool

映射 RBD 到客户端

bash

复制编辑

rbd map myimage --pool rbd_pool mkfs.ext4 /dev/rbd/rbd_pool/myimage mount /dev/rbd/rbd_pool/myimage /mnt


3.3 文件存储(CephFS)

部署 MDS

bash

复制编辑

ceph-deploy mds create mon1

挂载 CephFS

bash

复制编辑

mkdir /mnt/cephfs mount -t ceph mon1:/ /mnt/cephfs


4. Ceph 高级用法

4.1 纠删码存储池

提高存储效率:


bash

复制编辑

ceph osd pool create ec_pool 128 erasure


4.2 配置 Crush Map

自定义数据分布策略:


bash

复制编辑

ceph osd crush rule create-replicated myrule default host


4.3 Ceph Dashboard

启用 Dashboard

bash

复制编辑

ceph mgr module enable dashboard ceph dashboard create-self-signed-cert ceph dashboard set-login-credentials admin StrongPassword

访问:https://mon1:8443


5. Ceph 企业级实战

5.1 私有云存储解决方案

  • 为 OpenStack 提供后端块存储(Cinder)

  • 为 Kubernetes 提供持久卷(Rook-Ceph)


5.2 多数据中心同步

结合 Ceph Multisite 配置异地备份和容灾。


5.3 与大数据平台集成

  • HDFS-on-Ceph:将 Ceph 作为 Hadoop 的存储后端。


6. 性能优化与安全

  • 使用 SSD 作为 OSD 的日志设备

  • 调整 osd_mclock_scheduler 参数优化 IOPS

  • 启用加密存储和客户端访问认证

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值