笔记:Ceph: A Scalable, High-Performance Distributed File System

本文介绍了Ceph存储系统的架构及关键技术。Ceph是一款提供高性能、可靠性和可扩展性的分布式文件系统,支持对象存储、块存储及文件系统。通过CRUSH算法优化数据分布,并采用智能设备分散数据复制、故障检测与恢复等功能。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

关于Ceph的名篇。Ceph是现在很火的一个存储系统,不同于HDSF主要是面向大数据应用,Ceph是立志要做一个通用的存储解决方案,要同时很好的支持对象存储(Object Storage),块存储(Block Storage)以及文件系统(File System) 。现在很多Openstack私有云的存储都是基于Ceph的。Ceph就是基于这篇论文做得。

摘要
很明确的指出了Ceph的使命:
We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability.
以及关键方法和技术:
Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs).
We leverage device intelligence by distributing data replication, failure detection and recovery to semi-autonomous OSDs running a specialized local object file system.
A dynamic distributed metadata cluster provides extremely efficient metadata management and seamlessly adapts to a wide range of general purpose and scientific computing file system workloads.
然后就是性能
Performance measurements under a variety of workloads show that Ceph has excellent I/O performance and scalable metadata management, supportingmore than 250,000metadata operations per second.

介绍:
先把NFS和传统OSD的问题说了一下。
然后介绍Ceph:
We present Ceph, a distributed file system that provides excellent performance and reliability while promising unparalleled scalability.
这句是一个关键:Our architecture is based on the assumption that systems at the petabyte scale are inherently dynamic: large systems are inevitably built incrementally, node failures are the norm rather than the exception, and the quality and character of workloads are constantly shifting over time.
Ceph的架构如下:

系统介绍:
Ceph分3部分:
the client, each instance of which exposes a near-POSIX file system interface to a host or process;
a cluster of OSDs, which collectively stores all data and metadata;
A metadata server cluster, which manages the namespace (file names and directories) while coordinating security, consistency and coherence (see Figure 1).
如下图所示:
screenshot

主要做法:
Decoupled Data and Metadata
Dynamic Distributed Metadata Management
Reliable Autonomic Distributed Object Storage

后面几章是对每部分具体实现的介绍,没有什么太高深的公式和理论,大家一般都能看明白,挺有意思的。
原文链接:
http://www.ece.eng.wayne.edu/~sjiang/ECE7650-winter-15/topic5B-S.pdf
如果下不了可以去百度学术上再搜一下。

2025-03-29 10:14:00.654114 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2025-03-29 10:14:00.660132 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2025-03-29 10:14:00.660179 I | cephclient: getting or creating ceph auth key "mgr.b" 2025-03-29 10:14:00.918206 I | op-mgr: deployment for mgr rook-ceph-mgr-b already exists. updating if needed 2025-03-29 10:14:00.922071 I | op-k8sutil: deployment "rook-ceph-mgr-b" did not change, nothing to update 2025-03-29 10:14:02.012907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph mgr: failed to enable mgr services: failed to enable service monitor: service monitor could not be enabled: failed to retrieve servicemonitor. servicemonitors.monitoring.coreos.com "rook-ceph-mgr" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph" 2025-03-29 10:14:02.052359 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2025-03-29 10:14:02.048230153 +0000 UTC m=+9812.323876586 LastTransitionTime:2025-03-29 08:40:15 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2025-03-29 10:25:15.419892 E | ceph-nodedaemon-controller: node reconcile failed: failed to enable service monitor: service monitor could not be enabled: failed to retrieve servicemonitor. servicemonitors.monitoring.coreos.com "rook-ceph-exporter" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph" 2025-03-29 10:25:16.017269 E | ceph-nodedaemon-controller: node reconcile failed: failed to enable service monitor: service monitor could not be enabled: failed to retrieve servicemonitor. servicemonitors.monitoring.coreos.com "rook-ceph-exporter" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph" 2025-03-29 10:25:18.622406 E | ceph-nodedaemon-controller: node reconcile failed: failed to enable service monitor: service monitor could not be enabled: failed to retrieve servicemonitor. servicemonitors.monitoring.coreos.com "rook-ceph-exporter" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph"
03-30
<think>好的,我现在需要帮助用户分析他们提供的日志信息。首先,用户展示的日志涉及Rook-Ceph在Kubernetes集群中的部署问题,特别是关于ServiceMonitor资源的访问权限问题。 首先,我要理解日志中的关键错误信息。第一条错误是用户`system:serviceaccount:rook-ceph:rook-ceph-system`无法获取`servicemonitors.monitoring.coreos.com`资源。这表明服务账号缺少必要的RBAC权限。接下来的错误提到了`servicemonitors.monitoring.coreos.com "rook-ceph-mgr"`和`"rook-ceph-exporter"`无法被访问,同样是因为权限问题。另外,还有一个关于更新集群状态时的资源版本冲突,这可能是因为并发修改导致的,但核心问题还是权限问题。 接下来,我需要考虑解决方案。通常,Kubernetes中的权限问题需要通过Role和RoleBinding来解决。用户需要确保服务账号`rook-ceph-system`在命名空间`rook-ceph`中有权限访问`servicemonitors`资源。这涉及到创建或更新对应的Role和RoleBinding,赋予`get`、`create`、`update`等权限。 另外,用户可能需要检查Rook的配置,特别是CephCluster CRD中的`spec.monitoring`部分是否启用了ServiceMonitor,并且相关的Prometheus监控配置是否正确。如果配置正确,但权限不足,仍然会导致问题。 还要考虑Kubernetes集群中是否安装了Prometheus Operator,因为ServiceMonitor是Prometheus Operator提供的CRD。如果未安装,那么即使有权限也无法创建该资源,但根据错误信息,问题更可能出在权限而非资源不存在。 总结处理步骤应该是:检查并添加RBAC权限,验证配置,然后重新部署或更新Rook-Ceph集群。此外,资源版本冲突可能需要重试操作,但首要解决权限问题。 需要确保回答结构清晰,分步骤说明,并且建议用户如何实施每个步骤,比如应用RBAC配置文件,检查现有配置等。同时,提醒用户可能需要的后续操作,如重启Operator或验证监控配置是否正确。</think>以下是日志问题的逐步分析及解决方案: --- ### **问题定位** 日志核心错误是 **RBAC权限不足**,具体表现为: ``` servicemonitors.monitoring.coreos.com "rook-ceph-mgr" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" ``` 服务账号 `rook-ceph-system` 缺少操作 `servicemonitors` 资源的权限。 --- ### **根本原因** 1. **缺少RBAC配置** Rook-Ceph的Operator需要权限管理 `servicemonitors.monitoring.coreos.com` 资源(属于Prometheus监控体系),但默认安装可能未自动授予相关权限。 2. **Prometheus监控集成未正确配置** 如果启用了Ceph集群的监控功能(例如通过 `spec.monitoring.enabled=true`),但未正确配置ServiceAccount权限,则触发此错误。 --- ### **解决方案** #### **步骤1:添加RBAC权限** 为 `rook-ceph-system` 服务账号绑定操作 `servicemonitors` 的权限: ```yaml # service-monitor-rbac.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rook-ceph-monitoring namespace: rook-ceph rules: - apiGroups: ["monitoring.coreos.com"] resources: ["servicemonitors"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rook-ceph-system-monitoring namespace: rook-ceph roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rook-ceph-monitoring subjects: - kind: ServiceAccount name: rook-ceph-system namespace: rook-ceph ``` 应用配置: ```bash kubectl apply -f service-monitor-rbac.yaml ``` #### **步骤2:验证CephCluster配置** 检查CephCluster自定义资源中是否启用了监控: ```yaml apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: monitoring: enabled: true # 确保Prometheus Operator已安装 rulesNamespace: rook-ceph ``` #### **步骤3:重启Rook-Ceph Operator** 更新RBAC后,重启Operator以加载新权限: ```bash kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator ``` #### **步骤4:检查Prometheus Operator** 确保集群中已部署 **Prometheus Operator**,因为 `ServiceMonitor` 是其管理的CRD: ```bash kubectl get pods -n monitoring | grep prometheus-operator ``` --- ### **附加说明** - **资源版本冲突警告** 日志中 `Operation cannot be fulfilled on cephclusters.ceph.rook.io...` 是典型的并发修改冲突,通常由Operator自动重试解决,无需手动干预。 - **权限生效延迟** RBAC更新后可能需要1~2分钟生效,若问题未解决,请检查RoleBinding是否关联到正确的ServiceAccount。 --- ### **验证操作** 观察Operator日志,确认权限错误消失: ```bash kubectl -n rook-ceph logs -l app=rook-ceph-operator --tail=100 ```
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值