Ceph删除/添加故障OSD（ceph-deploy）

最新推荐文章于 2025-05-07 17:56:24 发布

bobpen

最新推荐文章于 2025-05-07 17:56:24 发布

阅读量1.6w

点赞数 5

CC 4.0 BY-SA版权

分类专栏： linux 存储云计算 ceph

本文链接：https://blog.youkuaiyun.com/bobpen/article/details/40424069

linux 同时被 3 个专栏收录

53 篇文章

订阅专栏

ceph

15 篇文章

订阅专栏

存储

11 篇文章

订阅专栏

本文介绍了一种解决Ceph集群中OSD(osd.5)因硬盘故障导致无法启动的方法。通过删除故障OSD并重新准备及激活新的OSD，最终使Ceph集群恢复正常运行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天osd.5起不来，发现是硬盘的问题

[root@ceph-osd-1 ceph-cluster]# ceph osd tree
# id    weight  type name       up/down reweight
-1      11.63   root default
-2      5.26            host ceph-osd-1
0       5.26                    osd.0   up      1
-3      2.73            host ceph-osd-2
1       0.91                    osd.1   up      1
2       0.91                    osd.2   up      1
3       0.91                    osd.3   up      1
-4      3.64            host ceph-osd-3
4       1.82                    osd.4   up      1
5       0.91                    osd.5   down    1
6       0.91                    osd.6   up      1

更换磁盘后，将osd.5删除，然后通过ceph-deploy重新添加

[root@ceph-osd-1 ceph-cluster]# ceph osd rm 5
[root@ceph-osd-1 ceph-cluster]# ceph-deploy osd prepare 10.10.200.165：/osd5
[root@ceph-osd-1 ceph-cluster]# ceph-deploy osd activate 10.10.200.165：/osd5

在执行activate的时候，提示以下错误

[10.10.200.165][WARNIN] Error EINVAL: entity osd.5 exists but key does not match
[10.10.200.165][WARNIN] Traceback (most recent call last):
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 2591, in <module>
[10.10.200.165][WARNIN]     main()
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 2569, in main
[10.10.200.165][WARNIN]     args.func(args)
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 1929, in main_activate
[10.10.200.165][WARNIN]     init=args.mark_init,
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 1761, in activate_dir
[10.10.200.165][WARNIN]     (osd_id, cluster) = activate(path, activate_key_template, init)
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 1897, in activate
[10.10.200.165][WARNIN]     keyring=keyring,
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 1520, in auth_key
[10.10.200.165][WARNIN]     'mon', 'allow profile osd',
[10.10.200.165][WARNIN]   File "/usr/sbin/ceph-disk", line 304, in command_check_call
[10.10.200.165][WARNIN]     return subprocess.check_call(arguments)
[10.10.200.165][WARNIN]   File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
[10.10.200.165][WARNIN]     raise CalledProcessError(retcode, cmd)
[10.10.200.165][WARNIN] subprocess.CalledProcessError: Command '['/usr/bin/ceph', '--cluster', 'ceph', '--name', 'client.bootstrap-osd', '--keyring', '/var/lib/ceph/bootstrap-osd/ceph.keyring', 'auth', 'add', 'osd.5', '-i', '/osd5/keyring', 'osd', 'allow *', 'mon', 'allow profile osd']' returned non-zero exit status 22
[10.10.200.165][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk -v activate --mark-init sysvinit --mount /osd5

解决办法：

在删除故障OSD的时候，执行以下操作

[root@ceph-osd-1 ceph-cluster]# ceph auth del osd.5
updated
[root@ceph-osd-1 ceph-cluster]# ceph osd rm 5
removed osd.5

而后在执行

[root@ceph-osd-1 ceph-cluster]# ceph-deploy prepare 10.10.200.165：/osd5
[root@ceph-osd-1 ceph-cluster]# ceph-deploy activate 10.10.200.165：/osd5

查看osd状态

[root@ceph-osd-1 ceph-cluster]# ceph osd tree
# id    weight  type name       up/down reweight
-1      11.63   root default
-2      5.26            host ceph-osd-1
0       5.26                    osd.0   up      1
-3      2.73            host ceph-osd-2
1       0.91                    osd.1   up      1
2       0.91                    osd.2   up      1
3       0.91                    osd.3   up      1
-4      3.64            host ceph-osd-3
4       1.82                    osd.4   up      1
5       0.91                    osd.5   up      1
6       0.91                    osd.6   up      1

查看ceph状态，已经开始recovery操作

[root@ceph-osd-1 ceph-cluster]# ceph -s
    cluster 374b8b6b-8b47-4d14-af47-f383d42af2ba
     health HEALTH_WARN 7 pgs backfill; 8 pgs backfilling; 1 pgs degraded; 6 pgs recovering; 34 pgs recovery_wait; 55 pgs stuck unclean; recovery 15753/191906 objects degraded (8.209%)
     monmap e1: 1 mons at {ceph-osd-1=10.10.200.163:6789/0}, election epoch 1, quorum 0 ceph-osd-1
     osdmap e371: 7 osds: 7 up, 7 in
      pgmap v52585: 704 pgs, 7 pools, 251 GB data, 61957 objects
            800 GB used, 11105 GB / 11905 GB avail
            15753/191906 objects degraded (8.209%)
                  34 active+recovery_wait
                 649 active+clean
                   7 active+remapped+wait_backfill
                   1 active+degraded+remapped+backfilling
                   6 active+recovering
                   7 active+remapped+backfilling
recovery io 8150 kB/s, 1 objects/s
  client io 2037 B/s wr, 0 op/s