容器部署ceph如何使用ceph-objectstore-tool修复

背景

ceph在后期进行了容器化部署,主推这个模式，容器是部署容易，出问题的时候就黑盒了，特别对于不熟悉容器内部的情况下
很容易陷入一种循环，不启动容器看不到故障，启动容器又拉不起来
容器是在系统之上的，只要找到相关的对应关系都好处理

搭建一个容器的集群

安装部署相关

yum install cephadm
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

添加源

[root@lab201 ~]# cephadm add-repo --version 15.2.17

这个会添加
/etc/yum.repos.d/epel.repo
和
/etc/yum.repos.d/ceph.repo

可以修改里面的改成国内源即可

systemctl start docker

下载指定版本的镜像

docker pull quay.io/ceph/ceph:v15.2.17

创建mon

指定了镜像和指定了版本，以及指定了mon的ip

 cephadm --image quay.io/ceph/ceph:v15.2.17 bootstrap  --mon-ip 192.168.0.201

 Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:

         URL: https://lab201:8443/
        User: admin
    Password: dx56afg2dp

You can access the Ceph CLI with:

    sudo /usr/sbin/cephadm shell --fsid 1c74efac-63a5-11f0-89e0-525400a61fc0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

    ceph telemetry on

For more information see:

    https://docs.ceph.com/docs/master/mgr/telemetry/

Bootstrap complete.

查看状态
可以看到启动了容器

[root@lab201 ~]# docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED              STATUS              PORTS     NAMES
820f6c901c94   quay.io/ceph/ceph:v15.2.17                "/usr/bin/ceph-crash…"   38 seconds ago       Up 37 seconds                 ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-crash.lab201
6c9590aa6486   quay.io/prometheus/alertmanager:v0.20.0   "/bin/alertmanager -…"   39 seconds ago       Up 38 seconds                 ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-alertmanager.lab201
4a83d225a254   quay.io/ceph/ceph:v15.2.17                "/usr/bin/ceph-mgr -…"   About a minute ago   Up About a minute             ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-mgr.lab201.nzkrry
cc8444b630fa   quay.io/ceph/ceph:v15.2.17                "/usr/bin/ceph-mon -…"   About a minute ago   Up About a minute             ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-mon.lab201

这个就可以进去执行ceph命令了

/usr/sbin/cephadm shell --fsid 1c74efac-63a5-11f0-89e0-525400a61fc0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

因为容器共用了配置文件，所以系统上如果安装了ceph的包，那么也可以直接使用命令

[root@lab201 ~]# ceph -s
  cluster:
    id:     1c74efac-63a5-11f0-89e0-525400a61fc0
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum lab201 (age 2m)
    mgr: lab201.nzkrry(active, since 114s)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

添加osd

[root@lab201 ~]# ceph orch daemon add osd lab201:/dev/vdb
Created osd(s) 0 on host 'lab201'

放一点数据

ceph osd pool create rbd 8
ceph osd pool set rbd size 1
rados -p rbd put anaconda-ks.cfg anaconda-ks.cfg

查看状态

[root@lab201 ~]# ceph -s
  cluster:
    id:     1c74efac-63a5-11f0-89e0-525400a61fc0
    health: HEALTH_WARN
            1 pool(s) do not have an application enabled
            2 pool(s) have no replicas configured
            OSD count 1 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum lab201 (age 13m)
    mgr: lab201.nzkrry(active, since 13m)
    osd: 1 osds: 1 up (since 12m), 1 in (since 12m)

  data:
    pools:   2 pools, 33 pgs
    objects: 1 objects, 1.6 KiB
    usage:   1.0 GiB used, 79 GiB / 80 GiB avail
    pgs:     33 active+clean

模拟异常

集群搭建好了后我们尝试停止一个osd

[root@lab201 ~]# systemctl status ceph-1c74efac-63a5-11f0-89e0-525400a61fc0@osd.0.service
● ceph-1c74efac-63a5-11f0-89e0-525400a61fc0@osd.0.service - Ceph osd.0 for 1c74efac-63a5-11f0-89e0-525400a61fc0
   Loaded: loaded (/etc/systemd/system/ceph-1c74efac-63a5-11f0-89e0-525400a61fc0@.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-07-18 15:33:54 CST; 26s ago
  Process: 8549 ExecStopPost=/bin/bash /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/%i/unit.poststop (code=exited, status=0/SUCCESS)
  Process: 8543 ExecStop=/bin/docker stop ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-%i (code=exited, status=1/FAILURE)
 Main PID: 9326 (bash)
   CGroup: /system.slice/system-ceph\x2d1c74efac\x2d63a5\x2d11f0\x2d89e0\x2d525400a61fc0.slice/ceph-1c74efac-63a5-11f0-89e0-525400a61fc0@osd.0.service
           ├─9326 /bin/bash /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/unit.run
           └─9539 /bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --name ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-osd.0 -e CONTAINER_IMAGE=quay.i...

我们模拟这个osd停止了

systemctl stop ceph-1c74efac-63a5-11f0-89e0-525400a61fc0@osd.0.service

实践

我们在部署容器的时候，系统上面一般不会准备一个ceph环境，但是这个其实可以提前准备一个，方便维护，容器出故障的时候好处理
那么本篇我们基于宿主机有ceph环境和没ceph环境两种情况下进行实践

纯容器环境

osd的启动脚本

[root@lab201 ~]# cat /etc/systemd/system/ceph-1c74efac-63a5-11f0-89e0-525400a61fc0@.service
# generated by cephadm
[Unit]
Description=Ceph %i for 1c74efac-63a5-11f0-89e0-525400a61fc0

# According to:
#   http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
# these can be removed once ceph-mon will dynamically change network
# configuration.
After=network-online.target local-fs.target time-sync.target docker.service
Wants=network-online.target local-fs.target time-sync.target
Requires=docker.service


PartOf=ceph-1c74efac-63a5-11f0-89e0-525400a61fc0.target
Before=ceph-1c74efac-63a5-11f0-89e0-525400a61fc0.target

[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
EnvironmentFile=-/etc/environment
ExecStart=/bin/bash /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/%i/unit.run
ExecStop=-/bin/docker stop ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-%i
ExecStopPost=-/bin/bash /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/%i/unit.poststop

就是启动ceph osd容器的过程

再看这个脚本

[root@lab201 ~]# cat /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/unit.run
set -e
/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0
# LVM OSDs use ceph-volume lvm activate
! /bin/docker rm -f ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-osd.0-activate 2> /dev/null
/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --name ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-osd.0-activate -e CONTAINER_IMAGE=quay.io/ceph/ceph:v15.2.17 -e NODE_NAME=lab201 -v /var/run/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0:/var/run/ceph:z -v /var/log/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0:/var/log/ceph:z -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0:/var/lib/ceph/osd/ceph-0:z -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs quay.io/ceph/ceph:v15.2.17 lvm activate 0 77520224-b55e-4c71-bf94-605707caaa95 --no-systemd
# osd.0
! /bin/docker rm -f ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-osd.0 2> /dev/null
/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-osd --privileged --group-add=disk --name ceph-1c74efac-63a5-11f0-89e0-525400a61fc0-osd.0 -e CONTAINER_IMAGE=quay.io/ceph/ceph:v15.2.17 -e NODE_NAME=lab201 -v /var/run/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0:/var/run/ceph:z -v /var/log/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0:/var/log/ceph:z -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0:/var/lib/ceph/osd/ceph-0:z -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs quay.io/ceph/ceph:v15.2.17 -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug '

上面实际有两步，一个是lvm的activate，然后才是start

可以看到，实际上是每次启动的时候去绑定的磁盘和容器的关系的，这里我们找到这个osd对应的物理盘

这个lvm实际在系统盘上面有的，这个地方有个问题是，ceph的osd如果异常，那么osd就无法启动的，无法启动就无法去容器内操作，成了循环了

进入容器但是不启动osd
因为ceph的容器每次停止都是销毁的，我们需要一个临时容器与数据交互
我们参考这个 /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/unit.run 启动一个容器

docker run -it --rm --privileged \
  --name ceph-dev \
  -e NODE_NAME=lab201 \
  -v /dev:/dev \
  -v /run/udev:/run/udev \
  -v /sys:/sys \
  -v /run/lvm:/run/lvm \
  -v /run/lock/lvm:/run/lock/lvm \
  -v /:/rootfs \
  -v /var/run/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0:/var/run/ceph:z \
  -v /var/log/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0:/var/log/ceph:z \
  -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/crash:/var/lib/ceph/crash:z \
  -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0:/var/lib/ceph/osd/ceph-0:z \
  -v /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/config:/etc/ceph/ceph.conf:z \
  quay.io/ceph/ceph:v15.2.17 \
  /bin/bash

然后进去操作

[root@1fda8a9031c7 /]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-0/ --op list
["2.4",{"oid":"anaconda-ks.cfg","key":"","snapid":-2,"hash":3896821636,"max":0,"pool":2,"namespace":"","max":0}]

这个就可以在容器内操作了，命令在容器内都自带了

宿主机上面操作

安装好同版本的ceph后，直接在系统上面操作即可

[root@lab201 ~]# ceph-objectstore-tool --data  /var/lib/ceph/1c74efac-63a5-11f0-89e0-525400a61fc0/osd.0/ --op list
["2.4",{"oid":"anaconda-ks.cfg","key":"","snapid":-2,"hash":3896821636,"max":0,"pool":2,"namespace":"","max":0}]

这个就注意下，还是在/var/lib/ceph下面，多了一个字符串id，再下一层才是数据

总结

本篇记录了两种方式的ceph的修复方式，上面的操作只是启用了修复，具体的怎么修复就不同情况不同处理了