快照原始对象缺失引起的osd崩溃

问题

unexpectedclone

修复pg的时候出现了 unexpected clone

可以看到这个对象后面有编号，这个编号有两种情况

  • 快照的对象(snapid)
  • 纠删对象的中间版本（对应是generation）
1
{"oid":"rbd_data.1.101a6b8b4567.00000000000000b3","key":"","snapid":26,"hash":3266086655,"max":0,"pool":2,"namespace":"","generation":251,"shard_id":2,"max":0}

这个地方需要根据自己的使用情况分析是哪种数据

通过在osd上面进行object list操作的时候发现
只有快照对象，没有原始对象

这个说明这个对象其实是做了删除的，但是快照对象没删除掉，遗留在这里了，所以出现了这个情况
这个情况正常情况是无法出现的，所以这里osd.12上面的磁盘做删除的时候没有删除成功,一般是出现在磁盘出现了异常的时候

问题复现方法

模拟一个删除了快照和对象
一个只删除了对象，快照数据都留着的，然后进行一次scrub的模拟 ，这个对象完全就没有了的

osd.0 是对象和快照都删除
osd.1 是删除对象，不删除快照对象 （需要加参数，内部正常不会出现这个情况）
原始对象是snapid -2
快照对象就是快照的编号id

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-1/ --pgid 1.75 --op list
Error getting attr on : 1.75_head,#1:ae000000::::head#, (61) No data available
["1.75",{"oid":"","key":"","snapid":-2,"hash":117,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000117","key":"","snapid":-2,"hash":4088145525,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":18,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":-2,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":16,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":18,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":18,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":-2,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]
[root@lab101 ceph]#
[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-0/ --pgid 1.75 --op list
Error getting attr on : 1.75_head,#1:ae000000::::head#, (61) No data available
["1.75",{"oid":"","key":"","snapid":-2,"hash":117,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000117","key":"","snapid":-2,"hash":4088145525,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":18,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":-2,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":16,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":18,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":18,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":-2,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]




[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-0/ --pgid 1.75 '{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}' remove
Clones are present, use removeall to delete everything
[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-0/ --pgid 1.75 '{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}' removeall
remove clone #1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:10#
remove clone #1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:12#
remove #1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:head#



[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-1/ --pgid 1.75 '{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}' remove
Clones are present, use removeall to delete everything
[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-1/ --pgid 1.75 '{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}' remove --force
WARNING: only removing head with clones present
remove #1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:head#


[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-0/ --pgid 1.75 --op list
Error getting attr on : 1.75_head,#1:ae000000::::head#, (61) No data available
["1.75",{"oid":"","key":"","snapid":-2,"hash":117,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000117","key":"","snapid":-2,"hash":4088145525,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":18,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":-2,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":18,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":-2,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]
[root@lab101 ceph]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-1/ --pgid 1.75 --op list
Error getting attr on : 1.75_head,#1:ae000000::::head#, (61) No data available
["1.75",{"oid":"","key":"","snapid":-2,"hash":117,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000117","key":"","snapid":-2,"hash":4088145525,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":18,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000014","key":"","snapid":-2,"hash":2027662965,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":16,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":18,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":18,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]
["1.75",{"oid":"rbd_data.606c9027f4e3.0000000000000057","key":"","snapid":-2,"hash":250437109,"max":0,"pool":1,"namespace":"","max":0}]

都启动了

1
[root@lab101 ceph]# rados -p rbd ls|grep rbd_data.606c9027f4e3.0000000000000064

找不到相关的对象，这个正常的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@lab101 ceph]# ceph -s
  cluster:
    id:     40edb456-3006-4189-aced-68abcdac0f72
    health: HEALTH_WARN
            mon is allowing insecure global_id reclaim
            noout flag(s) set
            1 pool(s) do not have an application enabled
            3 daemons have recently crashed
            OSD count 2 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum lab101 (age 44m)
    mgr: lab101(active, since 44m)
    osd: 2 osds: 2 up (since 61s), 2 in (since 6d)
         flags noout

  data:
    pools:   1 pools, 128 pgs
    objects: 435 objects, 1.6 GiB
    usage:   3.2 GiB used, 442 GiB / 445 GiB avail
    pgs:     128 active+clean

也正常的,我们来一次scrub

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@lab101 ceph]# ceph pg deep-scrub 1.75
instructing pg 1.75 on osd.1 to deep-scrub


        flags noout

  data:
    pools:   1 pools, 128 pgs
    objects: 435 objects, 1.6 GiB
    usage:   3.2 GiB used, 442 GiB / 445 GiB avail
    pgs:     127 active+clean
             1   active+clean+inconsistent

1
2
3
 [root@lab101 ceph]# rados list-inconsistent-obj 1.75
{"epoch":704,"inconsistents":[{"object":{"name":"rbd_data.606c9027f4e3.0000000000000064","nspace":"","locator":"","snap":16,"version":28},"errors":[],"union_shard_errors":["missing"],"selected_object_info":{"oid":{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":16,"hash":2791802613,"max":0,"pool":1,"namespace":""},"version":"688'29","prior_version":"535'28","last_reqid":"client.24688.0:849","user_version":28,"size":131072,"mtime":"2024-11-01T00:26:29.890995+0800","local_mtime":"2024-11-01T00:26:29.908788+0800","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x3fdefe12","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}},"shards":[{"osd":0,"primary":false,"errors":["missing"]},{"osd":1,"primary":true,"errors":[],"size":131072,"omap_digest":"0xffffffff","data_digest":"0x3fdefe12"}]},{"object":{"name":"rbd_data.606c9027f4e3.0000000000000064","nspace":"","locator":"","snap":18,"version":31},"errors":[],"union_shard_errors":["missing"],"selected_object_info":{"oid":{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":18,"hash":2791802613,"max":0,"pool":1,"namespace":""},"version":"697'43","prior_version":"688'31","last_reqid":"client.24688.0:1028","user_version":31,"size":4194304,"mtime":"2024-11-01T00:37:32.918239+0800","local_mtime":"2024-11-01T00:37:32.922857+0800","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x2dea776f","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}},"shards":[{"osd":0,"primary":false,"errors":["missing"]},{"osd":1,"primary":true,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x2dea776f"}]}]}[root@lab101 ceph]#

1
2
[root@lab101 ceph]# ceph pg repair 1.75
instructing pg 1.75 on osd.1 to repair

尝试修复

1
2
3
4
5
6
7
8
9
10
11
12
13
14
data:
    pools:   1 pools, 128 pgs
    objects: 432 objects, 1.6 GiB
    usage:   3.2 GiB used, 442 GiB / 445 GiB avail
    pgs:     127 active+clean
             1   active+clean+inconsistent+failed_repair



  2024-11-01T00:46:21.640+0800 7f7ae0d5a700 -1 log_channel(cluster) log [ERR] : 1.75 shard 0 soid 1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:12 : data_digest 0x43d61c5d != data_digest 0x2dea776f from shard 1, data_digest 0x43d61c5d != data_digest 0x2dea776f from auth oi 1:af61e
665:::rbd_data.606c9027f4e3.0000000000000064:12(697'43 client.24688.0:1028 dirty|data_digest s 4194304 uv 31 dd 2dea776f alloc_hint [0 0 0])
2024-11-01T00:46:21.640+0800 7f7ae0d5a700 -1 log_channel(cluster) log [ERR] : deep-scrub 1.75 1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:12 : is an unexpected clone
2024-11-01T00:46:21.640+0800 7f7ae0d5a700 -1 log_channel(cluster) log [ERR] : deep-scrub 1.75 1:af61e665:::rbd_data.606c9027f4e3.0000000000000064:10 : is an unexpected clone

查看日志 ，可以看到一模一样了

问题根因

情况一：

集群内需要删除数据的时候，在osd上面无法操作删除，同时也看到了日志有read error，这种读都读不出来的情况，说明磁盘出了问题，同时也有这个删除无法删除的情况，就是磁盘上面的数据已经不能正常的处理请求了，数据混乱状态了

情况二：

原始对象丢失了，只剩下快照对象，从现场的情况看，快照499还在，对象还在，说明并没有删除快照的操作，然后这个原始对象没了，那么丢失的原始对象的可能性比较大
这个丢失什么情况造成的就没法判断了

测试验证了一个情况，就是只删除主本的原始对象，这种情况是可以修复的，那么现场的情况，应该是修复也无法修复了
说明还是应该有删除的操作，或者就是这个对象做快照的时候应该就删除了，但是这个异常节点并没有删除，做快照的时候就出现了快照对象，然后原始对象又没有了，就是出现了这种主本没有的情况了

解决办法

通过这个删除相关的对象，进行启动

1
ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-1/ --pgid 1.75 '{"oid":"rbd_data.606c9027f4e3.0000000000000064","key":"","snapid":-2,"hash":2791802613,"max":0,"pool":1,"namespace":"","max":0}' remove

如果环境多个osd 出现down了。也需要按这个删除对象，避免osd 无法启动了
这种有快照对象没原始对象的就是异常对象的 ,删除的时候需要找到所有节点的这种情况进行pg内的快照对象的删除，否则无法启动osd

总结

这个属于ceph里面中间垃圾数据引起的异常的情况，这个一般处理方法就是要么删除垃圾数据，要么有的情况无法删除，就构造对象后再删除,避免无法启动osd的情况造成pg的异常