纠删码中间对象属性丢失引起osd的崩溃

背景

迁移的时候出现osd的崩溃，然后进行pg的备份的时候出现了无法获取属性的情况，本篇记录问题和解决的方法

问题

1
2
3
4
5
6
7
8
9
10
11
Error getting attr on : 2.7s2_head,2#2:f7d032a7:::rbd_data.1.101a6b8b4567.00000000000000a1:head#f6, (61) No data available
Error getting attr on : 2.7s2_head,2#2:fecb9c0c:::rbd_data.1.101a6b8b4567.0000000000000089:head#46, (61) No data available
Error getting attr on : 2.7s2_head,2#2:fecb9c0c:::rbd_data.1.101a6b8b4567.0000000000000089:head#48, (61) No data available
Error getting attr on : 2.7s2_head,2#2:fecb9c0c:::rbd_data.1.101a6b8b4567.0000000000000089:head#ed, (61) No data available
["2.7s2",{"oid":"rbd_data.1.101a6b8b4567.0000000000000089","key":"","snapid":-2,"hash":809096063,"max":0,"pool":2,"namespace":"","generation":120,"shard_id":2,"max":0}]
["2.7s2",{"oid":"rbd_data.1.101a6b8b4567.0000000000000089","key":"","snapid":-2,"hash":809096063,"max":0,"pool":2,"namespace":"","generation":157,"shard_id":2,"max":0}]
["2.7s2",{"oid":"rbd_data.1.101a6b8b4567.0000000000000089","key":"","snapid":-2,"hash":809096063,"max":0,"pool":2,"namespace":"","generation":194,"shard_id":2,"max":0}]
["2.7s2",{"oid":"rbd_data.1.101a6b8b4567.0000000000000089","key":"","snapid":-2,"hash":809096063,"max":0,"pool":2,"namespace":"","generation":287,"shard_id":2,"max":0}]
["2.7s2",{"oid":"rbd_data.1.101a6b8b4567.0000000000000089","key":"","snapid":-2,"hash":809096063,"max":0,"pool":2,"namespace":"","generation":326,"shard_id":2,"max":0}]
["2.7s2",{"oid":"rbd_data.1.101a6b8b4567.0000000000000089","key":"","snapid":-2,"hash":809096063,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]
[root@lab103 mnt]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-2 --pgid 2.7s2 --op list|grep rbd_data.1.101a6b8b4567.0000000000000089

做list或者export的时候会报错

1
2
3
4
5
[root@lab103 mnt]# ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-2 --pgid 2.7s2 --op export --file zp
Exporting 2.7s2
Read 2#2:e0384c0e:::rbd_data.1.101a6b8b4567.0000000000000072:head#3e
getattr failure object_info -61
export_files error -61

然后这个情况下在做backfill的时候osd 就崩溃了

问题原因

存储系统的盘出现了故障，造成了一些属性没有写上去，有的对象没删除，就出现这种中间状态了，这种一般是阵列卡引起或者磁盘问题，出现后，就可能出现卡pg的状态了，必须修复才能恢复环境

出现问题的时候，开始理解错了，上面的2.7s2_head,2#2:fecb9c0c:::rbd_data.1.101a6b8b4567.0000000000000089:head#ed这个结尾的编号跟快照的编号一样的
误认为这个地方是快照的，这个地方实际上是纠删码的覆盖写过程中的中间对象的，正常情况下会自动删除了，但是没删除的时候，就出现问题了

模拟出这个问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
vim ./ceph-xxxx/src/osd/PGBackend.cc
void PGBackend::rollback_extents(
  version_t gen,
  const vector<pair<uint64_t, uint64_t> > &extents,
  const hobject_t &hoid,
  ObjectStore::Transaction *t) {
  auto shard = get_parent()->whoami_shard().shard;
  for (auto &&extent: extents) {
    t->clone_range(
      coll,
      ghobject_t(hoid, gen, shard),
      ghobject_t(hoid, ghobject_t::NO_GEN, shard),
      extent.first,
      extent.second,
      extent.first);
  }
//  t->remove(
//    coll,
//    ghobject_t(hoid, gen, shard));
}

void PGBackend::trim_rollback_object(
  const hobject_t &hoid,
  version_t old_version,
  ObjectStore::Transaction *t) {
  assert(!hoid.is_temp());
//  t->remove(
//    coll, ghobject_t(hoid, old_version, get_parent()->whoami_shard().shard));
}

触发故障

屏蔽掉两个删除中间对象的地方
然后配置一个纠删码的集群，然后对着rbd进行覆盖写的操作，然后list对象

1
2
ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-2 --pgid 2.7s2 '{"oid":"rbd_data.1.101a6b8b4567.00000000000000b3","key":"","snapid":-2,"hash":3266086655,"max":0,"pool":2,"namespace":"","generation":127,"shard_id":2,"max":0}' rm-attr snapset
ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-2 --pgid 2.7s2 '{"oid":"rbd_data.1.101a6b8b4567.00000000000000b3","key":"","snapid":-2,"hash":3266086655,"max":0,"pool":2,"namespace":"","generation":127,"shard_id":2,"max":0}' rm-attr _

执行完这个以后，就可以发现，对象无法删除了，模拟出了问题的现象

问题解决方式

问题比较清晰了，就是中间对象的扩展属性丢失了，我们需要处理这种情况，通过上面的模拟，我们找到了对象的命名规则
这个地方的snapid跟原始对象一样，就是generation这个地方是编号的16进制转10进制

设置snapset属性

从其它正常的对象上面获取到snapset的属性

然后通过

1
ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-2 --pgid 2.7s2 '{"oid":"rbd_data.1.101a6b8b4567.00000000000000b3","key":"","snapid":-2,"hash":3266086655,"max":0,"pool":2,"namespace":"","generation":127,"shard_id":2,"max":0}' set-attr napset < snapset

设置这个属性后才能删除这个对象,attr _这个属性可以不设置，也可以删除对象，缺这个snapset的属性是不能删除的

1
ceph-objectstore-tool --data /var/lib/ceph/osd/ceph-2 --pgid 2.7s2 '{"oid":"rbd_data.1.101a6b8b4567.00000000000000b3","key":"","snapid":-2,"hash":3266086655,"max":0,"pool":2,"namespace":"","generation":127,"shard_id":2,"max":0}' remove

同样的方法把其它的垃圾对象处理掉

总结

问题就是ec的情况下出现了中间对象的扩展属性丢失的情况，扩展属性丢失无法删除对象，无法export，backfill也崩溃，多个问题
处理问题的思路就是把对象给构造回去然后再删除

这个地方跟之前的快照对象存在，原始对象缺失造成的崩溃有点类似，但是这个地方难点是需要知道中间对象命名规则

这个问题一般出现在硬件出问题或者掉电情况下，场景比较限定，使用纠删场景，并且使用rbd，还触发了覆盖写，正好碰上了磁盘数据没法删除产生垃圾文件的情况