Ceph 状态报警告 pool rbd has many more objects per pg than average (too few pgs?)

定位问题

1
2
3
4
5
6
7
8
9
10
11
[root@lab8106 ~]# ceph -s
    cluster fa7ec1a1-662a-4ba3-b478-7cb570482b62
     health HEALTH_WARN
            pool rbd has many more objects per pg than average (too few pgs?)
     monmap e1: 1 mons at {lab8106=192.168.8.106:6789/0}
            election epoch 30, quorum 0 lab8106
     osdmap e157: 2 osds: 2 up, 2 in
            flags sortbitwise
      pgmap v1023: 417 pgs, 13 pools, 18519 MB data, 15920 objects
            18668 MB used, 538 GB / 556 GB avail
                 417 active+clean

集群出现了这个警告，pool rbd has many more objects per pg than average (too few pgs?) 这个警告在hammer版本里面的提示是 pool rbd has too few pgs

这个地方查看集群详细信息：

1
2
3
[root@lab8106 ~]# ceph health detail
HEALTH_WARN pool rbd has many more objects per pg than average (too few pgs?); mon.lab8106 low disk space
pool rbd objects per pg (1912) is more than 50.3158 times cluster average (38)

看下集群的pool的对象状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@lab8106 ~]# ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED 
    556G      538G       18668M          3.28 
POOLS:
    NAME       ID     USED       %USED     MAX AVAIL     OBJECTS 
    rbd        6      16071M      2.82          536G       15296 
    pool1      7        204M      0.04          536G          52 
    pool2      8        184M      0.03          536G          47 
    pool3      9        188M      0.03          536G          48 
    pool4      10       192M      0.03          536G          49 
    pool5      11       204M      0.04          536G          52 
    pool6      12       148M      0.03          536G          38 
    pool7      13       184M      0.03          536G          47 
    pool8      14       200M      0.04          536G          51 
    pool9      15       200M      0.04          536G          51 
    pool10     16       248M      0.04          536G          63 
    pool11     17       232M      0.04          536G          59 
    pool12     18       264M      0.05          536G          67

查看存储池的pg个数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@lab8106 ~]# ceph osd dump|grep pool
pool 6 'rbd' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 132 flags hashpspool stripe_width 0
pool 7 'pool1' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 134 flags hashpspool stripe_width 0
pool 8 'pool2' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 136 flags hashpspool stripe_width 0
pool 9 'pool3' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 138 flags hashpspool stripe_width 0
pool 10 'pool4' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 140 flags hashpspool stripe_width 0
pool 11 'pool5' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 142 flags hashpspool stripe_width 0
pool 12 'pool6' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 144 flags hashpspool stripe_width 0
pool 13 'pool7' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 146 flags hashpspool stripe_width 0
pool 14 'pool8' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 148 flags hashpspool stripe_width 0
pool 15 'pool9' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 150 flags hashpspool stripe_width 0
pool 16 'pool10' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 152 flags hashpspool stripe_width 0
pool 17 'pool11' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 154 flags hashpspool stripe_width 0
pool 18 'pool12' replicated size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 200 pgp_num 200 last_change 156 flags hashpspool stripe_width 0

我们看下这个是怎么得到的

pool rbd objects per pg (1912) is more than 50.3158 times cluster average (38)

rbd objects_per_pg = 15296 / 8 = 1912

objects_per_pg = 15920 /417 ≈ 38

50.3158 = rbd objects_per_pg / objects_per_pg = 1912 / 38

也就是出现其他pool的对象太少，而这个pg少，对象多，就会提示这个了，我们看下代码里面的判断

https://github.com/ceph/ceph/blob/master/src/mon/PGMonitor.cc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
int average_objects_per_pg = pg_map.pg_sum.stats.sum.num_objects / pg_map.pg_stat.size();
     if (average_objects_per_pg > 0 &&
         pg_map.pg_sum.stats.sum.num_objects >= g_conf->mon_pg_warn_min_objects &&
         p->second.stats.sum.num_objects >= g_conf->mon_pg_warn_min_pool_objects) {
int objects_per_pg = p->second.stats.sum.num_objects / pi->get_pg_num();
float ratio = (float)objects_per_pg / (float)average_objects_per_pg;
if (g_conf->mon_pg_warn_max_object_skew > 0 &&
    ratio > g_conf->mon_pg_warn_max_object_skew) {
  ostringstream ss;
  ss << "pool " << name << " has many more objects per pg than average (too few pgs?)";
  summary.push_back(make_pair(HEALTH_WARN, ss.str()));
  if (detail) {
    ostringstream ss;
    ss << "pool " << name << " objects per pg ("
       << objects_per_pg << ") is more than " << ratio << " times cluster average ("
       << average_objects_per_pg << ")";
    detail->push_back(make_pair(HEALTH_WARN, ss.str()));
  }

主要下面的几个限制条件

mon_pg_warn_min_objects = 10000 //总的对象超过10000

mon_pg_warn_min_pool_objects = 1000 //存储池对象超过1000

mon_pg_warn_max_object_skew = 10 //就是上面的存储池的平均对象与所有pg的平均值的倍数关系

解决问题

有三个方法解决这个警告的提示：

  • 删除无用的存储池
    如果集群中有一些不用的存储池，并且相对的pg数目还比较高，那么可以删除一些这样的存储池，从而降低mon_pg_warn_max_object_skew这个值，警告就会没有了

  • 增加提示的pool的pg数目
    有可能的情况就是，这个存储池的pg数目从一开始就不够，增加pg和pgp数目，同样降低了mon_pg_warn_max_object_skew这个值了

  • 增加mon_pg_warn_max_object_skew的参数值
    如果集群里面已经有足够多的pg了，再增加pg会不稳定，如果想去掉这个警告，就可以增加这个参数值，默认为10

总结

这个警告是比较的是存储池中的对象数目与整个集群的pg的平均对象数目的偏差，如果偏差太大就会发出警告

检查的步骤：

1
2
3
ceph health detail
ceph df
ceph osd dump | grep pool

mon_pg_warn_max_object_skew = 10.0

((objects/pg_num) in the affected pool)/(objects/pg_num in the entire system) >= 10.0 警告就会出现

变更记录

Why Who When
创建 武汉-运维-磨渣 2016-07-27