rgw的d3n功能配置

背景

最近在看缓存相关，文件系统可以通过fscache加速，加速的效果就是读取的时候能够缓存，原理是在网关的地方加入一个高速缓存盘，这样在后续读取的时候，能够直接从缓存盘读取，这样能够减少与集群的交互，从而提供更大的性能，并且这个是缓存读取，所以数据安全性没有问题

rgw的d3n

这个功能就是给rgw的网关加入了一个缓存盘，指定一个目录，然后能够缓存数据到目录

版本要求

1
2
[root@lab201 ~]# ceph -v
ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable)

至少要这个版本才可以

操作系统要centos9

配置方法

1
2
3
4
5
6
7
8
debug_rgw=20
[client.rgw.lab201]
  host = lab201
  rgw_frontends="beast port=7481"
  rgw_content_length_compat = true
  rgw_d3n_l1_local_datacache_enabled = true
  rgw_d3n_l1_datacache_persistent_path = "/mnt/nvme0/rgw_datacache"
  rgw_d3n_l1_datacache_size = 10737418240

改好了重启进程即可

1
systemctl restart ceph-radosgw@rgw.lab201

有问题的时候看日志

注意问题

  • 文件需要大于4M才会缓存
  • 注意缓存目录的权限需要给ceph权限

4M的原因是:
D3N 当前仅缓存尾部对象，因为它们是不可变的（默认情况下它是大于 4MB 的对象的一部分）。（NGINX RGW 数据缓存和 CDN 支持缓存所有对象大小）
也就是文件的4M之后的部分是不可变的，默认去缓存这部分的对象

没有权限的报错信息

1
2
3
4
5
6
2025-04-02T17:38:30.910+0800 7fee1770e640  1 D3nDataCache: create_aio_write_request fail, r=-1
2025-04-02T17:38:30.916+0800 7fee1770e640 10 D3nDataCache::put(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1, len=4194304
2025-04-02T17:38:30.916+0800 7fee1770e640 20 D3nDataCache: Before eviction _free_data_cache_size:10737418240, _outstanding_write_size:0, freed_size:0
2025-04-02T17:38:30.916+0800 7fee1770e640  0 ERROR: D3nCacheAioWriteRequest::create_io: open file failed, errno=13, location='/mnt/nvme0/rgw_datacache/16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1'
2025-04-02T17:38:30.916+0800 7fee1770e640  0 ERROR: D3nDataCache: d3n_libaio_create_write_request() prepare libaio write op r=-1
2025-04-02T17:38:30.916+0800 7fee1770e640  1 D3nDataCache: create_aio_write_request fail, r=-1

测试读取

1
2
[root@lab201 ceph]# aws   --endpoint=http://192.168.0.201:7481 s3 cp s3://testbucket/testfile8M /tmp/cccccc
download: s3://testbucket/testfile8M to ../../tmp/cccccc

查看日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2025-04-02T17:43:07.435+0800 7f860fc987c0  1 D3N datacache enabled: 1
2025-04-02T17:43:07.596+0800 7f860fc987c0  5 D3nDataCache: init: evicting the persistent storage directory on start
2025-04-02T17:50:55.169+0800 7f85416e2640 20 req 134200468731872662 0.004000050s s3:get_obj D3nDataCache::get_obj_iterate_cb(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1, is_head_obj=0, obj-ofs=0, read_ofs=0, len=4194304
2025-04-02T17:50:55.169+0800 7f85416e2640 20 req 134200468731872662 0.004000050s s3:get_obj D3nDataCache: get_obj_iterate_cb(): WRITE TO CACHE: oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1, obj-ofs=0, read_ofs=0 len=4194304
2025-04-02T17:50:55.169+0800 7f85416e2640 20 req 134200468731872662 0.004000050s s3:get_obj D3nDataCache::get_obj_iterate_cb(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1, is_head_obj=0, obj-ofs=4194304, read_ofs=0, len=4194304
2025-04-02T17:50:55.169+0800 7f85416e2640 20 req 134200468731872662 0.004000050s s3:get_obj D3nDataCache: get_obj_iterate_cb(): WRITE TO CACHE: oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1, obj-ofs=4194304, read_ofs=0 len=4194304
2025-04-02T17:50:55.183+0800 7f853eedd640 10 D3nDataCache::put(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1, len=4194304
2025-04-02T17:50:55.183+0800 7f853eedd640 20 D3nDataCache: Before eviction _free_data_cache_size:10737418240, _outstanding_write_size:0, freed_size:0
2025-04-02T17:50:55.196+0800 7f85446e8640 10 D3nDataCache::put(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1, len=4194304
2025-04-02T17:50:55.196+0800 7f85446e8640 20 D3nDataCache: Before eviction _free_data_cache_size:10737418240, _outstanding_write_size:4194304, freed_size:0
2025-04-02T17:50:55.212+0800 7f8461522640  5 D3nDataCache: d3n_libaio_write_completion_cb(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1
2025-04-02T17:50:55.367+0800 7f8461522640  5 D3nDataCache: d3n_libaio_write_completion_cb(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1
[root@lab201 ~]# ll /mnt/nvme0/rgw_datacache/
total 8192
-rw-r--r-- 1 ceph ceph 4194304 Apr  2 17:50 16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1
-rw-r--r-- 1 ceph ceph 4194304 Apr  2 17:50 16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1

上面的可以看到能够缓存文件到目录

第二次读取

1
2
3
4
5
6
7
8
9
10
11
[root@lab201 ~]# cat  /var/log/ceph/ceph-client.rgw.lab201.log |grep -i d3n
2025-04-02T17:52:29.311+0800 7f850de7b640 20 req 10062238110711655751 0.004000050s s3:get_obj D3nDataCache::get_obj_iterate_cb(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1, is_head_obj=0, obj-ofs=0, read_ofs=0, len=4194304
2025-04-02T17:52:29.311+0800 7f850de7b640 20 req 10062238110711655751 0.004000050s s3:get_obj D3nDataCache: get_obj_iterate_cb(): READ FROM CACHE: oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1, obj-ofs=0, read_ofs=0, len=4194304
2025-04-02T17:52:29.311+0800 7f850de7b640 20 req 10062238110711655751 0.004000050s s3:get_obj D3nDataCache: file_aio_read_abstract(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1
2025-04-02T17:52:29.311+0800 7f850de7b640 20 req 10062238110711655751 0.004000050s s3:get_obj D3nDataCache: async_read(): location=/mnt/nvme0/rgw_datacache//16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1
2025-04-02T17:52:29.311+0800 7f850de7b640 20 req 10062238110711655751 0.004000050s s3:get_obj D3nDataCache: init_async_read(): location=/mnt/nvme0/rgw_datacache//16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__multipart_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1
2025-04-02T17:52:29.312+0800 7f850de7b640 20 req 10062238110711655751 0.005000062s s3:get_obj D3nDataCache: async_read(): ::aio_read(), ret=0
2025-04-02T17:52:29.312+0800 7f850de7b640 20 req 10062238110711655751 0.005000062s s3:get_obj D3nDataCache::get_obj_iterate_cb(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1, is_head_obj=0, obj-ofs=4194304, read_ofs=0, len=4194304
2025-04-02T17:52:29.312+0800 7f850de7b640 20 req 10062238110711655751 0.005000062s s3:get_obj D3nDataCache: get_obj_iterate_cb(): READ FROM CACHE: oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1, obj-ofs=4194304, read_ofs=0, len=4194304
2025-04-02T17:52:29.312+0800 7f850de7b640 20 req 10062238110711655751 0.005000062s s3:get_obj D3nDataCache: file_aio_read_abstract(): oid=16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1
2025-04-02T17:52:29.312+0800 7f850de7b640 20 req 10062238110711655751 0.005000062s s3:get_obj D3nDataCache: async_read(): location=/mnt/nvme0/rgw_datacache//16e7b827-16ac-471c-a407-ef1872bd0058.34134.1__shadow_testfile8M.2~PGlAYwIJKyDrpFDYY4WWQJ1c_E0eVWV.1_1

可以看到从本地磁盘读取的

总结

d3n功能非常适合作为读取的场景的使用，直接从本地盘的缓存读取，能够加速并且能够降低内部的网络消耗

附加

后面还有个D4N，这个功能多了一个write-back，从数据安全性角度来看，读取的不会破坏存储的当前的稳定性，带写入的，这个就需要多考虑下了，后面稳定了再看看,写缓存涉及的东西就多了，缓存什么时候下刷，什么时候清理，满了怎么处理客户端io，维持一个什么缓存数目，等等，需要调测的东西比较多

存储相关
#ceph
rgw的d3n功能配置
https://zphj1987.com/2025/04/02/rgw的d3n功能配置/
作者
zphj1987
发布于
2025年4月2日
许可协议