ceph-radosgw配置bucket的policy

背景

ceph通过radosgw对外提供了s3功能，这个能够提供一个s3接口供外部使用，s3的功能选项很多，本篇记录的是policy的功能配置

配置方法

配置policy可以通过s3cmd，或者windows的s3客户端都可以，这里我们使用s3cmd进行配置

安装配置s3cmd

下载客户端

1
[root@lab101 ceph]# yum install s3cmd

配置s3cmd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[root@lab101 ceph]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: test1
Secret Key: test1
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.0.101:7481

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket).192.168.0.101:7481

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: test1
  Secret Key: test1
  Default Region: US
  S3 Endpoint: 192.168.0.101:7481
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket).192.168.0.101:7481
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

如果有多个用户操作的需求，可以指定配置文件

1
s3cmd --config s3cfguser1

编写policy规则

首先需要写一个policy规则

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  "Version":  "2012-10-17",
  "Id":  "bucketname-write",
  "Statement":  [
    {
      "Sid":  "bucketname-write",
      "Effect":  "Allow",
      "Principal":  {
        "AWS":  [
           "arn:aws:iam:::user/test2"
        ]
      },
      "Action":  [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
      "Resource":  [
        "arn:aws:s3:::mybucket1/*",
        "arn:aws:s3:::mybucket1"
      ]
    }
  ]
}
  • Version : 这个是policy的规则的版本，这个地方只有两个固定日期可以写，2012-10-17 和2008-10-17 ,写其它日期后台会无法解析
  • Id: 这个就是这个policy的id，用于区分不同的policy的
  • Statement: 就是主体的配置
  • Sid: 就是statement内部的不同配置的标识
  • Effect: 这个就是配置允许，还是禁止的，有Allow和Deny
  • Principal: 这个里面就是控制对哪个角色进行的配置
  • Action: 这个就是有哪些操作
  • Resource: 这个就是对哪些资源的配置

上面的配置就是允许test2的用户对mybucket1的一些操作

1
2
[root@lab101 ~]# s3cmd setpolicy w.json s3://mybucket1
s3://mybucket1/: Policy updated

查询当前的policy的命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@lab101 ~]# s3cmd info s3://mybucket1
s3://mybucket1/ (bucket):
   Location:  default
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    {
  "Version":  "2012-10-17",
  "Id":  "bucketname-write",
  "Statement":  [
    {
      "Sid":  "bucketname-write",
      "Effect":  "Allow",
      "Principal":  {
        "AWS":  [
           "arn:aws:iam:::user/test2"
        ]
      },
      "Action":  [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
      "Resource":  [
        "arn:aws:s3:::mybucket1/*",
        "arn:aws:s3:::mybucket1"
      ]
    }
  ]
}

   CORS:      none
   ACL:       test1: FULL_CONTROL

删除policy的命令

1
2
[root@lab101 ~]# s3cmd delpolicy  s3://mybucket1
s3://mybucket1/: Policy deleted

附加

作为集群的维护者，我们需要掌握更多的信息，这个policy是在客户端进行设置，如果客户认为设置跟自己想象的有区别，那么我们如何去查看这些policy的信息，或者说，是否存在设置不正确，权限过大的情况

ceph本身没有命令直接查询这个policy的，这个是作为一个xattr扩展属性存储在对象里面的，我们看下怎么处理这个

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@lab101 ~]#  rados -p default.rgw.meta ls --all
users.keys  test2
users.keys  test3
root    .bucket.meta.mybucket2:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.2
root    mybucket3
users.uid   test2.buckets
root    .bucket.meta.mybucket3:27ff3ab2-6caf-43b1-9281-af0d05a57319.24160.1
users.uid   test1.buckets
users.keys  test1
users.uid   test2
root    .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1
users.swift test1:test3
users.uid   test1
root    mybucket2
root    mybucket1
users.keys  INMJZ9W82AFSJYA9T5Z4

对象是存储在root命令空间里面

1
2
3
4
5
6
7
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root ls
.bucket.meta.mybucket2:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.2
mybucket3
.bucket.meta.mybucket3:27ff3ab2-6caf-43b1-9281-af0d05a57319.24160.1
.bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1
mybucket2
mybucket1

上面的bucket meta里面就是存储的这个policy的信息的，设置了才有没有设置就没有

看下没设置的情况

1
2
3
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root  listxattr  .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1
ceph.objclass.version
user.rgw.acl

设置以后的情况

1
2
3
4
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root  listxattr  .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1
ceph.objclass.version
user.rgw.acl
user.rgw.iam-policy

也就是如果设置了policy就会多一个user.rgw.iam-policy

我们看下内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root  getxattr  .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1 user.rgw.iam-policy
{
  "Version":  "2008-10-17",
  "Id":  "bucketname-write",
  "Statement":  [
    {
      "Sid":  "bucketname-write",
      "Effect":  "Allow",
      "Principal":  {
        "AWS":  [
           "arn:aws:iam:::user/test2"
        ]
      },
      "Action":  [
        "s3:ListBucket",
        "s3:ListAllMyBuckets",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
      "Resource":  [
        "arn:aws:s3:::mybucket1/*",
        "arn:aws:s3:::mybucket1"
      ]
    }
  ]
}

这个是明文的json的没有进行序列化的，那么我们就可以从底层查看到哪些设置了哪些没有设置

这个地方还可以从底层进行修改

1
2
3
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root  getxattr  .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1 user.rgw.iam-policy > user.rgw.iam-policy.json
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root  setxattr  .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1 user.rgw.iam-policy < user.rgw.iam-policy.json
[root@lab101 ~]# rados -p default.rgw.meta --namespace=root  getxattr  .bucket.meta.mybucket1:27ff3ab2-6caf-43b1-9281-af0d05a57319.24123.1 user.rgw.iam-policy

但是这个地方有个问题，设置了后，底层是马上更新了，但是客户端那边

1
[root@lab101 ~]# s3cmd info s3://mybucket1

这个并没有更新

1
2
3
2024-09-05T11:42:18.074+0800 7f15435db700 20 get_system_obj_state: rctx=0x7f15435d1988 obj=default.rgw.meta:users.uid:test1 state=0x55a9b8284040 s->prefetch_data=0
2024-09-05T11:42:18.074+0800 7f15435db700 10 cache get: name=default.rgw.meta+users.uid+test1 : hit (requested=0x6, cached=0x17)
2024-09-05T11:42:18.074+0800 7f15435db700 20 get_system_obj_state: s->obj_tag was set empty

日志看是命中了缓存，说明这个信息在rgw这边缓存了，这个重启下rgw的进程就刷新了，所以这个底层的操作并不适合频繁的去设置

这个地方查看检查还是可以的，也就是我们哪些bucket设置了policy，设置了什么policy，这个是可以拿到的，设置的通过前端设置即可，或者有比较特殊的批量的需求的时候，再考虑在底层设置

总结

本篇记录了policy的设置以及从底层查询设置的规则方法