こんにちは。OpenStack KeystoneをCeph認証メカニズムとして統合して、CephオブジェクトストレージをOpenStack Swiftバックエンドとして使用できるようにしたいと考えています。
環境：
Kernel : Ubunutu Server LTS 22.04 (minimal)
Openstack : Zed (Manual Installation)
Ceph : quiny (Cephadm Installation)
//コントローラーノード
openstack service create --name swift object-store
openstack user create --domain default --password-prompt swift
openstack user create --domain default --password-prompt rgw
openstack role add --user swift --project service admin
openstack role add --user swift --project service swiftoperator
openstack role add --user rgw --project service admin
openstack role add --user rgw --project service swiftoperator
openstack endpoint create --region Tehran object-store public http://<rados_gatway>:8080/swift/v1
openstack endpoint create --region Tehran object-store internal http://<rados_gatway>:8080/swift/v1
openstack endpoint create --region Tehran object-store admin http://<rados_gatway>:8080/swift/v1
//Ceph クラスター
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_api_version 3
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_url http://<keystone_url>:5000
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_accepted_roles admin,member,swiftoperator,Member,_member_
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_token_cache_size 500
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_admin_user rgw
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_admin_password rgw
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_admin_domain default
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_keystone_admin_project service
ceph config set client.rgw.default.ceph-2.ncsnqh rgw_s3_auth_use_keystone true
今実行するとswift listこのエラーが発生します;(
Account GET failed: http://<rados_gatway>:8080/swift/v1?format=json 401 Unauthorized [first 60 chars of response] b'{"Code":"AccessDenied","RequestId":"tx00000ff92593343f6fbac-'
Failed Transaction ID: tx00000ff92593343f6fbac-0063b3dcd8-455e0-default
何か見逃しているような気がします。たくさんのドキュメントを読みましたが、解決策を見つけたのはそのうちの1つだけで、それはOpenStackでradosgwユーザーを作成し、それにSwiftオペレーターロールを割り当てることでした。私はそれを実行しましたが、問題は依然として存在します。ちなみに、Swiftユーザーはサービスプロジェクトで管理者ロールを持っているにもかかわらず、Swiftオペレーターロールを割り当てました。それでも問題は解決しません。
curl -v http://<keystone_url>:5000 (on ceph-2 returns no error)
ここに完全なswift list --debug
DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://<keystone_url>:5000/v3/auth/tokens
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): <keystone_url>:5000
DEBUG:urllib3.connectionpool:http://<keystone_url>:5000 "POST /v3/auth/tokens HTTP/1.1" 201 4678
DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "6622244113204a689e3a367847291166", "name": "hoodad", "password_expires_at": null}, "audit_ids": ["zNYqN-lESbCt8U1MA3tl5Q"], "expires_at": "2023-01-03T08:41:55.000000Z", "issued_at": "2023-01-03T07:41:55.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "36905f5fbaa64feaa0a47dcc3d8f5455", "name": "admin"}, "is_domain": false, "roles": [{"id": "5365f6dcb2fc4577a3c31693e671e5ee", "name": "reader"}, {"id": "7d90492c8771403b93d5bf8e1d33e40b", "name": "admin"}, {"id": "514cde82919e436aaec7568ad1ba4bee", "name": "member"}], "catalog": [{"endpoints": [{"id": "349bda8b61cc4bee932887f213de41c7", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:8776/v3/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "8c981d7f64f74174ba1a0bc3eaf4aa91", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:8776/v3/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "9c94c3bdc0394abea5f3646f8986022f", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:8776/v3/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}], "id": "061f492117d74190bc0084986feb377a", "type": "volumev3", "name": "cinder"}, {"endpoints": [{"id": "3fd3b010a41b4a3a86fa76b308f3a053", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:5000/v3/", "region": "Tehran"}, {"id": "455c040c8f304f4e99eae8104a57ec17", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:5000/v3/", "region": "Tehran"}, {"id": "86e24eb7164d449da0a8bf56af1d56b7", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:5000/v3/", "region": "Tehran"}], "id": "1c97054f81db4fcc8ed16d3aa42869a9", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "16be2834ab4d4fdb9c4c293b550d4980", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:8778", "region": "Tehran"}, {"id": "b58356f6e25747a7bce5e9c9c4a0bd7e", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:8778", "region": "Tehran"}, {"id": "cd7430601c6d4e928e3ea279aa75d63d", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:8778", "region": "Tehran"}], "id": "6408ac009be64d93b82c6803aad17607", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "a21330d45c8f4530a06c99a62c187e14", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:9696", "region": "Tehran"}, {"id": "dbc4d63dfbb94ae7afcf20458b428319", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:9696", "region": "Tehran"}, {"id": "de26e420bc994cb9b9332922f088a670", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:9696", "region": "Tehran"}], "id": "694e0790a22d41c29585b786bc263009", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "85434ec1ef2e4d2ca52f0467df6a9001", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:9292", "region": "Tehran"}, {"id": "98773c3f67b640f18f53885b569e4d73", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:9292", "region": "Tehran"}, {"id": "c1e80464da124b7eaa0279e28c1f25d2", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:9292", "region": "Tehran"}], "id": "c8fe90a32cfc417fa5369b60092c0dfc", "type": "image", "name": "glance"}, {"endpoints": [{"id": "26464c37332e4c0da96ca4e8f7b82ae9", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:8774/v2.1/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "623dc6a74b634117b3edcc5892cc1bbb", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:8774/v2.1/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "8314196253d2446aaeec6e9e6e45fd47", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:8774/v2.1/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}], "id": "ccc120553ee14e3f8b3157f698190492", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "8751532bb2ca4f81a0f51ecb67df6eb4", "interface": "public", "region_id": "Tehran", "url": "http://<rados_gatway>:8080/swift/v1", "region": "Tehran"}, {"id": "9f31cf8882554f199ab9ead345e05825", "interface": "internal", "region_id": "Tehran", "url": "http://<rados_gatway>:8080/swift/v1", "region": "Tehran"}, {"id": "e98011666e844f13aac4e423a316fde6", "interface": "admin", "region_id": "Tehran", "url": "http://<rados_gatway>:8080/swift/v1", "region": "Tehran"}], "id": "e38f897497d547c6a06bb6a52be1be13", "type": "object-store", "name": "swift"}]}}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): <rados_gatway>:8080
DEBUG:urllib3.connectionpool:http://<rados_gatway>:8080 "GET /swift/v1?format=json HTTP/1.1" 401 119
INFO:swiftclient:REQ: curl -i http://<rados_gatway>:8080/swift/v1?format=json -X GET -H "X-Auth-Token: gAAAAABjs9xDrD6NgcD6Uyatc0QH4q74_SiztiLkYPpoHKK0b8yGWwyXfAw-V4klq7x6nCekqmHwa2ELQVHI_Cj5AzygU98Hdr6rrrpL3Wihl1CdqMyoXnw_GdNWh4dNQPGxOQatYXR2XwU5U7r9Juv-G4cJjFYFh5RRKyPNCzN6z_vhI-xm5sc" -H "Accept-Encoding: gzip"
INFO:swiftclient:RESP STATUS: 401 Unauthorized
INFO:swiftclient:RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx00000a7c6f54a4f0a7eac-0063b3dc43-455e0-default', 'X-Openstack-Request-Id': 'tx00000a7c6f54a4f0a7eac-0063b3dc43-455e0-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Tue, 03 Jan 2023 07:41:55 GMT', 'Connection': 'Keep-Alive'}
INFO:swiftclient:RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx00000a7c6f54a4f0a7eac-0063b3dc43-455e0-default","HostId":"455e0-default-default"}'
DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://<keystone_url>:5000/v3/auth/tokens
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): <keystone_url>:5000
DEBUG:urllib3.connectionpool:http://<keystone_url>:5000 "POST /v3/auth/tokens HTTP/1.1" 201 4678
DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "6622244113204a689e3a367847291166", "name": "hoodad", "password_expires_at": null}, "audit_ids": ["B3g606MNTUqZS6tUgEHyHQ"], "expires_at": "2023-01-03T08:41:56.000000Z", "issued_at": "2023-01-03T07:41:56.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "36905f5fbaa64feaa0a47dcc3d8f5455", "name": "admin"}, "is_domain": false, "roles": [{"id": "5365f6dcb2fc4577a3c31693e671e5ee", "name": "reader"}, {"id": "7d90492c8771403b93d5bf8e1d33e40b", "name": "admin"}, {"id": "514cde82919e436aaec7568ad1ba4bee", "name": "member"}], "catalog": [{"endpoints": [{"id": "349bda8b61cc4bee932887f213de41c7", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:8776/v3/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "8c981d7f64f74174ba1a0bc3eaf4aa91", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:8776/v3/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "9c94c3bdc0394abea5f3646f8986022f", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:8776/v3/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}], "id": "061f492117d74190bc0084986feb377a", "type": "volumev3", "name": "cinder"}, {"endpoints": [{"id": "3fd3b010a41b4a3a86fa76b308f3a053", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:5000/v3/", "region": "Tehran"}, {"id": "455c040c8f304f4e99eae8104a57ec17", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:5000/v3/", "region": "Tehran"}, {"id": "86e24eb7164d449da0a8bf56af1d56b7", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:5000/v3/", "region": "Tehran"}], "id": "1c97054f81db4fcc8ed16d3aa42869a9", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "16be2834ab4d4fdb9c4c293b550d4980", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:8778", "region": "Tehran"}, {"id": "b58356f6e25747a7bce5e9c9c4a0bd7e", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:8778", "region": "Tehran"}, {"id": "cd7430601c6d4e928e3ea279aa75d63d", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:8778", "region": "Tehran"}], "id": "6408ac009be64d93b82c6803aad17607", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "a21330d45c8f4530a06c99a62c187e14", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:9696", "region": "Tehran"}, {"id": "dbc4d63dfbb94ae7afcf20458b428319", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:9696", "region": "Tehran"}, {"id": "de26e420bc994cb9b9332922f088a670", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:9696", "region": "Tehran"}], "id": "694e0790a22d41c29585b786bc263009", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "85434ec1ef2e4d2ca52f0467df6a9001", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:9292", "region": "Tehran"}, {"id": "98773c3f67b640f18f53885b569e4d73", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:9292", "region": "Tehran"}, {"id": "c1e80464da124b7eaa0279e28c1f25d2", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:9292", "region": "Tehran"}], "id": "c8fe90a32cfc417fa5369b60092c0dfc", "type": "image", "name": "glance"}, {"endpoints": [{"id": "26464c37332e4c0da96ca4e8f7b82ae9", "interface": "admin", "region_id": "Tehran", "url": "http://<keystone_url>:8774/v2.1/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "623dc6a74b634117b3edcc5892cc1bbb", "interface": "public", "region_id": "Tehran", "url": "http://<keystone_url>:8774/v2.1/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}, {"id": "8314196253d2446aaeec6e9e6e45fd47", "interface": "internal", "region_id": "Tehran", "url": "http://<keystone_url>:8774/v2.1/36905f5fbaa64feaa0a47dcc3d8f5455", "region": "Tehran"}], "id": "ccc120553ee14e3f8b3157f698190492", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "8751532bb2ca4f81a0f51ecb67df6eb4", "interface": "public", "region_id": "Tehran", "url": "http://<rados_gatway>:8080/swift/v1", "region": "Tehran"}, {"id": "9f31cf8882554f199ab9ead345e05825", "interface": "internal", "region_id": "Tehran", "url": "http://<rados_gatway>:8080/swift/v1", "region": "Tehran"}, {"id": "e98011666e844f13aac4e423a316fde6", "interface": "admin", "region_id": "Tehran", "url": "http://<rados_gatway>:8080/swift/v1", "region": "Tehran"}], "id": "e38f897497d547c6a06bb6a52be1be13", "type": "object-store", "name": "swift"}]}}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): <rados_gatway>:8080
DEBUG:urllib3.connectionpool:http://<rados_gatway>:8080 "GET /swift/v1?format=json HTTP/1.1" 401 119
INFO:swiftclient:REQ: curl -i http://<rados_gatway>:8080/swift/v1?format=json -X GET -H "X-Auth-Token: gAAAAABjs9xEgshf9a7GAexTEQ27dZFkFSP7TaC-o-2Bba_WbaH7WeMS9ohHrJhlU_tFdcWsd-71UEE4e33bOEtA8vM6yA6Nu2IAm8SU2QN6Ox5tuhps5Dc0E_inQfqxg-9cAgpjwsm8czG06SsCku6Cgxt-UqSdyCGn9CcShRgH0u7Mb1eyEvw" -H "Accept-Encoding: gzip"
INFO:swiftclient:RESP STATUS: 401 Unauthorized
INFO:swiftclient:RESP HEADERS: {'Content-Length': '119', 'X-Trans-Id': 'tx0000081618694ce1134ad-0063b3dc44-455e0-default', 'X-Openstack-Request-Id': 'tx0000081618694ce1134ad-0063b3dc44-455e0-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Tue, 03 Jan 2023 07:41:56 GMT', 'Connection': 'Keep-Alive'}
INFO:swiftclient:RESP BODY: b'{"Code":"AccessDenied","RequestId":"tx0000081618694ce1134ad-0063b3dc44-455e0-default","HostId":"455e0-default-default"}'
ERROR:swiftclient.service:Account GET failed: http://<rados_gatway>:8080/swift/v1?format=json 401 Unauthorized [first 60 chars of response] b'{"Code":"AccessDenied","RequestId":"tx0000081618694ce1134ad-' (txn: tx0000081618694ce1134ad-0063b3dc44-455e0-default)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swiftclient/service.py", line 949, in _list_account_job
_, items = conn.get_account(
File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 1911, in get_account
return self._retry(None, get_account, marker=marker, limit=limit,
File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 1856, in _retry
rv = func(self.url, self.token, *args,
File "/usr/lib/python3/dist-packages/swiftclient/client.py", line 883, in get_account
raise ClientException.from_response(resp, 'Account GET failed', body)
swiftclient.exceptions.ClientException: Account GET failed: http://<rados_gatway>:8080/swift/v1?format=json 401 Unauthorized [first 60 chars of response] b'{"Code":"AccessDenied","RequestId":"tx0000081618694ce1134ad-' (txn: tx0000081618694ce1134ad-0063b3dc44-455e0-default)
Account GET failed: http://<rados_gatway>:8080/swift/v1?format=json 401 Unauthorized [first 60 chars of response] b'{"Code":"AccessDenied","RequestId":"tx0000081618694ce1134ad-'
Failed Transaction ID: tx0000081618694ce1134ad-0063b3dc44-455e0-default
答え1
[未完了] Ceph Octopus で動作させることができました (構成を少し変更しました)。詳細は以下を参照してください。まだ Pacific RGW 構成を動作させようとしていますが、Quincy も現在は動作しません。
最新のクロスプロジェクトノートを見ましたドキュメント完全に満足のいく回答ではないかもしれませんが、メモを貼り付けるにはコメントだけでは不十分です。Ceph Octopus と OpenStack Victoria でうまくいった方法は次のとおりです。
# ceph.conf
[client.rgw.keystone.storage01.vtakeh]
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_api_version 3
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_accepted_roles "admin,Member,_member_,member"
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_admin_user rgw
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_admin_password ****
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_admin_domain default
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_admin_project service
ceph config set client.rgw.keystone.storage01.vtakeh rgw_s3_auth_use_keystone true
ceph config set client.rgw.keystone.storage01.vtakeh rgw_keystone_url https://control.fqdn:5000
ceph config set client.rgw.keystone.storage01.vtakeh rgw_swift_account_in_url true
---
# create swift service
$ openstack service create --name=swift --description="Swift Service" object-store
$ openstack user create rgw --password *****
# add role to user
$ openstack role add --user rgw --project service admin
# create keystone endpoints
$ openstack endpoint create --region RegionOne swift admin "http://ses6-mon1.fqdn:80/swift/v1/AUTH_$(project_id)s"
$ openstack endpoint create --region RegionOne swift internal "http://ses6-mon1.fqdn:80/swift/v1/AUTH_$(project_id)s"
$ openstack endpoint create --region RegionOne swift public "http://ses6-mon1.fqdn:80/swift/v1/AUTH_$(project_id)s"
これらのオプションを設定した後、正常に実行できましたopenstack container create swift1。上記のコマンドは、新しい RGW をセットアップし、OpenStack 経由でアクセスするのに役立ちました。
control01:~ # openstack container create swift1
+---------+-----------+---------------------------------------------------+
| account | container | x-trans-id |
+---------+-----------+---------------------------------------------------+
| v1 | swift1 | tx0000095214f842753ecaa-00639b24cd-606d44-default |
+---------+-----------+---------------------------------------------------+
control01:~ # openstack container list
+--------+
| Name |
+--------+
| swift1 |
+--------+
小さな変更を加えると、基本的に ceph nautilus と openstack rocky でも同じように動作しました。