Ich habe gerade drei Speicherknoten für Ceph in einer QEMU/KVM-Umgebung eingerichtet, um mehr darüber zu erfahren. Das Betriebssystem ist Ubuntu Server 22.04 LTS. Die Ceph- und Ceph-Common-Pakete stammen aus Standard-Repos. Ich habe die Richtlinien auf der Ceph-Website befolgt. Ich habe eine VM eingerichtet und sie geklont, bevor ich Ceph installiert habe. Die Hosts und Dashboards sind einsatzbereit. Jeder Host verfügt über ein leeres Blockgerät /dev/vdb. Gemäß der Anleitung habe ich versucht, sie hinzuzufügen, indem ich
ceph orch apply osd --all-available-devices
Bei zwei Knoten hat es funktioniert, beim dritten ist es jedoch fehlgeschlagen. Der Daemon versucht das Blockgerät zu integrieren, scheitert dabei. Er versucht es alle 10 Sekunden erneut. Leider ist die Fehlermeldung fast unlesbar:
5/31/23 2:29:05 PM[WRN]Health check update: Failed to apply 2 service(s): osd.all-available-devices,osd.cost_capacity (CEPHADM_APPLY_SPEC_FAIL)
5/31/23 2:29:05 PM[ERR]Failed to apply osd.cost_capacity spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: cost_capacity service_name: osd.cost_capacity placement: host_pattern: '*' spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/mon.san2/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 -e NODE_NAME=san2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/run/ceph:z -v /var/log/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/log/ceph:z -v /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpu0fdkklo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbb0nzrrm:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 lvm batch --no-auto /dev/vdb --yes --no-systemd /usr/bin/docker: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/docker: stderr --> relative data size: 1.0 /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /dev/vdb /usr/bin/docker: stderr stdout: Physical volume "/dev/vdb" successfully created. /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully created /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 2559 -n osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" created. /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 /usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /var/lib/ceph/osd/ceph-1/block /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap /usr/bin/docker: stderr stderr: got monmap epoch 3 /usr/bin/docker: stderr --> Creating keyring file for osd.1 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid fec3748c-f6d3-4388-9402-1bde5664df85 --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.012+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bdev(0x55746fa91400 /var/lib/ceph/osd/ceph-1//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (13) Permission denied[0m /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.1 /usr/bin/docker: stderr --> Zapping: /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 bs=1M count=10 conv=fsync /usr/bin/docker: stderr stderr: 10+0 records in /usr/bin/docker: stderr 10+0 records out /usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0415005 s, 253 MB/s /usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stderr: Removing ceph--7408d699--5fa5--4ad7--94c6--364338c2ef86-osd--block--fec3748c--f6d3--4388--9402--1bde5664df85 (253:0) /usr/bin/docker: stderr stderr: Archiving volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" metadata (seqno 5). /usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" /usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" (seqno 6). /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" successfully removed /usr/bin/docker: stderr stderr: Removing physical volume "/dev/vdb" from volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully removed /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/vdb /usr/bin/docker: stderr stdout: Labels on physical volume "/dev/vdb" successfully wiped. /usr/bin/docker: stderr --> Zapping successful for OSD: 1 /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 441, in main /usr/bin/docker: stderr self._execute(plan) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 460, in _execute /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create /usr/bin/docker: stderr prepare_step.safe_prepare(args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare /usr/bin/docker: stderr self.prepare() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 394, in prepare /usr/bin/docker: stderr osd_fsid, /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 119, in prepare_bluestore /usr/bin/docker: stderr db=db /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 484, in osd_mkfs_bluestore /usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) /usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid fec3748c-f6d3-4388-9402-1bde5664df85 --setuser ceph --setgroup ceph Traceback (most recent call last): File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 9653, in main() File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 9641, in main r = ctx.func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2153, in _infer_config return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2069, in _infer_fsid return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2181, in _infer_image return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2056, in _validate_fsid return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 6254, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 -e NODE_NAME=san2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/run/ceph:z -v /var/log/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/log/ceph:z -v /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpu0fdkklo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbb0nzrrm:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 lvm batch --no-auto /dev/vdb --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 509, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 77, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 64, in create_from_spec_one replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host code, '\n'.join(err))) RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/mon.san2/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 -e NODE_NAME=san2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/run/ceph:z -v /var/log/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/log/ceph:z -v /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpu0fdkklo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbb0nzrrm:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 lvm batch --no-auto /dev/vdb --yes --no-systemd /usr/bin/docker: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/docker: stderr --> relative data size: 1.0 /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /dev/vdb /usr/bin/docker: stderr stdout: Physical volume "/dev/vdb" successfully created. /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully created /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 2559 -n osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" created. /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 /usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /var/lib/ceph/osd/ceph-1/block /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap /usr/bin/docker: stderr stderr: got monmap epoch 3 /usr/bin/docker: stderr --> Creating keyring file for osd.1 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid fec3748c-f6d3-4388-9402-1bde5664df85 --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.012+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bdev(0x55746fa91400 /var/lib/ceph/osd/ceph-1//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (13) Permission denied[0m /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.1 /usr/bin/docker: stderr --> Zapping: /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 bs=1M count=10 conv=fsync /usr/bin/docker: stderr stderr: 10+0 records in /usr/bin/docker: stderr 10+0 records out /usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0415005 s, 253 MB/s /usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stderr: Removing ceph--7408d699--5fa5--4ad7--94c6--364338c2ef86-osd--block--fec3748c--f6d3--4388--9402--1bde5664df85 (253:0) /usr/bin/docker: stderr stderr: Archiving volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" metadata (seqno 5). /usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" /usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" (seqno 6). /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" successfully removed /usr/bin/docker: stderr stderr: Removing physical volume "/dev/vdb" from volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully removed /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/vdb /usr/bin/docker: stderr stdout: Labels on physical volume "/dev/vdb" successfully wiped. /usr/bin/docker: stderr --> Zapping successful for OSD: 1 /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
[...] Der wichtige Teil scheint zu sein (neu formatiert):
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.132+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.308+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 bdev(0x55897a203400 /var/lib/ceph/osd/ceph-1//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (13) Permission denied[0m
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.1
/usr/bin/docker: stderr --> Zapping: /dev/ceph-d36a727e-b4e7-43c9-aa8b-bd8d82c0f84f/osd-block-42b7003b-a98d-4020-8513-07299f23c63a /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-d36a727e-b4e7-43c9-aa8b-bd8d82c0f84f/osd-block-42b7003b-a98d-4020-8513-07299f23c63a bs=1M count=10 conv=fsync
Ich habe versucht, jedem Schreibrechte für den gesamten Baum /var/lib/ceph zu erteilen. Aber es hat nicht geholfen. Es ist bereits das zweite Mal, dass ich von vorne anfange. Das andere Mal hatte ich denselben Fehler auf einem anderen Knoten.
Antwort1
ich hatte dasselbe Problem und nachdem ich den Befehl „ceph orch daemon osd add ceph_hostname:/dev/sdX“ verfolgt hatte, wurde mir klar, dass einige Pakete nicht installiert waren. Nach der Installation des Ceph-Pakets auf diesem Knoten war das Problem gelöst.
strace ceph orch daemon fügt osd ceph_hostname hinzu:/dev/sdX
Ubuntu: apt-get install ceph-common