Introduction

The impetus for this blog post was the quite frustrating endeavor to swap 2 disk on my Proxmox cluster between two of my nodes. Both of these disks were originally part of the Ceph cluster I use for my shared storage. The reason for the swap was simply to make the storage per node a little bit more balanced, the disks in question were a 240GB and a 500GB SSD.

Disclaimer I do not know how dangerous the commands in this post are and they may do more harm than good

The problem

The start the disk swap I set both disks as "Out" in the Proxmox Ceph GUI and then set them to down, after which I destroyed the running OSDs. (This had no real effect on the health of the pool as I am running with size=3 on the cluster, so this only triggered some balance/back-filling).

I assumed, wrongly, that the 'destroy' operation neatly cleaned up the disks and the underlying LVM structures that Ceph uses for its OSDs.

Once I inserted the 500GB disk in the next host it popped up in lsblk as having a LVM entry. This entry did not exist in the various vgdisplay, lvdisplay, pvdisplay lvm outputs. With parted I decided to just reinitialize the disk as having a new clean GPT, expecting that to do the trick. After having done that re-reading the partition table using partprobe the LVM entry was still there. Even more curious, in the Proxmox GUI the disk was shown to be of the type 'Device Mapper'. This should have triggered some bells but it didn't.

Instead of lookup up what exactly a 'Device Mapper' disk was I kept trying to erase the disk with classical means, wipefs to clear the signatures, gdisk to zap all GPT tables, etc. None of this worked and I was stuck with a disk that I could not re-add to the pool.

The solution

Finally I decided to actually see what was going on with the disk and using the fdisk -l output I saw the LVM entry showing up as a /dev/mapper/ device.

I have no idea what exactly device mapper devices are and how they are significant, but I found that they could be managed with the dmsetup command.

I found the device at fault under dmsetup ls, with its long Ceph string. And then simply removed it using dmsetup remove $longcephstring$. As soon as that was done the entry was gone from lsblk and the disk showed up as unused under the Proxmox GUI.

Finally, I could re-use the disk in Ceph as a new OSD.