bcachefs device add yields inconsistent FS UUID in udev db and disk

[tasleson]

Note: using bcachefs-tools 3ac510f with fedora 39 6.8.9-200 kernel

If you format a 2 device FS, eg.

# ./bcachefs format /dev/mapper/mpathc /dev/mapper/mpathd --replicas=2

format out shows FS UUID = 90dac8ea-7960-4cb6-b92e-3f86b7eed350

we check using blkid -p and they match our expectations

# blkid -p /dev/mapper/mpathc
/dev/mapper/mpathc: UUID="90dac8ea-7960-4cb6-b92e-3f86b7eed350" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="aaa03f60-0900-49ec-b683-1053b67f01cc" FSSIZE="2000409329664" TYPE="bcachefs" USAGE="filesystem"
# blkid -p /dev/mapper/mpathd
/dev/mapper/mpathd: UUID="90dac8ea-7960-4cb6-b92e-3f86b7eed350" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="7d84a5a6-1118-496f-9350-199cee694c71" FSSIZE="2000409329664" TYPE="bcachefs" USAGE="filesystem"

mount it:

# ./bcachefs mount  /dev/mapper/mpathc /mnt/bcachefs

add another device to it

./bcachefs device add /mnt/bcachefs/ /dev/mapper/mpathb

checks it's blkid info:

# blkid -p /dev/mapper/mpathb
/dev/mapper/mpathb: UUID="e165c540-ea4a-45d6-8d7d-c412d0202840" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="020478e0-5a12-4a60-9c6c-477944787107" FSSIZE="1000204664832" TYPE="bcachefs" USAGE="filesystem"

and find that its FS UUID is different!

The udev db for this block device is also showing ID_FS_UUID=e165c540-ea4a-45d6-8d7d-c412d0202840

If we un-mount the bcachefs FS and check FS UUID on disk we find

# umount /mnt/bcachefs 
# blkid -p /dev/mapper/mpathb
/dev/mapper/mpathb: UUID="90dac8ea-7960-4cb6-b92e-3f86b7eed350" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="020478e0-5a12-4a60-9c6c-477944787107" FSSIZE="3000613994496" TYPE="bcachefs" USAGE="filesystem"

that the FS UUID now matches the original formatted FS.

However, the udev db still shows ID_FS_UUID=e165c540-ea4a-45d6-8d7d-c412d0202840

So if we now try to mount the FS using 1 of the device nodes or the FS UUID we fail to find all of the devices!

 ./bcachefs mount -v /dev/mapper/mpathc /mnt/bcachefs/
DEBUG - bcachefs::commands::mount: Walking udev db!
DEBUG - bcachefs::commands::mount: enumerating devices with UUID 90dac8ea-7960-4cb6-b92e-3f86b7eed350
INFO - bcachefs::commands::mount: mounting with params: device: /dev/dm-10:/dev/dm-17, target: /mnt/bcachefs/, options: 
DEBUG - bcachefs::commands::mount: parsing mount options: 
INFO - bcachefs::commands::mount: mounting bcachefs filesystem, /mnt/bcachefs/
INFO - bcachefs::commands::mount: mounting filesystem
ERROR - bcachefs::commands::mount: Fatal error: Invalid argument

dmesg shows:

[11617.259122] bcachefs: bch2_fs_open() bch_fs_open err opening /dev/dm-10: insufficient_devices_to_start

We can get it to mount if we supply all the block devices, eg.
./bcachefs mount -v /dev/mapper/mpathc:/dev/mapper/mpathb:/dev/mapper/mpathd /mnt/bcachefs/

So the question is, why does a newly added device have a different FS UUID when added?

I'm not sure how to fix this, so that when the super block gets written out is has the correct FS UUID during device add.

[tasleson]

If we update the udev db, # udevadm control --reload-rules && udevadm trigger
we can mount as expected

./bcachefs mount -v /dev/mapper/mpathc /mnt/bcachefs/
DEBUG - bcachefs::commands::mount: Walking udev db!
DEBUG - bcachefs::commands::mount: enumerating devices with UUID 90dac8ea-7960-4cb6-b92e-3f86b7eed350
INFO - bcachefs::commands::mount: mounting with params: device: /dev/dm-10:/dev/dm-17:/dev/dm-23, target: /mnt/bcachefs/, options: 
DEBUG - bcachefs::commands::mount: parsing mount options: 
INFO - bcachefs::commands::mount: mounting bcachefs filesystem, /mnt/bcachefs/
INFO - bcachefs::commands::mount: mounting filesystem
INFO - bcachefs::commands::mount: Successfully mounted