This was unexpected:
Write something to a partition device (eg. /dev/vda1) and immediately call blockdev --rereadpt /dev/vda to re-read the partition table of the whole device. Sometimes (about 50% for me) the blockdev command fails with:
blockdev: BLKRRPART: Device or resource busy
Nothing else is using /dev/vda, nothing from it was mounted, and the error was intermittent which indicates a race condition.
Why this happens:
udev has a rule that runs blkid -o udev -p /dev/vda1. It does this every time you close a block device so that blkid can rescan the content of the device.
The act of blkid running very briefly behind our backs causes the device to be open during the blockdev operation, causing it to fail.
Adding udevadm settle between the close and the blockdev fixed the problem for us, although this command is also inherently racy (what happens if it runs before the kernel has sent a message to udev?)
Well, tools should not believe they are the only ones running on the system. Maybe a “retry” in case of EBUSY should be on their agenda
Actually in the libguestfs appliance case, we very much can assume we’re the only one running, except as it happens for udev.