This was unexpected:
Write something to a partition device (eg. /dev/vda1) and immediately call
blockdev --rereadpt /dev/vda to re-read the partition table of the whole device. Sometimes (about 50% for me) the blockdev command fails with:
blockdev: BLKRRPART: Device or resource busy
Nothing else is using
/dev/vda, nothing from it was mounted, and the error was intermittent which indicates a race condition.
Why this happens:
udev has a rule that runs
blkid -o udev -p /dev/vda1. It does this every time you close a block device so that blkid can rescan the content of the device.
The act of blkid running very briefly behind our backs causes the device to be open during the blockdev operation, causing it to fail.
udevadm settle between the close and the blockdev fixed the problem for us, although this command is also inherently racy (what happens if it runs before the kernel has sent a message to udev?)