Disk detach is unsuccessfull while the guest is still booting
Software environment
Host:
- Operating system: Debian sid
- Architecture: x86_64
- kernel version: 5.17.0-1-amd64 #1 (closed) SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux
- libvirt version: 8.2.0-1
- Hypervisor and version: qemu-system-x86_64 1:7.0+dfsg-1
Guest:
- Operating system: Ubuntu 22.04 (cloud image)
Description of problem
If a disk is attached to a guest while the guest OS is still booting, then such disk cannot be detached from the guest. It seems that the detach succeeds from virsh perspective. But the disk is still visible both from the guest and from virsh as being attached. However when the detach is retried it fails with "error: device not found: no target device".
I would expect that either: a) the original attach fails and the disk is not attached to the guest as it is too early to attach it b) both the attach and the detach succeeds and the disk is detached.
This was observed in OpenStack upstream CI with cirros 0.5.2 guest OS. But now reproduced without OpenStack with a more normal guest (Ubuntu 22.04). The OpenStack bug is being worked around by changing the test to wait until the guest is fully booted before trying to attach the volume.
Steps to reproduce
See the reproducer commands and outputs in reproduction-logs.txt. Look for human comments prefixed with ###
- Modify the Ubuntu cloudguest image to have the boot_delay=100 added to the kernel args to simulate a slowly host
- Start the Ubuntu domain and connect to the serial console to see it boot
- Wait until the first messages appear in the console. This is around T+50sec from the virsh start. But note that the guest boot is slowed down with the boot_delay=100 kernel arg.
- From a second terminal attach an additional disk to the guest. It succeeds.
- Wait a second
- Detach the additional disk from the guest. The virsh command hangs for couple of seconds, but then succeeds.
- Check the domain XML, the disk is still attached
- Check the lsblk command from the guest (after it is fully booted). The disk is still attached.
- Check the virsh domblklist output. The disk is still attached.
- Try to detach the disk again. It fails with "error: device not found: no target device"
Additional information
- The domain XML used to start the guest: test-ubuntu.xml
- A full reproduction with command outputs: reproduction-logs.txt