qemu-6.1.0 causes I/O errors in VMs leading to data corruption
see https://bugs.gentoo.org/815379
Description of problem
after upgrading around 10 gentoo hosts from qemu-6.0.0-r53 to 6.1.0 most VMs (around 85 of 100, our VMs with PostgreSQL have 100% chance of hitting this) after some time (few minutes) will have I/O Errors, causing crashes and data corruption. The VMs are stored on ZFS volumes. Downgrading to qemu-6.0.0-r53 instantly fixes this. Happens on completely different hardware (quad core Xeons to 32C Epyc2).
Reproducible: Always
Steps to Reproduce:
- upgrade qemu-6.1.0
- keep gentoo VMs on ZFS running for some time
- VMs will have I/O errors Actual Results:
[ 1503.559878] blk_update_request: I/O error, dev vda, sector 23056464 op 0x1:(WRITE) flags 0x4800 phys_seg 254 prio class 0
[ 1503.559881] blk_update_request: I/O error, dev vda, sector 23058496 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[ 1503.559883] EXT4-fs warning (device vda): ext4_end_bio:342: I/O error 10 writing to inode 656425 starting block 2882314)
[ 1503.559963] blk_update_request: I/O error, dev vda, sector 23058512 op 0x1:(WRITE) flags 0x4800 phys_seg 254 prio class 0
[ 1503.559965] blk_update_request: I/O error, dev vda, sector 23060544 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[ 1503.559966] EXT4-fs warning (device vda): ext4_end_bio:342: I/O error 10 writing to inode 656425 starting block 2882570)
[ 1503.560033] blk_update_request: I/O error, dev vda, sector 23060560 op 0x1:(WRITE) flags 0x4800 phys_seg 254 prio class 0
[ 1503.560035] blk_update_request: I/O error, dev vda, sector 23062600 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[ 1503.560036] EXT4-fs warning (device vda): ext4_end_bio:342: I/O error 10 writing to inode 656425 starting block 2882827)
[ 1503.560100] blk_update_request: I/O error, dev vda, sector 23062616 op 0x1:(WRITE) flags 0x4800 phys_seg 254 prio class 0
[ 1503.560102] blk_update_request: I/O error, dev vda, sector 23064664 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[ 1503.560103] EXT4-fs warning (device vda): ext4_end_bio:342: I/O error 10 writing to inode 656425 starting block 2883086)
[ 1503.560167] blk_update_request: I/O error, dev vda, sector 23064688 op 0x1:(WRITE) flags 0x4800 phys_seg 250 prio class 0
[ 1503.560168] EXT4-fs warning (device vda): ext4_end_bio:342: I/O error 10 writing to inode 656425 starting block 2883418)
[ 1503.560237] EXT4-fs warning (device vda): ext4_end_bio:342: I/O error 10 writing to inode 656425 starting block 2883584)
configure/USE flags
emerge -pO qemu
These are the packages that would be merged, in order:
[binary U ] app-emulation/qemu-6.1.0-1::gentoo [6.0.0-r53::gentoo] USE="aio bzip2 caps curl doc fdt filecaps fuse io-uring jemalloc jpeg lzo ncurses oss pin-upstream-blobs png seccomp ssh udev vhost-net vnc xattr zstd -accessibility -alsa -capstone -debug -glusterfs -gnutls -gtk -infiniband -iscsi -jack -multipath -nfs -nls -numa -opengl -plugins -pulseaudio -python -rbd -sasl -sdl -sdl-image (-selinux) -slirp -smartcard -snappy -spice -static -static-user -systemtap -test -usb -usbredir -vde -vhost-user-fs -virgl -virtfs -vte -xen -xfs" PYTHON_TARGETS="python3_8 python3_9 -python3_10" QEMU_SOFTMMU_TARGETS="x86_64 -aarch64 -alpha -arm -avr -cris -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -nios2 -or1k -ppc -ppc64 -riscv32 -riscv64 -rx -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -xtensa -xtensaeb (-lm32%) (-moxie%) (-unicore32%)" QEMU_USER_TARGETS="-aarch64 -aarch64_be -alpha -arm -armeb -cris -hexagon -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -x86_64 -xtensa -xtensaeb" 0 KiB
qemu command line used
root.img and swap.img are ZFS volumes, but I don't think that's relevant qemu-system-x86_64 -name template5 -smp cores=4,threads=1,sockets=1 -k de -m 6G -vnc 0.0.0.0:1,lossy=on -net nic,macaddr=DE:AD:BE:EF:05:01,model=virtio,netdev=net0 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -enable-kvm -machine q35,accel=kvm -cpu host -curses -display none -vga virtio -daemonize -rtc base=localtime -kernel /mnt/data1/vm/kernel-5.14.8-gentoo -append 'root=/dev/vda' -drive aio=io_uring,media=disk,discard=on,cache=writeback,if=virtio,format=raw,file=root.img,index=0,throttling.iops-read=125000,throttling.iops-write=25000,throttling.bps-read=536870912,throttling.bps-write=104857600 -drive aio=io_uring,media=disk,discard=on,cache=unsafe,if=virtio,format=raw,file=swap.img,index=1,throttling.iops-read=200000,throttling.bps-read=838860800,throttling.iops-write=50000,throttling.bps-write=209715200 --device virtio-balloon -watchdog i6300esb -watchdog-action reset -chroot /mnt/data1/vm/_TEMPLATE -runas kvm -pidfile _TEMPLATE.pid -monitor unix:monitor,server=on,wait=off -chardev socket,path=guestagent,server=on,wait=off,id=qga0 -device virtio-serial -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
Host environment
- Operating system: Gentoo Linux
- OS/kernel version: Linux 5.10.69-gentoo #1 SMP Sun Sep 26 23:28:12 CEST 2021 x86_64 Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz GenuineIntel GNU/Linux
- Architecture: amd64
- QEMU flavor: qemu-system-x86_64
- QEMU version: 6.1.0
- QEMU command line: see above
Emulated/Virtualized environment
- Operating system: Gentoo Linux
- OS/kernel version: 5.14.8
- Architecture: amd64