Skip to content

On Windows, qcow2 is corrupted on expansion

priority: high
reproducibility: easy

This issue is important because qcow2 is qemu-project's home-base virtual-disk-format. The performance and reliability of this format should set the gold-standard for virtual disks. It is a bit disappointing that a reliability bug exists within the home-base format.

No matter what fancy feature qemu-project adds to qemu and qcow2, value is compromised if bug concerning data-integrity like this remains unresolved for huge length of time.

Its possible that this bug may also exist in Linux, but does not surface as easily. Maybe also exists in other virtual-disk formats, if sufficiently stress tested.

I had much hesitancy in filing this bug, because I would think qcow2 would be very robust. There are many bugs in the area, and I have been trying to separate what what could be different issues. The fact that there are so many changes in core parts of kernel-5.15 (io_uring, nvmem, folios, ntfs3, etc) which potentially could introduce their own share of bugs. It helps to have independent reproduction of this issue on a different system.

Host environment

  • Operating system: Windows 10 21H2
  • OS/kernel version: 19044.1415
  • Architecture: x86_64
  • QEMU flavor: qemu-system-x86_64.exe
  • QEMU version: 
      C:\Users\gana>qemu-system-x86_64 --version
  QEMU emulator version 6.2.0 (v6.2.0-11889-g5b72bf03f5-dirty)
  Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
  • QEMU command line: 
    C:\WINDOWS\system32>C:\vol\scoop_01\scoopg\shims\qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35,kernel-irqchip=off" -accel whpx -smp "sockets=1,cores=8,threads=1" -boot d -cdrom E:\transcend\Fedora-Workstation-Live-x86_64-Rawhide-20220111.n.1.iso -hda H:\gkpics01.qcow2  -hdb E:\test\sgdata.vhdx  -display gtk -vga virtio -rtc base=utc -netdev user,id=vmnic1,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15 -device virtio-net,netdev=vmnic1

Emulated/Virtualized environment

  • Operating system: Fedora Rawhide (Fedora-Workstation-Live-x86_64-Rawhide-20220111.n.1.iso)
  • OS/kernel version: kernel-5.16.0-60
  • Architecture: x86_64

Description of problem

On Windows, the qcow2 loses blocks on account of which the filesystem withing is corrupted as data is copied to it, just the same way as in #727 (closed) VHDX is corrupted on expansion on both Linux/Windows.

After filing a bug for WNBD https://github.com/cloudbase/wnbd/issues/63 , I was suggested to try raw and qcow2. In the process I found that qcow2 is also affected. But it is also true that the kernel-5.15.4 ... 5.15.13 series have also been buggy https://bugzilla.kernel.org/show_bug.cgi?id=215460 . On Linux, qcow2 never showed any signs of corruption. On Windows, however, qcow2 does corrupt.

It is possible that, as Linux is so much more efficient at files and disk-IO, the kernel-block-code, qemu-block-code and qemu-qcow2-code do not hit the bug, and so the corruption does not show up as easily in Linux. Windows, being a little slower at this, might be causing the bug to show up in this qcow2 test. Possibly, the issue more likely to show up on slower machines. I am using an 2013-era intel-4rth gen i7-4700mq Haswell machine.

It is possible that, the resolution for this issue and that for #727 (closed) could be the same or very closely related. The bug may not be in qcow2.c or vhdx.c but maybe in the qemu/block subsystem. If the data-block that arrives from the VM-interface/nbd-interface which has to be written to file, but never gets to the virtual-disk code, not allocated and written to, then the data-block is lost.

Steps to reproduce

  1. Prepare virtual-disk1 as empty qcow2. In my-setup, the qcow2 file resides on an 150 GiB ExFAT partition on 512 GiB SSD. I use ExFAT as the ExFAT-filesystem does not have a concept of sparse files, eliminating that factor from troubleshooting. qemu-img.exe create -f qcow2 H:\gkpics01.qcow2 99723771904
  2. Prepare virtual-disk2 VHDX with synthetic generated data (sgdata). Scriptlets to recreate sgdata are described in #727 (comment 739930694) . In my-setup, the vhdx file resides on an 1 TiB NTFS partition on a 2 TiB HDD.
  3. Start qemu with arguments as given above.
  4. Inside VM, boot and bringup livecd desktop, close the installer and open a terminal
  5. Use gdisk to put an ext4 partition on /dev/sda
  6. Put ext4 partition on sda1 mkfs.ext4 -L fs_gkpics01 /dev/sda1
  7. Create mount directories mkdir /mnt/a /mnt/b
  8. Mount the empty partition from virtual-disk-1 mount -t ext4 /dev/sda1 /mnt/a
  9. Mount the sgdata partition from virtual-disk-2 mount.ntfs-3g /dev/sdb2 /mnt/b or mount -t ntfs3 /dev/sdb2 /mnt/b
  10. Keep a terminal tab open with dmesg -w running
  11. Rsync sgdata ( sdate=`date` ; cd /mnt/b ; rsync -avH ./photos001 /mnt/a | tee /tmp/rst.txt ; echo $sdate ; date )
  12. Check sha256sum ( sdate=`date` ; cd /mnt/a/photos001 ; shas256sum -c ./find.CHECKSUM --quiet ; echo $sdate ; date )
    corruption will show even without needing to unmount-remount or reboot-remount.
  • About 1.4 GiB free-space left on the ext4 partition.
  • Compared to #727 (closed), The number of files corrupted are less sha256sum: WARNING: 31 computed checksums did not match
  • After, VM guest OS warm reboot, a recheck of the sha256sum shows the same 31 files as corrupted
  • After, qemu poweroff, restart qemu, VM guest OS cold boot, a recheck of the sha256sum shows the same 31 files as corrupted
  • df shows: sda1 has 95271336 1k-blocks, of which 88840860 are used, 1544820 available, 99% used. The numbers don't add up. Either file-blocks are lost in lost-clusters or the ext4-filesystem has a large journal or the file-system-metadata is too large, or the ext4-filesystem has large cluster-size which results in inefficient space usage.
  • An unmount /dev/sda1 ; fsck -y /dev/sda1 ; mount -t ext4 /dev/sda1 /mnt/a did not find any lost clusters.

The reason I don't think this is a kernel bug, is because the raw-file as virtual-disk-1 doesn't show this issue. Also, it happens regardless of whether sgdata is on ntfs-3g or ntfs3-paragon.

Logs

qcow2 file growing in size as it fills up:

C:\Users\gana>qemu-img.exe create -f qcow2 H:\gkpics01.qcow2  99723771904
Formatting 'H:\gkpics01.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=99723771904 lazy_refcounts=off refcount_bits=16

C:\Users\gana>dir H:
 Volume in drive H is CPERF0
 Volume Serial Number is F196-DB9E

 Directory of H:\

01/12/2022  10:03 PM           198,144 gkpics01.qcow2
               1 File(s)        198,144 bytes
               0 Dir(s)  156,183,953,408 bytes free

C:\Users\gana>dir H:
 Volume in drive H is CPERF0
 Volume Serial Number is F196-DB9E

 Directory of H:\

01/12/2022  10:03 PM           198,144 gkpics01.qcow2
               1 File(s)        198,144 bytes
               0 Dir(s)  156,183,953,408 bytes free

C:\Users\gana>dir H:
 Volume in drive H is CPERF0
 Volume Serial Number is F196-DB9E

 Directory of H:\

01/12/2022  10:05 PM           524,288 gkpics01.qcow2
               1 File(s)        524,288 bytes
               0 Dir(s)  156,179,759,104 bytes free

C:\Users\gana>dir H:
 Volume in drive H is CPERF0
 Volume Serial Number is F196-DB9E

 Directory of H:\

01/12/2022  10:05 PM       549,388,288 gkpics01.qcow2
               1 File(s)    549,388,288 bytes
               0 Dir(s)  155,634,499,584 bytes free

C:\Users\gana>dir H:
 Volume in drive H is CPERF0
 Volume Serial Number is F196-DB9E

 Directory of H:\

01/12/2022  10:49 PM    93,117,218,816 gkpics01.qcow2
               1 File(s) 93,117,218,816 bytes
               0 Dir(s)  63,066,210,304 bytes free

C:\Users\gana>qemu-system-x86_64 --version
QEMU emulator version 6.2.0 (v6.2.0-11889-g5b72bf03f5-dirty)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

Additional information

Tests

  • There was no guestVM-kernel-traceback and corruption did happen on Win10-21H2-19044-1415/WHPX/ExFAT/VM-qemu-6.2.0/Fedora-Workstation-Live-x86_64-Rawhide-20220111.n.1.iso/kernel-5.16.0-60/qcow2/ext4 with sgdata on ntfs3/vhdx
  • There was no guestVM-kernel-traceback and corruption did happen on Win10-21H2-19044-1415/WHPX/ExFAT/VM-qemu-6.2.0/Fedora-Workstation-Live-x86_64-Rawhide-20220111.n.1.iso/kernel-5.16.0-60/qcow2/ext4 with sgdata on ntfs-fuseblk/vhdx
  • There was no guestVM-kernel-traceback and corruption did not happen on Win10-21H2-19044-1415/WHPX/ExFAT/VM-qemu-6.2.0/Fedora-Workstation-Live-x86_64-Rawhide-20220111.n.1.iso/kernel-5.16.0-60/raw/ext4 with sgdata on ntfs3/vhdx
  • There was no guestVM-kernel-traceback and corruption did not happen on Win10-21H2-19044-1415/WHPX/ExFAT/VM-qemu-6.2.0/Fedora-Workstation-Live-x86_64-Rawhide-20220111.n.1.iso/kernel-5.16.0-60/raw/ext4 with sgdata on ntfs-fuseblk/vhdx
Edited by GK
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information