Qemu increased memory usage with TCG

Host environment

• Operating system: Ubuntu 21.04, Debian 11, CentOS 8 Stream
• OS/kernel version: 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux
• Architecture: x86
• QEMU flavor: qemu-system-x86_64
• QEMU version: QEMU emulator version 5.2.0 (Debian 1:5.2+dfsg-11+deb11u1)
• QEMU command line:

  /usr/bin/qemu-system-x86_64 -name guest=instance-0000000b,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-11-instance-0000000b/master-key.aes -machine pc-i440fx-5.2,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram -cpu qemu64 -m 256 -object memory-backend-ram,id=pc.ram,size=268435456 -overcommit mem-lock=off -smp 1,sockets=1,dies=1,cores=1,threads=1 -uuid a5004daf-4ca0-43ff-ba61-83b9d2ff1896 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=24.1.0,serial=a5004daf-4ca0-43ff-ba61-83b9d2ff1896,uuid=a5004daf-4ca0-43ff-ba61-83b9d2ff1896,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=35,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -blockdev {"driver":"file","filename":"/opt/stack/data/nova/instances/_base/55f40b77aa962d16128a0139ad0743651b51251e","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -blockdev {"driver":"file","filename":"/opt/stack/data/nova/instances/a5004daf-4ca0-43ff-ba61-83b9d2ff1896/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"} -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on -netdev tap,fd=40,id=hostnet0 -device virtio-net-pci,host_mtu=1450,netdev=hostnet0,id=net0,mac=fa:16:3e:ac:1f:58,bus=pci.0,addr=0x3 -add-fd set=2,fd=42 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:2 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -device vmcoreinfo -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
  

Emulated/Virtualized environment

• Operating system: Linux / Cirros 0.5.2
• OS/kernel version: Ubuntu 5.3.0-26.28~18.04.1-generic 5.3.13
• Architecture: x86

Description of problem

The issue is that instances that are supposed to use only a small amount of memory (like 256MB) suddenly use a much higher amount of RSS when running the accel=tcg, around 512MB in the above example. This was not happening with qemu-4.2 (on Ubuntu 20.04). This is also not happening when using accel=kvm instead. The issue has been first noticed on Debian 11 (Bullseye) with the versions above, but it is happening in the same way on Centos 8 Stream, Ubuntu 21.10 and a pre-release version of Ubuntu 22.04. It also also seen when testing with qemu-6.1 built from source.

Steps to reproduce

1. Deploy devstack (https://opendev.org/openstack/devstack) with VIRT_TYPE=qemu on a VM
2. Start an instance with cirros image and a flavor allocating 256MB
3. Do a ps and see a RSS size of about 512MB being used after the instance has finished booting
4. Expected result (seen with qemu-4.2 or VIRT_TYPE=kvm): RSS stays < 256MB

Additional information

I can try to find a smaller commandline for manual reproduction if needed. The above sample is generated by OpenStack Nova via libvirt.