You need to sign in or sign up before continuing.

guest Linux Kernel hangs and reports CPU lockup/stuck (Qemu >= 6.0.1 regression)

Host environment

Operating system: openSUSE-15.4 (Linux-5.14) or Debian-11 (Linux-5.10)
OS/kernel version: default distro kernel
Architecture: x86_64
QEMU flavor: qemu-system-x86_64
QEMU version: 6.0.1 - 7.1.0-rc2

Emulated/Virtualized environment

Operating system: openSUSE-15.4 (Linux-5.14) or Debian-11 (Linux-5.10)
OS/kernel version: default distro kernel
Architecture: x86_64

Description of problem

Since at least qemu-6.0.1 my VM guest is having CPU problems. It looks like qemu-6.0.0 is fine, but I can't confirm this 100 %.

Problem: The guest hangs for about 30 seconds and dmesg reports errors.

dmesg

[  310.791732] watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [swapper/1:0]
[  310.791753] Modules linked in: ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter bpfilter af_packet iscsi_ibft iscsi_boot_sysfs rfkill dm_crypt essiv authenc pktcdvd intel_rapl_msr intel_rapl_common kvm_intel kvm cirrus drm_kms_helper irqbypass cec pcspkr joydev rc_core syscopyarea sysfillrect sysimgblt virtio_balloon fb_sys_fops i2c_piix4 button nls_iso8859_1 nls_cp437 vfat fat drm fuse configfs ip_tables x_tables ext4 crc16 mbcache jbd2 hid_generic usbhid sd_mod t10_pi virtio_scsi virtio_net net_failover virtio_blk failover sr_mod cdrom ata_generic crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd xhci_pci xhci_pci_renesas xhci_hcd cryptd serio_raw ehci_pci uhci_hcd ehci_hcd usbcore ata_piix ahci libahci virtio_pci virtio_pci_modern_dev libata floppy qemu_fw_cfg dm_mirror dm_region_hash dm_log dm_mod sg scsi_mod
[  310.792102] Supported: Yes
[  310.792108] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.14.21-150400.22-default #1 SLE15-SP4 0b6a6578ade2de5c4a0b916095dff44f76ef1704
[  310.792121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[  310.792127] RIP: 0010:__do_softirq+0x6e/0x2bc
[  310.792146] Code: 8b 70 2c 81 60 2c ff f7 ff ff 89 74 24 14 c7 44 24 10 0a 00 00 00 48 c7 c0 c0 30 03 00 65 66 c7 00 00 00 fb 66 0f 1f 44 00 00 <bb> ff ff ff ff 41 0f bc de 83 c3 01 89 1c 24 0f 84 92 00 00 00 49
[  310.792154] RSP: 0018:ffffb9a8c00d0f98 EFLAGS: 00000206
[  310.792163] RAX: 00000000000330c0 RBX: ffffb9a8c0093e18 RCX: 0000000034b47837
[  310.792169] RDX: ffff9835c02dd100 RSI: 0000000004200042 RDI: 0000000000000040
[  310.792175] RBP: 0000000000000022 R08: ffffb9a8c0093e18 R09: 0000000000000001
[  310.792180] R10: 0000000000000002 R11: 0000000000000283 R12: 0000000000000001
[  310.792185] R13: 0000000000000000 R14: 0000000000000040 R15: 0000000000000000
[  310.792191] FS:  0000000000000000(0000) GS:ffff9836f7d00000(0000) knlGS:0000000000000000
[  310.792197] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  310.792203] CR2: 000055ed8cffbaf8 CR3: 00000001025c0001 CR4: 0000000000170ee0
[  310.792216] Call Trace:
[  310.792247]  <IRQ>
[  310.792284]  irq_exit_rcu+0x9c/0xc0
[  310.792305]  common_interrupt+0x5d/0xa0
[  310.792331]  </IRQ>
[  310.792335]  <TASK>
[  310.792339]  asm_common_interrupt+0x1e/0x40
[  310.792358] RIP: 0010:native_safe_halt+0xb/0x10
[  310.792368] Code: f0 80 48 02 20 48 8b 00 a8 08 74 82 eb c1 cc eb 07 0f 00 2d 89 f3 5f 00 f4 c3 0f 1f 44 00 00 eb 07 0f 00 2d 79 f3 5f 00 fb f4 <c3> cc cc cc cc 0f 1f 44 00 00 65 8b 15 14 ee 60 69 0f 1f 44 00 00
[  310.792375] RSP: 0018:ffffb9a8c0093ec8 EFLAGS: 00000212
[  310.792382] RAX: ffffffff96a0ca50 RBX: 0000000000000001 RCX: ffff9835c49c3700
[  310.792387] RDX: 00000000001df31e RSI: 0000000000000000 RDI: ffff9835c02a8000
[  310.792392] RBP: ffffffff97d47120 R08: 00000000001df31e R09: 0000000000029800
[  310.792397] R10: ffffb9a8c164bbe0 R11: 0000000000000198 R12: 0000000000000000
[  310.792402] R13: 0000000000000000 R14: ffffffffffffffff R15: ffff9835c02a8000
[  310.792409]  ? __sched_text_end+0x5/0x5
[  310.792425]  default_idle+0xa/0x10
[  310.792434]  default_idle_call+0x2d/0xe0
[  310.792441]  do_idle+0x1ec/0x2d0
[  310.792452]  cpu_startup_entry+0x19/0x20
[  310.792460]  start_secondary+0x11c/0x160
[  310.792475]  secondary_startup_64_no_verify+0xc2/0xcb
[  310.792501]  </TASK>

[  435.511342] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 30s!
[  435.511374] Showing busy workqueues and worker pools:
[  435.511377] workqueue events: flags=0x0
[  435.511380]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  435.511385]     pending: vmstat_shepherd
[  435.511395] workqueue events_power_efficient: flags=0x80
[  435.511398]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[  435.511402]     pending: neigh_periodic_work, neigh_periodic_work
[  435.511411] workqueue events_freezable_power_: flags=0x84
[  435.511414]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  435.511417]     in-flight: 4783:disk_events_workfn
[  435.511425] workqueue mm_percpu_wq: flags=0x8
[  435.511428]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  435.511431]     pending: vmstat_update
[  435.511440] workqueue writeback: flags=0x4a
[  435.511443]   pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3
[  435.511447]     pending: wb_workfn
[  435.511453] workqueue kblockd: flags=0x18
[  435.511455]   pwq 3: cpus=1 node=0 flags=0x0 nice=-20 active=3/256 refcnt=4
[  435.511459]     pending: blk_mq_timeout_work, blk_mq_timeout_work, blk_mq_timeout_work
[  435.511475] workqueue ata_sff: flags=0x8
[  435.511479]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/512 refcnt=2
[  435.511482]     pending: ata_sff_pio_task [libata]
[  435.511538] pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=30s workers=3 idle: 349 51

It looks like the problem mostly appears if SSH is being used over a "user" network connection. A typical situation is when editing a file in Vim (compiled with X support) via SSH and using the X clipboard ("+y"). But the problem also happens in other situations with SSH, e. g. when using SSHFS.
The type of NIC doesn't seem to make a difference (tested virtio and e1000). But "tap" network connections don't show a problem.

Git Bisect

I've tried to bisect the problem. The result:
(commits in order between 6.0.0 and 6.0.1)

Starting with f5cc5a5c a "certain VM" I use regularly gets stuck at booting (sorry, I can't share that VM here).
i386: split cpu accelerators from cpu.c, using AccelCPUClass (Claudio Fontana cfontana@suse.de)
Other VMs like the openSUSE live system is still booting fine.

d11ebe2c is the first commit where I can reproducibly let a VM become stuck.
This patch adds clipboard support to the qemu gtk ui. (Gerd Hoffmann kraxel@redhat.com)
But I guess this isn't the root cause of the problem. It only makes it happen on clipboard actions, which is the easiest way to reproduce it. But as said the problem also exists with SSHFS for example (but isn't easily reproducible with SSHFS).

Starting with 4db4385a my "certain VM" is booting again.
i386: run accel_cpu_instance_init as post_init (Claudio Fontana cfontana@suse.de)

Reproduction

This reproduces the problem based on a clipboard action (see commit f5cc5a5c above). The "clipboard thing" isn't the only situation triggering the problem, it's just the easiest to reproduce. But for example VMs also often hang when working with SSHFS between host and guest.

Download openSUSE-15.4 live system.
https://download.opensuse.org/distribution/leap/15.4/live/openSUSE-Leap-15.4-KDE-Live-x86_64-Media.iso
2022-08-13: currently it's linking to openSUSE-Leap-15.4-KDE-Live-x86_64-Build7.5-Media.iso (10-Aug-2022 02:58)
Run host with that live system and install Qemu: sudo zypper in qemu-kvm (it's Qemu-6.2.0)
Start VM with the same live system from that host:
qemu-system-x86_64 -m 4G -machine pc,accel=kvm -device virtio-net-pci,netdev=user.0 -netdev user,hostfwd=tcp:127.0.0.1:50022-:22,id=user.0 -cdrom openSUSE-Leap-15.4-KDE-Live-x86_64-Media.iso
Inside guest: Set user "linux" password, start sshd, stop firewalld and install gvim.
passwd
sudo systemctl start sshd
sudo systemctl stop firewalld
sudo zypper in gvim (installs X support for vim in openSUSE)
SSH from host into guest (important: -X): ssh -X -p50022 linux@127.0.0.1
Inside guest via SSH:
vim foobar
In vim press i, then type foobar, press ESC and then press "+y successively (copies from vim to X buffer).
-> VM will hang!

This reproduction only works if -display isn't set to something else than gtk. But as said this doesn't seem to workaround the root cause of the problem. The Gtk display is only needed for this way of reproduction.

This reproduction also works with Debian-11 (current Debian stable) as host and guest. But Debian-11 only comes with Qemu-5.2.0, so you need to compile Qemu >= 6.0.1 yourself (or install from Debian backports). And you need to run apt update && apt install openssh-server vim-gtk3 as root in the guest.

Details

Reproduced with CPUs:

i7-4700MQ (tested openSUSE-15.4 host OS)
Ryzen 5 3500U (tested Debian-11 host OS)
Ryzen 5 PRO 5650U (tested openSUSE-15.4 host OS)
Ryzen 5800X (Debian-11 host OS)

Setting -cpu to host or something old like core2duo doesn't help.

Edited Aug 20, 2022 by Moritz Duge

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information