test_kvm_xen_guest_novector_noapic sometimes fails
Host environment
- Operating system: Fedora 40
- Architecture: x86
- QEMU flavor: qemu-system-x86_64
- QEMU version: commit 8032c78e
- QEMU command line:
./qemu-system-x86_64 -display none -vga none -machine pc -chardev stdio,id=console -serial chardev:console -accel kvm,xen-version=0x4000a,kernel-irqchip=split -smp 2 -kernel /home/thuth/.cache/qemu/download/ec0ad7bb8c33c5982baee0a75505fe7dbf29d3ff5d44258204d6307c6fe0132a -append "printk.time=0 root=/dev/xvda console=ttyS0 xen_emul_unplug=ide-disks xen_no_vector_callback noapic" -drive file=/home/thuth/.cache/qemu/download/b11045d649006c649c184e93339aaa41a8fe20a1a86620af70323252eb29e40b,if=none,snapshot=on,format=raw,id=drv0 -device xen-disk,drive=drv0,vdev=xvda -device virtio-net-pci,netdev=unet -netdev user,id=unet,hostfwd=:127.0.0.1:0-:22
Emulated/Virtualized environment
- Operating system: Linux
- OS/kernel version: 6.3.0-rc3
- Architecture: x86
Description of problem
The test_kvm_xen_guest_novector_noapic test of tests/avocado/kvm_xen_guest.py (soon to be moved to tests/functional/test_x86_64_kvm_xen.py ) is sometimes (maybe 1 time out of 50) failing to boot to the shell prompt. The messages on the serial console are:
Linux version 6.3.0-rc3-00031-g1e760fa3596e (alex@zen) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #21 SMP PREEMPT_DYNAMIC Fri Mar 24 15:04:37 GMT 2023
Command line: printk.time=0 root=/dev/xvda console=ttyS0 xen_emul_unplug=ide-disks xen_no_vector_callback noapic
x86/fpu: x87 FPU will use FXSAVE
signal: max sigframe size: 1440
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x0000000007fdffff] usable
BIOS-e820: [mem 0x0000000007fe0000-0x0000000007ffffff] reserved
BIOS-e820: [mem 0x00000000feff8000-0x00000000feffffff] reserved
BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
NX (Execute Disable) protection: active
SMBIOS 2.8 present.
DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
Hypervisor detected: Xen HVM
Xen version 4.10.
last_pfn = 0x7fe0 max_arch_pfn = 0x400000000
x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
found SMP MP-table at [mem 0x000f5470-0x000f547f]
ACPI: Early table checksum verification disabled
ACPI: RSDP 0x00000000000F5290 000014 (v00 BOCHS )
ACPI: RSDT 0x0000000007FE237F 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001)
ACPI: FACP 0x0000000007FE222B 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001)
ACPI: DSDT 0x0000000007FE0040 0021EB (v01 BOCHS BXPC 00000001 BXPC 00000001)
ACPI: FACS 0x0000000007FE0000 000040
ACPI: APIC 0x0000000007FE229F 000080 (v03 BOCHS BXPC 00000001 BXPC 00000001)
ACPI: HPET 0x0000000007FE231F 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)
ACPI: WAET 0x0000000007FE2357 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)
ACPI: Reserving FACP table memory at [mem 0x7fe222b-0x7fe229e]
ACPI: Reserving DSDT table memory at [mem 0x7fe0040-0x7fe222a]
ACPI: Reserving FACS table memory at [mem 0x7fe0000-0x7fe003f]
ACPI: Reserving APIC table memory at [mem 0x7fe229f-0x7fe231e]
ACPI: Reserving HPET table memory at [mem 0x7fe231f-0x7fe2356]
ACPI: Reserving WAET table memory at [mem 0x7fe2357-0x7fe237e]
Zone ranges:
DMA [mem 0x0000000000001000-0x0000000000ffffff]
DMA32 [mem 0x0000000001000000-0x0000000007fdffff]
Normal empty
Device empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000001000-0x000000000009efff]
node 0: [mem 0x0000000000100000-0x0000000007fdffff]
Initmem setup node 0 [mem 0x0000000000001000-0x0000000007fdffff]
On node 0, zone DMA: 1 pages in unavailable ranges
On node 0, zone DMA: 97 pages in unavailable ranges
On node 0, zone DMA32: 32 pages in unavailable ranges
ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
ACPI: Skipping IOAPIC probe due to 'noapic' option.
ACPI: Using ACPI for processor (LAPIC) configuration information
ACPI: HPET id: 0x8086a201 base: 0xfed00000
Intel MultiProcessor Specification v1.4
MPTABLE: OEM ID: BOCHSCPU
MPTABLE: Product ID: 0.1
MPTABLE: APIC at: 0xFEE00000
IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Processors: 2
smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[mem 0x08000000-0xfeff7fff] available for PCI devices
Booting paravirtualized kernel on Xen HVM
clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
setup_percpu: NR_CPUS:64 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
percpu: Embedded 44 pages/cpu s149304 r0 d30920 u1048576
Built 1 zonelists, mobility grouping on. Total pages: 31968
Kernel command line: printk.time=0 root=/dev/xvda console=ttyS0 xen_emul_unplug=ide-disks xen_no_vector_callback noapic
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 102364K/130552K available (12288K kernel code, 1699K rwdata, 3004K rodata, 1040K init, 2632K bss, 27928K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Dynamic Preempt: full
rcu: Preemptible hierarchical RCU implementation.
rcu: RCU event tracing is enabled.
rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16
xen:events: Using 2-level ABI
rcu: srcu_init: Setting srcu_struct sizes based on contention.
Console: colour *CGA 80x25
Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
printk: console [ttyS0] enabled
ACPI: Core revision 20221020
ACPI: setting ELCR to 0200 (from 0c00)
clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
APIC: Switch to symmetric I/O mode setup
Not enabling interrupt remapping due to skipped IO-APIC setup
tsc: Unable to calibrate against PIT
tsc: using HPET reference calibration
tsc: Detected 2496.010 MHz processor
clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x23fa80a809f, max_idle_ns: 440795273818 ns
Calibrating delay loop (skipped), value calculated using timer frequency.. 4992.02 BogoMIPS (lpj=2496010)
pid_max: default: 32768 minimum: 301
LSM: initializing lsm=capability,yama,integrity,selinux
Yama: becoming mindful.
SELinux: Initializing.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Spectre V2 : Kernel not compiled with retpoline; no mitigation available!
Spectre V2 : Vulnerable
Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Speculative Store Bypass: Vulnerable
MDS: Vulnerable: Clear CPU buffers attempted, no microcode
MMIO Stale Data: Unknown: No mitigations
Freeing SMP alternatives memory: 24K
APIC timer disabled due to verification failure
smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0xf, model: 0x6b, stepping: 0x1)
Performance Events: unsupported Netburst CPU model 107 no PMU driver, software events only.
rcu: Hierarchical SRCU implementation.
rcu: Max phase no-delay instances is 400.
NMI watchdog: Perf NMI watchdog permanently disabled
smp: Bringing up secondary CPUs ...
x86: Booting SMP configuration:
.... node #0, CPUs: #1
smp: Brought up 1 node, 2 CPUs
smpboot: Max logical packages: 1
smpboot: Total of 2 processors activated (9984.04 BogoMIPS)
devtmpfs: initialized
x86/mm: Memory block size: 128MB
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
futex hash table entries: 512 (order: 3, 32768 bytes, linear)
PM: RTC time: 12:02:16, date: 2024-12-19
NET: Registered PF_NETLINK/PF_ROUTE protocol family
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1734609736.239:1): state=initialized audit_enabled=0 res=1
thermal_sys: Registered thermal governor 'step_wise'
thermal_sys: Registered thermal governor 'user_space'
cpuidle: using governor ladder
cpuidle: using governor menu
PCI: Using configuration type 1 for base access
HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
cryptd: max_cpu_qlen set to 1000
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ACPI: Interpreter enabled
ACPI: PM: (supports S0 S3 S5)
ACPI: Using PIC for interrupt routing
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
PCI: Using E820 reservations for host bridge windows
ACPI: Enabled 2 GPEs in block 00 to 0F
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3]
acpi PNP0A03:00: PCIe port services disabled; not requesting _OSC control
acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended configuration space under this bridge
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
pci_bus 0000:00: root bus resource [mem 0x08000000-0xfebfffff window]
pci_bus 0000:00: root bus resource [mem 0x100000000-0x17fffffff window]
pci_bus 0000:00: root bus resource [bus 00-ff]
pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f]
pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB
pci 0000:00:02.0: [5853:0001] type 00 class 0xff8000
pci 0000:00:02.0: reg 0x10: [io 0xc000-0xc0ff]
pci 0000:00:02.0: reg 0x14: [mem 0xfd000000-0xfdffffff pref]
pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
pci 0000:00:03.0: reg 0x10: [io 0xc100-0xc11f]
pci 0000:00:03.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff]
pci 0000:00:03.0: reg 0x20: [mem 0xfe000000-0xfe003fff 64bit pref]
pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref]
ACPI: PCI: Interrupt link LNKA configured for IRQ 10
ACPI: PCI: Interrupt link LNKB configured for IRQ 10
ACPI: PCI: Interrupt link LNKC configured for IRQ 11
ACPI: PCI: Interrupt link LNKD configured for IRQ 11
ACPI: PCI: Interrupt link LNKS configured for IRQ 9
xen:balloon: Initialising balloon driver
iommu: Default domain type: Translated
iommu: DMA domain TLB invalidation policy: lazy mode
SCSI subsystem initialized
ACPI: bus type USB registered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
Advanced Linux Sound Architecture Driver Initialized.
PCI: Using ACPI for IRQ routing
hpet: 3 channels of 0 reserved for per-cpu timers
clocksource: Switched to clocksource tsc-early
FS-Cache: Loaded
pnp: PnP ACPI init
pnp: PnP ACPI: found 6 devices
NET: Registered PF_INET protocol family
IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
tcp_listen_portaddr_hash hash table entries: 128 (order: 0, 4096 bytes, linear)
Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear)
TCP bind hash table entries: 1024 (order: 4, 65536 bytes, linear)
TCP: Hash tables configured (established 1024 bind 1024)
UDP hash table entries: 256 (order: 2, 24576 bytes, linear)
UDP-Lite hash table entries: 256 (order: 2, 24576 bytes, linear)
NET: Registered PF_UNIX/PF_LOCAL protocol family
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
pci_bus 0000:00: resource 7 [mem 0x08000000-0xfebfffff window]
pci_bus 0000:00: resource 8 [mem 0x100000000-0x17fffffff window]
pci 0000:00:01.0: PIIX3: Enabling Passive Release
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
PCI: CLS 0 bytes, default 64
kvm_intel: VMX not supported by CPU 0
workingset: timestamp_bits=46 max_order=15 bucket_order=0
squashfs: version 4.0 (2009/01/31) Phillip Lougher
fuse: init (API version 7.38)
9p: Installing v9fs 9p2000 file system support
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
io scheduler mq-deadline registered
io scheduler kyber registered
ACPI: \_SB_.LNKC: Enabled at IRQ 11
xen:xen_evtchn: Event-channel device installed
ACPI: \_SB_.LNKB: Enabled at IRQ 10
xen:grant_table: Grant tables using version 1 layout
Grant table initialized
Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
Non-volatile memory driver v1.3
ACPI: bus type drm_connector registered
loop: module loaded
Invalid max_queues (4), will use default max: 2.
tun: Universal TUN/TAP device driver, 1.6
e100: Intel(R) PRO/100 Network Driver
e100: Copyright(c) 1999-2006 Intel Corporation
e1000: Intel(R) PRO/1000 Network Driver
e1000: Copyright (c) 1999-2006 Intel Corporation.
e1000e: Intel(R) PRO/1000 Network Driver
e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
igb: Intel(R) Gigabit Ethernet Network Driver
igb: Copyright (c) 2007-2014 Intel Corporation.
igbvf: Intel(R) Gigabit Virtual Function Network Driver
igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
VMware vmxnet3 virtual NIC driver - version 1.7.0.0-k-NAPI
xen_netfront: Initialising Xen virtual ethernet driver
usbcore: registered new interface driver cdc_ether
usbcore: registered new interface driver cdc_eem
usbcore: registered new interface driver cdc_ncm
usbcore: registered new interface driver r8153_ecm
usbcore: registered new interface driver cdc_acm
cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
usbcore: registered new interface driver usb-storage
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
usbcore: registered new interface driver ch341
usbserial: USB Serial support registered for ch341-uart
usbcore: registered new interface driver cp210x
usbserial: USB Serial support registered for cp210x
usbcore: registered new interface driver ftdi_sio
usbserial: USB Serial support registered for FTDI USB Serial Device
usbcore: registered new interface driver keyspan
usbserial: USB Serial support registered for Keyspan - (without firmware)
usbserial: USB Serial support registered for Keyspan 1 port adapter
usbserial: USB Serial support registered for Keyspan 2 port adapter
usbserial: USB Serial support registered for Keyspan 4 port adapter
usbcore: registered new interface driver pl2303
usbserial: USB Serial support registered for pl2303
usbcore: registered new interface driver usb_serial_simple
usbserial: USB Serial support registered for carelink
usbserial: USB Serial support registered for zio
usbserial: USB Serial support registered for funsoft
usbserial: USB Serial support registered for flashloader
usbserial: USB Serial support registered for google
usbserial: USB Serial support registered for libtransistor
usbserial: USB Serial support registered for vivopay
usbserial: USB Serial support registered for moto_modem
usbserial: USB Serial support registered for motorola_tetra
usbserial: USB Serial support registered for nokia
usbserial: USB Serial support registered for novatel_gps
usbserial: USB Serial support registered for hp4x
usbserial: USB Serial support registered for suunto
usbserial: USB Serial support registered for siemens_mpi
rtc_cmos 00:05: RTC can wake from S4
rtc_cmos 00:05: registered as rtc0
rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs
fail to initialize ptp_kvm
intel_pstate: CPU model not supported
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
GACT probability NOT on
xt_time: kernel timezone is -0000
IPVS: Registered protocols (TCP, UDP)
IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
IPVS: ipvs loaded.
IPVS: [rr] scheduler registered.
Initializing XFRM netlink socket
NET: Registered PF_INET6 protocol family
Segment Routing with IPv6
In-situ OAM (IOAM) with IPv6
NET: Registered PF_PACKET protocol family
8021q: 802.1Q VLAN Support v1.8
9pnet: Installing 9P2000 support
NET: Registered PF_VSOCK protocol family
IPI shorthand broadcast: enabled
sched_clock: Marking stable (402156364, -4933103)->(420983909, -23760648)
Clockevents: could not switch to one-shot mode: lapic is not functional.
Could not switch to high resolution mode on CPU 0
Clockevents: could not switch to one-shot mode: lapic is not functional.
Could not switch to high resolution mode on CPU 1
tsc: Refined TSC clocksource calibration: 2495.955 MHz
clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x23fa4cd42c8, max_idle_ns: 440795310990 ns
clocksource: Switched to clocksource tsc
Clockevents: could not switch to one-shot mode: lapic is not functional.
Could not switch to high resolution mode on CPU 1
Clockevents: could not switch to one-shot mode: lapic is not functional.
Could not switch to high resolution mode on CPU 0
xenbus_probe_frontend: Waiting for devices to initialise: 25s...20s...15s...
random: crng init done
10s...5s...0s...Steps to reproduce
Either run the mentioned avocado/functional test, or directly the mentioned QEMU command line >= 50 times
Additional information
I think it reproduces more easily if the host machine is under load (not quite sure about it, though).
See this discussion on the mailing list for some more details:
https://lore.kernel.org/qemu-devel/999a8203f0c800f1305aacdb500dbf6038ebf147.camel@infradead.org/