ivshmem: ivshmem-doorbell can't notify the MSI-X interrupt on Arm64 guest
Host environment
- Platform: NVIDIA Jetson AGX Xavier
- Operating system: (Ubuntu 20.04 for Tegra)
- OS/kernel version: (Linux nvidia-desktop 5.10.65-tegra)
- Architecture: (ARM64)
- QEMU flavor: (qemu-aarch64)
- QEMU version: (QEMU emulator version 7.0.0)
- QEMU command line:
qemu-system-aarch64 -M virt -cpu host -smp 2 -m 4096M -hda vm1.img -boot d -monitor none -bios QEMU_EFI.fd -device ivshmem-doorbell,vectors=2,chardev=fd-doorbell -chardev socket,path=/tmp/ivshmem_socket,id=fd-doorbell
Emulated/Virtualized environment
- Operating system: (ubuntu-20.04.4-live-server-arm64)
- OS/kernel version: (Linux 5.4.0.124-generic)
- Architecture: (ARM64)
Description of problem
I init several qemu-kvm VMs on my arm64 host, which is a NVIDIA Xavier board. I want to use qemu's ivshmem-doorbell to build a sync shared memory communition with its MSI-X interrupt mechanism. I init the ivshmem-server and ivshmem-client on the host first, after then init the guests. The visul PCI-e device named "Inter-VM shared memory" can be successfully seen in my guests with command "lspci". I write a driver for this pci-e device to request and handle the MSI-X interrupts, which init well in the guest and can ring or receive from an interrupt vector on other peerID with the driver's IOCTL interface, the peer that receive vector in my environment is the ivshmem-client. However, when i use the ivshmem-client command "int" to ring my guest , the guest can't receive the msi-x interrupt notification.
Steps to reproduce
- init ivshmem-server on the host, with command "ivshmem-server -l 4M -M fg-doorbell -n 8 -F -v".
- init ivshmem-client on the host, with command "ivshmem-client -v".
- init the qemu-kvm VM .
- init the driver with "insmod" in guest to request the msi-x interrupt, while "cat /proc/interrupts" shows the interrupt request successfully!
- on host, ivshmem-client use command "int 1 0" to ring the guest's interrupt trigger, however ,nothing happened.
Additional information
I am fully sure that there is no problem about the driver I wrote for the pci-e inter-VM shared memory device, for i has tested that the driver works on my X86 PC, where I deployed qemu-x86 VMs and the driver can work well in X86 guests with the inshmem-doorbell mechanism. The ivshmem-client work on host can notify the guest to trigger the correct msix-x interrupt. Therefore, I digged the msi-x interrupt structure and use devmem tool to write the data to the messageAddress manually, which can correctly trigger the msi-x interrupt in my arm64 guest in the Xavier board, meaning the msi-x interrupt is OK in the guest. So I doubt maybe there is any issue on the ivshmem-doorbell mechanism that ring a interrupt vector in the guset of qemu-aarch64.