<!--This is the upstream QEMU issue tracker. If you are able to, it will greatly facilitate bug triage if you attempt to reproduce the problem with the latest qemu.git master built from source. See https://www.qemu.org/download/#source for instructions on how to do this. QEMU generally supports the last two releases advertised on https://www.qemu.org/. Problems with distro-packaged versions of QEMU older than this should be reported to the distribution instead. See https://www.qemu.org/contribute/report-a-bug/ for additional guidance. If this is a security issue, please consult https://www.qemu.org/contribute/security-process/--> ## Host environment - Operating system: Linux - OS/kernel version: Linux centos7 6.6.87.2-microsoft-standard-WSL2 - Architecture: x86 - QEMU flavor: qemu-system-riscv64 - QEMU version: v11.0.0-1713-gde5d8bfd61\`, reported version 11.0.50 - QEMU command line: <!--Give the smallest, complete command line that exhibits the problem. </li> </ul> <p data-sourcepos="39:1-40:62">If you are using libvirt, virsh, or vmm, you can likely find the QEMU command line arguments in /var/log/libvirt/qemu/$GUEST.log.--&gt;</p> <pre data-sourcepos="42:3-51:5"><code>qemu-system-riscv64 \ -M virt,iommu-sys=on,aia=aplic-imsic \ -cpu rv64,smstateen=true \ -m 8G \ -trace riscv_iommu_dma \ -nographic \ -device edu,dma_mask=0xFFFFFFFFFFFFFFFF \ -bios bin/iommu_iotinval_gvma.elf </code></pre> <h2 id="user-content-emulatedvirtualized-environment" data-sourcepos="53:1-53:35">Emulated/Virtualized environment<a href="#emulatedvirtualized-environment" aria-label="Link to heading 'Emulated/Virtualized environment'" data-heading-content="Emulated/Virtualized environment" class="anchor"></a></h2> <ul data-sourcepos="55:1-63:28"> <li data-sourcepos="55:1-57:41"> <p data-sourcepos="55:3-55:19">Operating system:</p> <!--Windows 10 21H1, Fedora 37, etc.--> - OS/kernel version: <!--For POSIX guests, use `uname -a`.--> - Architecture: <!--x86, ARM, s390x, etc.--> ## Description of problem For a two-stage translation, IOTINVAL.GVMA with GV=1 and AV=1 fails to invalidate the cached translation. After a G-stage mapping is changed and IOTINVAL.GVMA is issued with the correct GSCID and the GPA in ADDR, a device using the same IOVA still receives the old SPA. The IOATC entry for a two-stage translation is keyed by the device IOVA (the first-stage input address). IOTINVAL.GVMA's ADDR is a guest physical address. The model compares the command's GPA against the cached IOVA, which are different address spaces for a two-stage translation, so the entry never matches and is not invalidated. ## Steps to reproduce 1. [iommu_iotinval_gvma.c](/uploads/54ee41209773164a9e5e0892ec775f62/iommu_iotinval_gvma.c) 2. Built with: `riscv64-unknown-elf-gcc -march=rv64gc -mabi=lp64 -mcmodel=medany -nostartfiles -nostdlib` 3. [iommu_iotinval_gvma.elf](/uploads/050a5ab797baa309e232649fda947106/iommu_iotinval_gvma.elf) ``` qemu-system-riscv64 \ -M virt,iommu-sys=on,aia=aplic-imsic \ -cpu rv64,smstateen=true \ -m 8G \ -trace riscv_iommu_dma \ -nographic \ -device edu,dma_mask=0xFFFFFFFFFFFFFFFF \ -bios bin/iommu_iotinval_gvma.elf ``` ## Additional information The guest program (`tests/iommu_iotinval_gvma.c`): 1. Builds a two-stage mapping (VS-stage Sv39 + G-stage Sv39x4) with DC.iohgatp.GSCID = 1. Maps gva_src (0x100000) to gpa_src (0x1000) to old_buf (filled with 0xAA). 2. Reads gva_src once via EDU DMA, which caches the IOVA-\>SPA translation. 3. Remaps gpa_src to new_buf (filled with 0xBB) by rewriting the G-stage leaf. 4. Issues IOTINVAL.GVMA GV=1, AV=1, GSCID=DC.iohgatp.GSCID, Addr=gpa_src through the command queue and waits for it to be consumed. 5. Reads gva_src again and checks whether the data is old (0xAA) or new (0xBB). ## Expected behaviour (per spec) After IOTINVAL.GVMA invalidates the G-stage mapping for that GSCID and GPA, the next access to the same IOVA must use the new G-stage mapping, returning new_buf (0xBB). Spec references: - RISC-V IOMMU Architecture Specification, Table 12 (IOTINVAL.GVMA operands), GV=1 AV=1: "Invalidates information cached from leaf second-stage page table entries corresponding to the guest-physical-address in ADDR operand, but only for VM address spaces identified by the GSCID operand." - Section 4.1, IOTINVAL.GVMA note: "implementations contain address-translation caches that map guest virtual addresses directly to supervisor physical addresses, removing a level of indirection. For such implementations, any entry whose guest virtual address maps to a guest physical address that matches the IOTINVAL.GVMA address and GSCID arguments must be invalidated." - Section 7.3.3 (Changing MSI / second-stage page table entry) prescribes: "IOTINVAL.GVMA with GV=AV=1, ADDR\[63:12\]=A\[63:12\] and GSCID=DC.iohgatp.GSCID". ## Observed behaviour ``` DC.iohgatp.GSCID = 0x00000001 gva_src = 0x0000000000100000 gpa_src = 0x0000000000001000 old SPA = 0x0000000080092000 new SPA = 0x0000000080098000 After first transfer: dst_buf[0] = 0x000000AA (old data, as expected) IOTINVAL.GVMA GV=1 AV=1 GSCID=0x00000001 Addr=0x0000000000001000 Invalidation command consumed. After second transfer: dst_buf[0] = 0x000000AA BUG: after IOTINVAL.GVMA the IOVA still maps to the old SPA. The cached translation was not invalidated. ``` `riscv_iommu_dma` trace (both reads of the IOVA resolve to the old SPA): ``` riscv_iommu_dma (null): translate 0000:01.0 #0 RO 0x100000 -> 0x80092000 riscv_iommu_dma (null): translate 0000:01.0 #0 RO 0x100000 -> 0x80092000 ``` Contrast: the same test with AV=0 (GSCID only) returns new data (0xBB) on the second transfer, so the invalidation path itself works. ## Root cause In `hw/riscv/riscv-iommu.c`, the GV=1/AV=1 GVMA handler matches the cached entry by IOVA: ```c static void riscv_iommu_iot_inval_gscid_iova(gpointer key, gpointer value, gpointer data) { RISCVIOMMUEntry *iot = (RISCVIOMMUEntry *) value; RISCVIOMMUEntry *arg = (RISCVIOMMUEntry *) data; if (iot->tag == arg->tag && iot->gscid == arg->gscid && iot->iova == arg->iova) { /* iot->iova is the first-stage IOVA */ iot->perm = IOMMU_NONE; } } ``` The command dispatch sets `arg->iova` from the command ADDR (a GPA): ```c hwaddr iova = (cmd.dword1 << 2) & TARGET_PAGE_MASK; /* GPA from IOTINVAL.GVMA */ ... func = (av) ? riscv_iommu_iot_inval_gscid_iova : riscv_iommu_iot_inval_gscid; riscv_iommu_iot_inval(s, func, gscid, pscid, iova, RISCV_IOMMU_TRANS_TAG_VG); riscv_iommu_iot_inval(s, func, gscid, pscid, iova, RISCV_IOMMU_TRANS_TAG_VN); ``` The cached `iot->iova` is the device IOVA (set in `riscv_iommu_translate` as `iot->iova = PPN_DOWN(iotlb->iova)`), which for a two-stage translation is the first-stage virtual address, not the GPA. Comparing the command's GPA against the cached first-stage IOVA never matches for a two-stage entry, so the entry is not invalidated. ## Notes - Effect: after changing a G-stage mapping and issuing the address-specific IOTINVAL.GVMA, a device continues to use the stale SPA. As the IOT cache does not store the GPA in case of two stage translation. The IOTINVAL.GVMA with GV = 1, and AV = 1 can't invalidate the correct entry. There are can be two possible solutions, either add the GPA to the IOT cache or invalidate the IOT based only on GSCID even for IOTINVAl.GVMA_GV1_AV1 invalidation command. <!--The line below ensures that proper tags are added to the issue. Please do not remove it.-->