func-aarch64-aarch64_rme_virt function test hangs especially when built with --enable-debug

changed milestone to %10.0

added kindBug label

changed the description

added Tests accel: TCG target: arm labels

I've been able to reproduce it using rr chaos mode.

EFI stub: Booting Linux Kernel...
---> **Crash here**
SetMemoryAttributes: BaseAddress == 0x67E20000, Length == 0x1E10000, 
Attributes == 0x20000
SetMemoryAttributes: BaseAddress == 0x69C30000, Length == 0xF00000, 
Attributes == 0x4000
EFI stub: Using DTB from configuration table

Taking exception 4 [Data Abort] on CPU 0
...from EL2 to EL2
...with ESR 0x25/0x96000068
...with FAR 0x423c0000
...with SPSR 0x20000309
...with ELR 0xbf526498
...to EL2 PC 0xbeb0d200 PSTATE 0x23c9


Synchronous Exception at 0x00000000BF526498
Synchronous Exception at 0x00000000BF526498

Thread 5 hit Breakpoint 5, raise_exception (env=0x55644e97a7d0, excp=4, syndrome=2516582504, target_el=2) at ../target/arm/tcg/op_helper.c:50
50          CPUState *cs = env_cpu(env);
(rr) bt
#0  raise_exception (env=0x55644e97a7d0, excp=4, syndrome=2516582504, target_el=2) at ../target/arm/tcg/op_helper.c:50
#1  0x000055641e098963 in arm_deliver_fault (cpu=0x55644e977c00, addr=1111228416, access_type=MMU_DATA_STORE, mmu_idx=6, fi=0x272538b87950) at ../target/arm/tcg/tlb_helper.c:262
#2  0x000055641e098d45 in arm_cpu_tlb_fill_align (cs=0x55644e977c00, out=0x272538b87a40, address=1111228416, access_type=MMU_DATA_STORE, mmu_idx=6, memop=1284, size=16, probe=false, ra=134038978755970)
    at ../target/arm/tcg/tlb_helper.c:361
#3  0x000055641e2ba831 in tlb_fill_align (cpu=0x55644e977c00, addr=1111228416, type=MMU_DATA_STORE, mmu_idx=6, memop=1284, size=16, probe=false, ra=134038978755970) at ../accel/tcg/cputlb.c:1241
#4  0x000055641e2bd9ab in mmu_lookup1 (cpu=0x55644e977c00, data=0x272538b87bc0, memop=1284, mmu_idx=6, access_type=MMU_DATA_STORE, ra=134038978755970) at ../accel/tcg/cputlb.c:1652
#5  0x000055641e2bddb2 in mmu_lookup (cpu=0x55644e977c00, addr=1111228416, oi=20550, ra=134038978755970, type=MMU_DATA_STORE, l=0x272538b87bc0) at ../accel/tcg/cputlb.c:1755
#6  0x000055641e2c0de2 in do_st16_mmu (cpu=0x55644e977c00, addr=1111228416, val=1473420588432251469, oi=20550, ra=134038978755970) at ../accel/tcg/cputlb.c:2816
#7  0x000055641e2c17d0 in helper_st16_mmu (env=0x55644e97a7d0, addr=1111228416, val=1473420588432251469, oi=20550, retaddr=134038978755970) at ../accel/tcg/ldst_common.c.inc:114
#8  0x000079e861b6ff89 in code_gen_buffer ()
#9  0x000055641e2aea1f in cpu_tb_exec (cpu=0x55644e977c00, itb=0x18d049d9ef40, tb_exit=0x272538b88220) at ../accel/tcg/cpu-exec.c:454
#10 0x000055641e2af7f7 in cpu_loop_exec_tb (cpu=0x55644e977c00, tb=0x18d049d9ef40, pc=3086307396, last_tb=0x272538b88230, tb_exit=0x272538b88220) at ../accel/tcg/cpu-exec.c:904
#11 0x000055641e2afb76 in cpu_exec_loop (cpu=0x55644e977c00, sc=0x272538b882b0) at ../accel/tcg/cpu-exec.c:1018
#12 0x000055641e2afc3f in cpu_exec_setjmp (cpu=0x55644e977c00, sc=0x272538b882b0) at ../accel/tcg/cpu-exec.c:1035
#13 0x000055641e2afcd8 in cpu_exec (cpu=0x55644e977c00) at ../accel/tcg/cpu-exec.c:1061
#14 0x000055641e4ea9b5 in tcg_cpu_exec (cpu=0x55644e977c00) at ../accel/tcg/tcg-accel-ops.c:81
#15 0x000055641e4eb51f in mttcg_cpu_thread_fn (arg=0x55644e977c00) at ../accel/tcg/tcg-accel-ops-mttcg.c:94
#16 0x000055641e5cc419 in qemu_thread_start (args=0x55644ea26a90) at ../util/qemu-thread-posix.c:541
#17 0x0000508e32a8e1f5 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#18 0x0000508e32b0db00 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

#0  get_phys_addr_gpc (env=0x55644e97a7d0, ptw=0x272538b878b0, address=1111228416, access_type=MMU_DATA_STORE, memop=1284, result=0x272538b87980, fi=0x272538b87950) at ../target/arm/ptw.c:3534
$ p *fi
type = ARMFault_GPCFOnOutput

granule_protection_check
gpi | case 0b1011 | pspace != (gpi & 3)
We fallback in case:
GPCF_Fail

pspace == ARMSS_NonSecure
gpi.space == ARMSS_Realm

After reproducing a RME debug stack, exception happens in EDK2, when loading Linux Kernel. Original page protection is setup by TF-A:

#0  fill_desc (l1=<optimized out>, l1_desc=385, cnt=<optimized out>) at lib/gpt_rme/gpt_rme.c:192
#1  0x000000000e095850 in fill_l1_cont_desc (gpi=<optimized out>, length=12566528, first=<optimized out>, l1=<optimized out>) at lib/gpt_rme/gpt_rme.c:758
#2  fill_l1_tbl (gpi=8, last=249540608, first=<optimized out>, l1=0xee00000) at lib/gpt_rme/gpt_rme.c:868
#3  generate_l0_tbl_desc (pas=0xe0a3078 <pas_regions+24>) at lib/gpt_rme/gpt_rme.c:980
#4  gpt_init_pas_l1_tables (pgs=pgs@entry=GPCCR_PGS_4K, l1_mem_base=l1_mem_base@entry=249561088, l1_mem_size=l1_mem_size@entry=2097152, pas_regions=pas_regions@entry=0xe0a3060 <pas_regions>,
    pas_count=pas_count@entry=6) at lib/gpt_rme/gpt_rme.c:1267
#5  0x000000000e0927a0 in bl31_plat_gpt_setup () at plat/qemu/common/qemu_bl31_setup.c:199
#6  bl31_plat_arch_setup () at plat/qemu/common/qemu_bl31_setup.c:239
#7  0x000000000e09295c in bl31_setup (arg0=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>) at bl31/bl31_main.c:105
#8  0x000000000e0900e4 in bl31_entrypoint () at bl31/aarch64/bl31_entrypoint.S:79

Stack trace (given by EDK2 debug build):

Synchronous Exception at 0x00000000BF2908F8
PC 0x0000BF2908F8 (0x0000BF26F000+0x000218F8) [ 0] DxeCore.dll
PC 0x0000B787F2F0
PC 0x0000B7885A7C
PC 0x0000B78848B0
PC 0x0000B787D698
PC 0x0000BF276550 (0x0000BF26F000+0x00007550) [ 0] DxeCore.dll
PC 0x0000BE41C7B0 (0x0000BE414000+0x000087B0) [ 1] BdsDxe.dll
PC 0x0000BF27A0D0 (0x0000BF26F000+0x0000B0D0) [ 2] DxeCore.dll

[ 0] /home/user/.work/qemu-rme-stack/virt/build_debug/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 1] /home/user/.work/qemu-rme-stack/virt/build_debug/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
[ 2] /home/user/.work/qemu-rme-stack/virt/build_debug/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

Address triggering the page fault is randomly changing

/home/user/.work/qemu/run/tmp.8EpHO8v6uQ/int.log-...with FAR 0x41360000
/home/user/.work/qemu/run/tmp.9af0Jlric2/int.log-...with FAR 0x423b0000
/home/user/.work/qemu/run/tmp.ATzQQtMRAC/int.log-...with FAR 0x41370000
/home/user/.work/qemu/run/tmp.BlQZjL2DPG/int.log-...with FAR 0x40ab0000
/home/user/.work/qemu/run/tmp.bWWIO4ryux/int.log-...with FAR 0x41f10000
/home/user/.work/qemu/run/tmp.CvYiT7UbyT/int.log-...with FAR 0x424b0000
/home/user/.work/qemu/run/tmp.FbtNZjg4ke/int.log-...with FAR 0x40a60000
/home/user/.work/qemu/run/tmp.Qxa5BYCzPx/int.log-...with FAR 0x40440000
/home/user/.work/qemu/run/tmp.rzpXz4G0mO/int.log-...with FAR 0x408e0000
/home/user/.work/qemu/run/tmp.tsXhuEPICF/int.log-...with FAR 0x42830000
/home/user/.work/qemu/run/tmp.UNzudaLYJI/int.log-...with FAR 0x40210000
/home/user/.work/qemu/run/tmp.xNnPRgP5r8/int.log-...with FAR 0x41940000
/home/user/.work/qemu/run/tmp.YU2eOEdHGm/int.log-...with FAR 0x41710000

Adding nokaslr to kernel command line solves the issue.

Adding more memory will reduce probability to trigger the issue, but does not solve it.

For the reason behind the issue, Mathieu Poirier has some insights on it:

The problem is with the way the RME enablement was done on Virt in the TF-A.
On Virt, the DTB is installed at the very bottom of the RAM by QEMU.
The RMM is installed right after the DTB and the GPT is configured to protect the RMM area.
The RMM area is also added to the DTB as a reserved memory so that the kernel doesn't touch it.
All that is happening in the TF-A.
On Virt, EDK2 doesn't read the DTB and assumes all the RAM is available.
What I think is happening is that EDK2 tries to use the area marked as reserved for the RMM, resulting in a fault being triggered.  

Things on SBSA are different and I'm pretty sure you don't see that problem - ever.

We've been meaning to fix the Virt platform to align with what is done on SBSA for a long time,
but it is a lot of work and other activities have been prioritized.
Right now I don't see a real fix for this, but if disabling kaslr reduced the amount of failures you see,
I'd go with that.

The issue will be solved by adding nokaslr to test kernel command line.

For future work, a replayable QEMU test case can be easily created with: https://github.com/pbo-linaro/qemu/commit/rme_boot_fail/

Proposed fix: https://lore.kernel.org/qemu-devel/20250328183816.2687925-1-pierrick.bouvier@linaro.org/

added workflowPatch available label

func-aarch64-aarch64_rme_virt function test hangs especially when built with --enable-debug

Host environment

Emulated/Virtualized environment

Description of problem

Steps to reproduce

Additional information

Child items ...

Activity