Support running real-world Android on Arm by supporting one-register list for the POP (LDMIA) Thumb32 instruction.

Summary

Real-world Android code generated for 32-bit Arm software (even running on 64-bit Arm platforms) contains (unfortunately) Arm T32 instructions such as

ldmia.w sp!, {lr}

(0xe8bd4000)

The ldmia.w instruction has 2 parameters: a register (e.g. sp) and a register list (enclosed in braces). pop is an alias for such an instruction with sp as its first parameter. Instructions where the register list is empty or contains exactly one entry are considered unpredictable. Thus, the parameter list should contain at least 2 entries, and instructions with only 1 entry in the parameter list should not exist. Yet, they do occur in Android 11 at least (in the 32-bit zygote process), and probably in any other recent Android version using ART, too.

Real-world Arm CPUs (such as Arm Cortex-A55) seem to work fine with these "unpredictable" instructions, but the QEMU CPU (of more recent QEMU) does not. Executing such code on a QEMU Arm CPU (e.g. "cortex-a55") results in SIGILL with code ILL_ILLOPC. This is correct, as the QEMU Arm CPU works according to the specification. However, at the same time, it is incorrect, as any real Arm Cortex-A55 executes such an instruction just fine, and a QEMU user selecting a "cortex-a55" CPU in QEMU expects emulation like a real Arm Cortex-A55.

The solution is very simple. Just replace in target/arm/tcg/translate.c

    /* BitCount(list) < 2 is UNPREDICTABLE */
    return do_ldm(s, a, 2);

    /* BitCount(list) < 2 is UNPREDICTABLE, but BitCount(list)==1 is supported by each real-world Arm CPU supporting T32 */
    return do_ldm(s, a, 1);

This would make it possible to run real-world Android software for Arm CPUs on QEMU, which is currently not possible due to this bug.

Host environment

Operating system: Linux
OS/kernel version: 6.1.11
Architecture: x86_64
QEMU flavor: qemu-system-aarch64
QEMU version: 8.0.3
QEMU command line: ./qemu-system-aarch64 -machine virt-8.0,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,acpi=off -accel tcg -cpu cortex-a55 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny ...

Emulated/Virtualized environment

Operating system: LineageOS 18.1 on Waydroid 1.4.1 on PostmarketOS edge
OS/kernel version: 6.4.4-0-edge
Architecture: aarch64

Steps to reproduce

Get any aarch64 Linux on QEMU for x86_64 running. Make sure that Wayland is running. (For example, build PostmarketOS with "phosh" for aarch64 and install it.)
Install waydroid (e.g. apk add waydroid).
Install the LineageOS 18.1 for waydroid image (e.g. waydroid init).
Run the waydroid-container (e.g. rc-service waydroid-container restart).
Start the waydroid session (e.g. click on the "Waydroid" symbol on the graphical user interface).
Observe the waydroid log file (e.g. run waydroid logcat).

Additional information

The output of the Android log (using waydroid logcat) will be akin:

23908 23908 D AndroidRuntime: >>>>>> START com.android.internal.os.ZygoteInit uid 0 <<<<<<
23908 23908 I AndroidRuntime: Using default boot image
23908 23908 I AndroidRuntime: Leaving lock profiling enabled
23908 23908 E cutils-trace: Error opening trace file: No such file or directory (2)
23908 23908 I zygote  : option[0]=-Xzygote
23908 23908 I zygote  : option[1]=exit
23908 23908 I zygote  : option[2]=vfprintf
23908 23908 I zygote  : option[3]=sensitiveThread
23908 23908 I zygote  : option[4]=-verbose:gc
23908 23908 I zygote  : option[5]=-XX:PerfettoHprof=true
23908 23908 I zygote  : option[6]=-Xms8m
23908 23908 I zygote  : option[7]=-Xmx512m
23908 23908 I zygote  : option[8]=-XX:HeapGrowthLimit=192m
23908 23908 I zygote  : option[9]=-XX:HeapMinFree=8m
23908 23908 I zygote  : option[10]=-XX:HeapMaxFree=16m
23908 23908 I zygote  : option[11]=-XX:HeapTargetUtilization=0.6
23908 23908 I zygote  : option[12]=-Xusejit:true
23908 23908 I zygote  : option[13]=-Xjitsaveprofilinginfo
23908 23908 I zygote  : option[14]=-XjdwpOptions:suspend=n,server=y
23908 23908 I zygote  : option[15]=-XjdwpProvider:default
23908 23908 I zygote  : option[16]=-Xopaque-jni-ids:swapable
23908 23908 I zygote  : option[17]=-Xlockprofthreshold:500
23908 23908 I zygote  : option[18]=-Xcompiler-option
23908 23908 I zygote  : option[19]=--instruction-set-variant=generic
23908 23908 I zygote  : option[20]=-Xcompiler-option
23908 23908 I zygote  : option[21]=--instruction-set-features=default
23908 23908 I zygote  : option[22]=-Xcompiler-option
23908 23908 I zygote  : option[23]=--generate-mini-debug-info
23908 23908 I zygote  : option[24]=-Ximage-compiler-option
23908 23908 I zygote  : option[25]=--runtime-arg
23908 23908 I zygote  : option[26]=-Ximage-compiler-option
23908 23908 I zygote  : option[27]=-Xms64m
23908 23908 I zygote  : option[28]=-Ximage-compiler-option
23908 23908 I zygote  : option[29]=--runtime-arg
23908 23908 I zygote  : option[30]=-Ximage-compiler-option
23908 23908 I zygote  : option[31]=-Xmx64m
23908 23908 I zygote  : option[32]=-Ximage-compiler-option
23908 23908 I zygote  : option[33]=--dirty-image-objects=/system/etc/dirty-image-objects
23908 23908 I zygote  : option[34]=-Ximage-compiler-option
23908 23908 I zygote  : option[35]=--instruction-set-variant=generic
23908 23908 I zygote  : option[36]=-Ximage-compiler-option
23908 23908 I zygote  : option[37]=--instruction-set-features=default
23908 23908 I zygote  : option[38]=-Ximage-compiler-option
23908 23908 I zygote  : option[39]=--generate-mini-debug-info
23908 23908 I zygote  : option[40]=-Duser.locale=en-US
23908 23908 I zygote  : option[41]=--cpu-abilist=armeabi-v7a,armeabi
23908 23908 I zygote  : option[42]=-Xcore-platform-api-policy:just-warn
23908 23908 I zygote  : option[43]=-Xfingerprint:waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys
23908 23908 I zygote  : Core platform API reporting enabled, enforcing=false
23908 23908 D zygote  : Time zone APEX ICU file found: /apex/com.android.tzdata/etc/icu/icu_tzdata.dat
23908 23908 D zygote  : I18n APEX ICU file found: /apex/com.android.i18n/etc/icu/icudt66l.dat
23908 23908 I zygote  : Using memfd for future sealing
23908 23908 W zygote  : Using default instruction set features for ARM CPU variant (generic) using conservative defaults
   49    49 I tombstoned: received crash request for pid 23908
23908 23908 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
23908 23908 F DEBUG   : LineageOS Version: '18.1-20230723-VANILLA-waydroid_arm64'
23908 23908 F DEBUG   : Build fingerprint: 'waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys'
23908 23908 F DEBUG   : Revision: '0'
23908 23908 F DEBUG   : ABI: 'arm'
23908 23908 F DEBUG   : Timestamp: 2023-07-28 14:13:34+0000
23908 23908 F DEBUG   : pid: 23908, tid: 23908, name: main  >>> zygote <<<
23908 23908 F DEBUG   : uid: 0
23908 23908 F DEBUG   : signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x709443da (*pc=0x4000e8bd)
23908 23908 F DEBUG   :     r0  54647764  r1  3fb9709b  r2  fffffe56  r3  4337ffff
23908 23908 F DEBUG   :     r4  707184b0  r5  3fdaaaaa  r6  f295837e  r7  00000001
23908 23908 F DEBUG   :     r8  00000000  r9  f7986e00  r10 ffa33320  r11 ffa332e4
23908 23908 F DEBUG   :     ip  e9930ba4  sp  ffa332cc  lr  709443d5  pc  709443da
23908 23908 F DEBUG   : 
23908 23908 F DEBUG   : backtrace:
23908 23908 F DEBUG   :       #00 pc 0007e3da  /apex/com.android.art/javalib/arm/boot.oat (art_jni_trampoline+34) (BuildId: 4af94ec040111dd87be55d34780e36769428675c)
23908 23908 F DEBUG   :       #01 pc 000d39d5  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #02 pc 004f0759  /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #03 pc 0012ca93  /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #04 pc 00240bbf  /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #05 pc 002388df  /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+746) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #06 pc 004e44db  /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #07 pc 000ce594  /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #08 pc 003bdaa0  /system/framework/framework.jar
23908 23908 F DEBUG   :       #09 pc 0023182b  /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #10 pc 00238109  /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #11 pc 00239581  /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #12 pc 004e7239  /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #13 pc 000ce894  /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #14 pc 003bd9d4  /system/framework/framework.jar
23908 23908 F DEBUG   :       #15 pc 0023182b  /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #16 pc 00238109  /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #17 pc 00239581  /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #18 pc 004e7239  /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #19 pc 000ce894  /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #20 pc 003bc286  /system/framework/framework.jar
23908 23908 F DEBUG   :       #21 pc 0023182b  /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #22 pc 00238109  /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #23 pc 002388c7  /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+722) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #24 pc 004e44db  /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #25 pc 000ce594  /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #26 pc 003b1c7c  /system/framework/framework.jar
23908 23908 F DEBUG   :       #27 pc 0023182b  /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #28 pc 0023803d  /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #29 pc 004d321b  /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #30 pc 000d8561  /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #31 pc 0042dbaf  /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.isSrgb+446) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020)
23908 23908 F DEBUG   :       #32 pc 0042cddf  /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.<init>+822) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020)
23908 23908 F DEBUG   :       #33 pc 000d39d5  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #34 pc 004f0627  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub+282) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #35 pc 0012ca81  /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+148) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #36 pc 00240bbf  /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #37 pc 00239597  /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+558) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #38 pc 004e6b7d  /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+392) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #39 pc 000ce814  /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #40 pc 003bce74  /system/framework/framework.jar
23908 23908 F DEBUG   :       #41 pc 004e6cdd  /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #42 pc 000ce814  /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #43 pc 003bce8c  /system/framework/framework.jar
23908 23908 F DEBUG   :       #44 pc 004e6cdd  /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #45 pc 000ce814  /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #46 pc 003be6b6  /system/framework/framework.jar
23908 23908 F DEBUG   :       #47 pc 0023182b  /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #48 pc 0023803d  /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #49 pc 004d321b  /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #50 pc 000d8561  /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #51 pc 000d39d5  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #52 pc 004f0759  /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG   :       #53 pc 0012ca93  /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174)

Analyzing with gdb (by repeatedly calling gdb -p "$(ps xua | grep zygote | grep -v grep | grep -v zygote64 | awk {'print $2'})" until gdb attaches earlier to the current zygote process than the offending instruction is reached) reveals that the crash happens here:

   0x6fc373b0 <+944>:   cmp     r3, #223        @ 0xdf
   0x6fc373b2 <+946>:   movs    r6, r0
   0x6fc373b4 <+948>:   movs    r0, r5
   0x6fc373b6 <+950>:   movs    r0, r0
   0x6fc373b8 <+952>:   push    {lr}
   0x6fc373ba <+954>:   sub     sp, #4
   0x6fc373bc <+956>:   vstr    d0, [sp, #12]
   0x6fc373c0 <+960>:   vstr    d1, [sp, #20]
   0x6fc373c4 <+964>:   mov     r4, r0
   0x6fc373c6 <+966>:   ldr     r2, [sp, #20]
   0x6fc373c8 <+968>:   ldr     r3, [sp, #24]
   0x6fc373ca <+970>:   ldr     r0, [sp, #12]
   0x6fc373cc <+972>:   ldr     r1, [sp, #16]
   0x6fc373ce <+974>:   ldr.w   r12, [r4, #20]
   0x6fc373d2 <+978>:   blx     r12
   0x6fc373d4 <+980>:   vmov    d0, r0, r1
   0x6fc373d8 <+984>:   add     sp, #4
=> 0x6fc373da <+986>:   ldmia.w sp!, {lr}
   0x6fc373de <+990>:   bx      lr

(note that the actual address changes for every instance of zygote, probably due to address-space layout randomization)

The instruction at this location is 0xe8bd4000, as evidenced by:

(gdb) x/16hx 0x6fc373da
0x6fc373da <oatexec+986>:       0xe8bd  0x4000  0x4770  0x2c0f  0x0006  0x0020  0x0000  0xb500
0x6fc373ea <oatexec+1002>:      0xb081  0xed8d  0x0b03  0x4604  0x9803  0x9904  0xf8d4  0xc014

The disassembly into ldmia.w sp!, {lr} is indeed correct. However, such an instruction would be assembled into pop lr and then into ldr.w lr,[sp,#-4], which would be encoded differently. Hence, the assembly into this instruction was incorrect in the first place.

It turns out that the assembly error is due to an error in the vixl ARMv8 Runtime Code Generation Library, which is also used by Android. This error has been fixed by Feb 9, 2021. However, this fix has not made it into Android 13. Thus, at least Android 11, Android 12, Android 13 cannot run on current qemu-system-aarch64, while it should.

Users of the Android emulator (also based on QEMU) do not seem to suffer from this bug because the Android QEMU has bitrotted since the year 2018 and hence has not seen any Arm emulation modernization in QEMU (e.g. the Tiny Code Generator) since, and only this modernization has exposed this bug in the first place.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information