Support running real-world Android on Arm by supporting one-register list for the POP (LDMIA) Thumb32 instruction.
Summary
Real-world Android code generated for 32-bit Arm software (even running on 64-bit Arm platforms) contains (unfortunately) Arm T32 instructions such as
ldmia.w sp!, {lr}
(0xe8bd4000)
The ldmia.w
instruction has 2 parameters: a register (e.g. sp
) and a register list (enclosed in braces). pop
is an alias for such an instruction with sp
as its first parameter. Instructions where the register list is empty or contains exactly one entry are considered unpredictable. Thus, the parameter list should contain at least 2 entries, and instructions with only 1 entry in the parameter list should not exist. Yet, they do occur in Android 11 at least (in the 32-bit zygote
process), and probably in any other recent Android version using ART
, too.
Real-world Arm CPUs (such as Arm Cortex-A55) seem to work fine with these "unpredictable" instructions, but the QEMU CPU (of more recent QEMU) does not. Executing such code on a QEMU Arm CPU (e.g. "cortex-a55") results in SIGILL
with code ILL_ILLOPC
. This is correct, as the QEMU Arm CPU works according to the specification. However, at the same time, it is incorrect, as any real Arm Cortex-A55 executes such an instruction just fine, and a QEMU user selecting a "cortex-a55" CPU in QEMU expects emulation like a real Arm Cortex-A55.
The solution is very simple. Just replace in target/arm/tcg/translate.c
/* BitCount(list) < 2 is UNPREDICTABLE */
return do_ldm(s, a, 2);
by
/* BitCount(list) < 2 is UNPREDICTABLE, but BitCount(list)==1 is supported by each real-world Arm CPU supporting T32 */
return do_ldm(s, a, 1);
This would make it possible to run real-world Android software for Arm CPUs on QEMU, which is currently not possible due to this bug.
Host environment
- Operating system: Linux
- OS/kernel version: 6.1.11
- Architecture: x86_64
- QEMU flavor: qemu-system-aarch64
- QEMU version: 8.0.3
- QEMU command line: ./qemu-system-aarch64 -machine virt-8.0,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,acpi=off -accel tcg -cpu cortex-a55 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny ...
Emulated/Virtualized environment
- Operating system: LineageOS 18.1 on Waydroid 1.4.1 on PostmarketOS edge
- OS/kernel version: 6.4.4-0-edge
- Architecture: aarch64
Steps to reproduce
- Get any aarch64 Linux on QEMU for x86_64 running. Make sure that Wayland is running. (For example, build PostmarketOS with "phosh" for aarch64 and install it.)
- Install waydroid (e.g.
apk add waydroid
). - Install the LineageOS 18.1 for waydroid image (e.g.
waydroid init
). - Run the waydroid-container (e.g.
rc-service waydroid-container restart
). - Start the waydroid session (e.g. click on the "Waydroid" symbol on the graphical user interface).
- Observe the waydroid log file (e.g. run
waydroid logcat
).
Additional information
The output of the Android log (using waydroid logcat
) will be akin:
23908 23908 D AndroidRuntime: >>>>>> START com.android.internal.os.ZygoteInit uid 0 <<<<<<
23908 23908 I AndroidRuntime: Using default boot image
23908 23908 I AndroidRuntime: Leaving lock profiling enabled
23908 23908 E cutils-trace: Error opening trace file: No such file or directory (2)
23908 23908 I zygote : option[0]=-Xzygote
23908 23908 I zygote : option[1]=exit
23908 23908 I zygote : option[2]=vfprintf
23908 23908 I zygote : option[3]=sensitiveThread
23908 23908 I zygote : option[4]=-verbose:gc
23908 23908 I zygote : option[5]=-XX:PerfettoHprof=true
23908 23908 I zygote : option[6]=-Xms8m
23908 23908 I zygote : option[7]=-Xmx512m
23908 23908 I zygote : option[8]=-XX:HeapGrowthLimit=192m
23908 23908 I zygote : option[9]=-XX:HeapMinFree=8m
23908 23908 I zygote : option[10]=-XX:HeapMaxFree=16m
23908 23908 I zygote : option[11]=-XX:HeapTargetUtilization=0.6
23908 23908 I zygote : option[12]=-Xusejit:true
23908 23908 I zygote : option[13]=-Xjitsaveprofilinginfo
23908 23908 I zygote : option[14]=-XjdwpOptions:suspend=n,server=y
23908 23908 I zygote : option[15]=-XjdwpProvider:default
23908 23908 I zygote : option[16]=-Xopaque-jni-ids:swapable
23908 23908 I zygote : option[17]=-Xlockprofthreshold:500
23908 23908 I zygote : option[18]=-Xcompiler-option
23908 23908 I zygote : option[19]=--instruction-set-variant=generic
23908 23908 I zygote : option[20]=-Xcompiler-option
23908 23908 I zygote : option[21]=--instruction-set-features=default
23908 23908 I zygote : option[22]=-Xcompiler-option
23908 23908 I zygote : option[23]=--generate-mini-debug-info
23908 23908 I zygote : option[24]=-Ximage-compiler-option
23908 23908 I zygote : option[25]=--runtime-arg
23908 23908 I zygote : option[26]=-Ximage-compiler-option
23908 23908 I zygote : option[27]=-Xms64m
23908 23908 I zygote : option[28]=-Ximage-compiler-option
23908 23908 I zygote : option[29]=--runtime-arg
23908 23908 I zygote : option[30]=-Ximage-compiler-option
23908 23908 I zygote : option[31]=-Xmx64m
23908 23908 I zygote : option[32]=-Ximage-compiler-option
23908 23908 I zygote : option[33]=--dirty-image-objects=/system/etc/dirty-image-objects
23908 23908 I zygote : option[34]=-Ximage-compiler-option
23908 23908 I zygote : option[35]=--instruction-set-variant=generic
23908 23908 I zygote : option[36]=-Ximage-compiler-option
23908 23908 I zygote : option[37]=--instruction-set-features=default
23908 23908 I zygote : option[38]=-Ximage-compiler-option
23908 23908 I zygote : option[39]=--generate-mini-debug-info
23908 23908 I zygote : option[40]=-Duser.locale=en-US
23908 23908 I zygote : option[41]=--cpu-abilist=armeabi-v7a,armeabi
23908 23908 I zygote : option[42]=-Xcore-platform-api-policy:just-warn
23908 23908 I zygote : option[43]=-Xfingerprint:waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys
23908 23908 I zygote : Core platform API reporting enabled, enforcing=false
23908 23908 D zygote : Time zone APEX ICU file found: /apex/com.android.tzdata/etc/icu/icu_tzdata.dat
23908 23908 D zygote : I18n APEX ICU file found: /apex/com.android.i18n/etc/icu/icudt66l.dat
23908 23908 I zygote : Using memfd for future sealing
23908 23908 W zygote : Using default instruction set features for ARM CPU variant (generic) using conservative defaults
49 49 I tombstoned: received crash request for pid 23908
23908 23908 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
23908 23908 F DEBUG : LineageOS Version: '18.1-20230723-VANILLA-waydroid_arm64'
23908 23908 F DEBUG : Build fingerprint: 'waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys'
23908 23908 F DEBUG : Revision: '0'
23908 23908 F DEBUG : ABI: 'arm'
23908 23908 F DEBUG : Timestamp: 2023-07-28 14:13:34+0000
23908 23908 F DEBUG : pid: 23908, tid: 23908, name: main >>> zygote <<<
23908 23908 F DEBUG : uid: 0
23908 23908 F DEBUG : signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x709443da (*pc=0x4000e8bd)
23908 23908 F DEBUG : r0 54647764 r1 3fb9709b r2 fffffe56 r3 4337ffff
23908 23908 F DEBUG : r4 707184b0 r5 3fdaaaaa r6 f295837e r7 00000001
23908 23908 F DEBUG : r8 00000000 r9 f7986e00 r10 ffa33320 r11 ffa332e4
23908 23908 F DEBUG : ip e9930ba4 sp ffa332cc lr 709443d5 pc 709443da
23908 23908 F DEBUG :
23908 23908 F DEBUG : backtrace:
23908 23908 F DEBUG : #00 pc 0007e3da /apex/com.android.art/javalib/arm/boot.oat (art_jni_trampoline+34) (BuildId: 4af94ec040111dd87be55d34780e36769428675c)
23908 23908 F DEBUG : #01 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #02 pc 004f0759 /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #03 pc 0012ca93 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #04 pc 00240bbf /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #05 pc 002388df /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+746) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #06 pc 004e44db /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #07 pc 000ce594 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #08 pc 003bdaa0 /system/framework/framework.jar
23908 23908 F DEBUG : #09 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #10 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #11 pc 00239581 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #12 pc 004e7239 /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #13 pc 000ce894 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #14 pc 003bd9d4 /system/framework/framework.jar
23908 23908 F DEBUG : #15 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #16 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #17 pc 00239581 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #18 pc 004e7239 /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #19 pc 000ce894 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #20 pc 003bc286 /system/framework/framework.jar
23908 23908 F DEBUG : #21 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #22 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #23 pc 002388c7 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+722) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #24 pc 004e44db /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #25 pc 000ce594 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #26 pc 003b1c7c /system/framework/framework.jar
23908 23908 F DEBUG : #27 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #28 pc 0023803d /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #29 pc 004d321b /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #30 pc 000d8561 /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #31 pc 0042dbaf /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.isSrgb+446) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020)
23908 23908 F DEBUG : #32 pc 0042cddf /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.<init>+822) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020)
23908 23908 F DEBUG : #33 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #34 pc 004f0627 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub+282) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #35 pc 0012ca81 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+148) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #36 pc 00240bbf /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #37 pc 00239597 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+558) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #38 pc 004e6b7d /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+392) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #39 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #40 pc 003bce74 /system/framework/framework.jar
23908 23908 F DEBUG : #41 pc 004e6cdd /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #42 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #43 pc 003bce8c /system/framework/framework.jar
23908 23908 F DEBUG : #44 pc 004e6cdd /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #45 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #46 pc 003be6b6 /system/framework/framework.jar
23908 23908 F DEBUG : #47 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #48 pc 0023803d /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #49 pc 004d321b /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #50 pc 000d8561 /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #51 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #52 pc 004f0759 /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #53 pc 0012ca93 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174)
Analyzing with gdb
(by repeatedly calling gdb -p "$(ps xua | grep zygote | grep -v grep | grep -v zygote64 | awk {'print $2'})"
until gdb
attaches earlier to the current zygote
process than the offending instruction is reached) reveals that the crash happens here:
0x6fc373b0 <+944>: cmp r3, #223 @ 0xdf
0x6fc373b2 <+946>: movs r6, r0
0x6fc373b4 <+948>: movs r0, r5
0x6fc373b6 <+950>: movs r0, r0
0x6fc373b8 <+952>: push {lr}
0x6fc373ba <+954>: sub sp, #4
0x6fc373bc <+956>: vstr d0, [sp, #12]
0x6fc373c0 <+960>: vstr d1, [sp, #20]
0x6fc373c4 <+964>: mov r4, r0
0x6fc373c6 <+966>: ldr r2, [sp, #20]
0x6fc373c8 <+968>: ldr r3, [sp, #24]
0x6fc373ca <+970>: ldr r0, [sp, #12]
0x6fc373cc <+972>: ldr r1, [sp, #16]
0x6fc373ce <+974>: ldr.w r12, [r4, #20]
0x6fc373d2 <+978>: blx r12
0x6fc373d4 <+980>: vmov d0, r0, r1
0x6fc373d8 <+984>: add sp, #4
=> 0x6fc373da <+986>: ldmia.w sp!, {lr}
0x6fc373de <+990>: bx lr
(note that the actual address changes for every instance of zygote
, probably due to address-space layout randomization)
The instruction at this location is 0xe8bd4000, as evidenced by:
(gdb) x/16hx 0x6fc373da
0x6fc373da <oatexec+986>: 0xe8bd 0x4000 0x4770 0x2c0f 0x0006 0x0020 0x0000 0xb500
0x6fc373ea <oatexec+1002>: 0xb081 0xed8d 0x0b03 0x4604 0x9803 0x9904 0xf8d4 0xc014
The disassembly into ldmia.w sp!, {lr}
is indeed correct. However, such an instruction would be assembled into pop lr
and then into ldr.w lr,[sp,#-4]
, which would be encoded differently. Hence, the assembly into this instruction was incorrect in the first place.
It turns out that the assembly error is due to an error in the vixl
ARMv8 Runtime Code Generation Library, which is also used by Android. This error has been fixed by Feb 9, 2021. However, this fix has not made it into Android 13. Thus, at least Android 11, Android 12, Android 13 cannot run on current qemu-system-aarch64
, while it should.
Users of the Android emulator (also based on QEMU) do not seem to suffer from this bug because the Android QEMU has bitrotted since the year 2018 and hence has not seen any Arm emulation modernization in QEMU (e.g. the Tiny Code Generator) since, and only this modernization has exposed this bug in the first place.