Skip to content
  • Marek Habersack's avatar
    [Xamarin.Android.Build.Tasks] Make all assemblies RID-specific (#8478) · 86260ed3
    Marek Habersack authored
    Fixes: https://github.com/xamarin/xamarin-android/issues/8168
    
    Context: https://github.com/xamarin/xamarin-android/issues/4337
    Context: https://github.com/xamarin/xamarin-android/issues/8155
    
    Context: 55e5c349
    Context: 68368189
    Context: 929e7012
    Context: c9270261
    Context: 2f192386
    
    Issue xamarin/xamarin-android#8155 noted a *fundamental* mismatch in
    expectations between the Classic Xamarin.Android packaging worldview
    and the .NET worldview: In Classic Xamarin.Android, all assemblies
    are presumed to be architecture agnostic ("AnyCPU"), while in .NET:
    
     1. `System.Private.CoreLib.dll` was *always* an architecture-specific
        assembly (see xamarin/xamarin-android#4337), and
    
     2. The .NET Trimmer is extensible and can apply ABI-specific changes
        to IL which *effectively* results in an architecture-specific
        assembly (xamarin/xamarin-android#8155).  Meanwhile, there is no
        way of knowing that this is happening, and the trimmer doesn't
        mark the resulting assembly as architecture-specific.
    
    We long tried to "paper over" this difference, by trying to find --
    and preserve the "nature" of -- architecture-agnostic assemblies
    (55e5c349, …).  Unfortunately, all attempts at trying to preserve the
    concept of architecture-agnostic assemblies have failed; we're
    fighting against .NET tooling in attempting to do so.
    
    In commit 68368189 this came to a head: a long worked-on feature
    LLVM Marshal Methods (8bc7a3e8) had to be disabled because of
    hangs within MAUI+Blazor Hybrid+.NET Android apps, and we suspect
    that treating an assembly as architecture-agnostic when it was
    "actually" architecture-specific is a plausible culprit.
    
    Bite the bullet: there is no longer such a thing as an architecture-
    agnostic assembly.  Treat *all* assemblies as if they were
    architecture-specific.
    
    Additionally, alter assembly packaging so that instead of using
    `assemblies/assemblies*.blob` files (c9270261), we instead store the
    assemblies within `lib/ABI` of the `.apk`/`.aab`.
    
    The Runtime config blob `rc.bin` is stored as `lib/ABI/libarc.bin.so`.
    
    When `$(AndroidUseAssemblyStore)`=true, assemblies will be stored
    within `lib/ABI/libassemblies.ABI.blob.so`, e.g.
    `lib/arm64-v8a/libassemblies.arm64-v8a.blob.so`.
    
    When `$(AndroidUseAssemblyStore)`=false and Fast Deployment is *not*
    used, then assemblies are stored individually within `lib/ABI` as
    compressed assembly data, with the following "name mangling"
    convention:
    
      * Regular assemblies: `lib_` + Assembly File Name + `.so`
      * Satellite assemblies:
        `lib-` + culture + `-` + Assembly File Name + `.so`
    
    For example, consider this selected `unzip -l` output:
    
    	% unzip -l bin/Release/net9.0-android/*-Signed.apk | grep lib/arm64-v8a
    	   723560  01-01-1981 01:01   lib/arm64-v8a/libSystem.IO.Compression.Native.so
    	    70843  01-01-1981 01:01   lib/arm64-v8a/lib_Java.Interop.dll.so
    	   157256  01-01-1981 01:01   lib/arm64-v8a/libaot-Java.Interop.dll.so
    	     1512  01-01-1981 01:01   lib/arm64-v8a/libarc.bin.so
    
      * `libSystem.IO.Compression.Native.so` is a native shared library
        from .NET
      * `lib_Java.Interop.dll.so` is compressed assembly data for
        `Java.Interop.dll`
      * `libaot-Java.Interop.dll.so` contains Profiled AOT output for
        `Java.Interop.dll`
      * `libarc.bin.so` is the `rc.bin` file used by .NET runtime startup
    
    Additionally, note that Android limits the characters that can be
    used in native library filenames to the regex set `[-._A-Za-z0-9]`.
    
    TODO: No error checking is done to ensure that "Assembly File Name"
    stays within the limits of `[-.A-Za-z0-9]`, e.g. if you set
    `$(AssemblyName)=Emoji😅` *and `$(AndroidUseAssemblyStore)`=false,
    then we'll try to add `lib/arm64-v8a/lib_Emoji😅.dll.so`, which will
    fail at runtime.  This works when `$(AndroidUseAssemblyStore)`=true,
    which is the default.
    
    Pros:
    
      * We're no longer fighting against .NET tooling features such as
        ILLink Substitutions.
    
      * While `.aab` files will get larger, we expect that the actual
        `.apk` files sent to Android devices from the Google Play
        Store will be *smaller*, as the Google Play Store would always
        preserve/transmit *all* `assemblies/assemblies*.blob` files,
        while now it will be able to remove `lib/ABI/*` for unsupported
        ABIs.
        
    Cons:
    
      * `.apk` files containing more than one ABI ***will get larger***,
        as there will no longer be "de-duping" of architecture-agnostic
        assembly data.  We don't consider this a significant concern, as
        we believe `.aab` is the predominant packaging format.
    
    ~~ All assemblies are architecture-specific ~~
    
    Assembly pre-processing changes so that every assembly ends up in
    every target architecture batch, regardless of whether its MVID
    differs from its brethren or not.  This is done very early in the
    build process on our side, where we make sure that each assembly
    either has the `%(Abi)` metadata or is given one, and is placed in
    the corresponding batch.  Further processing of those batches is
    "parallel", in that no code attempts to de-duplicate the batches.
    
    
    ~~ Impact on Fast Deployment, `$(IntermediateOutputPath)` ~~
    
    The changes also required us to place all the assemblies in new
    locations on disk within `$(IntermediateOutputPath)` when building
    the application.  (Related: 2f192386.)  Assemblies are now placed in
    subdirectories named after either the target architecture/ABI or the
    .NET `$(RuntimeIdentifier)`, e.g.
    `obj/Release/netX.Y-android/android-arm64`.  This, in turn, affects
    e.g. Fast Deployment as now the synchronized content is in the
    `…/.__override__/ABI` directory on device, instead of just in
    `…/.__override__`.
    
    
    ~~ File Formats ~~
    
    The assembly store format (c9270261) is updated to use the following
    structures:
    
    	struct AssemblyStoreHeader {
    	    uint32_t magic;
    	    uint32_t version;
    	    uint32_t entry_count;               // Number of assemblies in the store
    	    uint32_2 index_entry_count;
    	    uint32_t index_size;
    	};
    	struct AssemblyStoreIndexEntry {
    	    intptr_t name_hash;	                // xxhash of assembly filename
    	    uint32_t descriptor_index;          // index into `descriptors` array
    	};
    	struct AssemblyStoreEntryDescriptor {
    	    uint32_t mapping_index;             // index into an internal runtime array
    	    uint32_t data_offset;               // index into `data` for assembly `.dll`
    	    uint32_t data_size;                 // size of assembly, in bytes
    	    uint32_t debug_data_offset;         // index into `data` for assembly `.pdb`; 0 if not present
    	    uint32_t debug_data_size;           // size of `.pdb`, in bytes; 0 if not present
    	    uint32_t config_data_offset;        // index into `data` for assembly `.config`; 0 if not present
    	    uint32_t config_data_size;          // size of `.config`, in bytes; 0 if not present
    	};
    	struct AssemblyStoreAssemblyInfo {
    	    uint32_t length;                    // bytes
    	    uint8_t  name[length];
    	};
    
    `libassemblies.ABI.blob.so` has the following format, and is *not* a
    valid ELF file:
    
    	AssemblyStoreHeader                 header {…};
    	AssemblyStoreIndexEntry             index [header.index_entry_count];
    	AssemblyStoreAssemblyDescriptor     descriptors [header.entry_count];
    	AssemblyStoreAssemblyInfo           names [header.entry_count];
    	uint8_t data[];
    86260ed3