Type hinting and Chroot's (Caleb's amazing pmbootstrap v3 patch series!)
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap API. The first component being Suffix and SuffixType. This explicitly defines what suffixes are possible, future changes should aim to further constrain this API (e.g. by validating against available device codenames or architectures for buildroot suffixes).
This patch also converts the codebase to use the new Suffix type, and introduces the pathlib Path type for all path handling, to replace the common string building patterns. This is less error prone and can be integrated nicely with the Chroot type to make the code more readable and easier to refactor in the future.
The general idea here is to encapsulate common patterns into type hinted code, and gradually reduce the amount of assumptions made around the codebase so that future changes are easier to implement.
Ok, this MR has grown quite a lot. Here's a rough overview of what's in it now:
Args removed from most of the codebase, except where it actually makes sense.
The next step here will be to expand on pmb/commands/ types and use them to abstract out args from the other areas like pmb/install where it's still used a lot.
Introduced pmb/core/context
This replaces args for some things, and allows accessing the pmbootstrap config. As implemented right now it can be accessed from anywhere by calling get_context(). This is not ideal since it still keeps context as global state, ironically it would be better to have this state be passed around to functions, however I think some larger restructuring is needed to make this easier (like right now some really "low level" functions need access to this, but they're called from so many places that we'd end up with just the args problem all over again).
Chroot management
See the first section of this MR. Additionally, we no longer call pmb.chroot.init() everywhere, instead pmb.chroot module expects the first user to initialise the chroot before use. This avoids all the weird places it was being called from and solves a whole class of recursion issues that were really easy to run into when hacking on pmb before.
One approach to consider in the future is if we can make the Chroot type manage state, by refactoring the codebase to pass around Chroot objects (even) more, and/or by having the Chroot constructors always return the same object (so if you call Chroot.native() from two places you'll always get the same Chroot object).
The second approach seems like something fairly idiomatic imo, since the Chroot class is meant to be representative of the actual chroot on your filesystem, so doing chroot = Chroot.native() already feels like you're getting a handle to the chroot.
pathlib Paths
See the first section of this MR. Makes path building really simple and helps cut down on common path building patterns (since you now do chroot / "etc/apk/repositories to get an absolute path for example).
Arch type
pmb/parse/arch.py has been moved over to pmb/core/arch.py and refactored into a proper type. Since this is used in so many places across the codebase, having a proper type simplifies all the usages and disambiguates what kind of Arch we're working with.
Deviceinfo type
Similarly to Arch, deviceinfo is now a class, next steps will be to move this to core (requires reworking some of the parsing). As well as maybe doing type conversions (currently only done for deviceinfo.arch).
Config rework
pmb/core/config.py now contains a Config class with some magic handling so we can parse pmbootstrap.cfg, validate the types, and use it.
Previously, the config values were merged into args. Now, the runtime config is updated with any relevant values from args.
config.mirrors
The mirror_alpine and mirrors_postmarketos config options are removed in favor of a new [mirrors] section in pmbootstrap.cfg. This let's us specify mirrors per-aports for multiple aports repositories.
package builder logic rework
The package building logic in pmb/build/_package.py is rewritten. The old builder used recursion to descend into package dependencies and build them first. This limits the UX since we don't know ahead of time what packages we're actually going to build, and messages can get lost in the middle.
The new system recurses dependencies first and figures out what to build (in what order), then we actually build the packages.
Cache decorator
The pmb.helpers.other.cache dictionary is (mostly) removed in favor of a new @Cache function decorator. The decorator replaces almost all uses allowing caching to be done based on the function arguments, and even letting you disable caching when certain function arguments have specific values.
One or two uses of caching are not covered (and maybe never will be?).
pkgrepo API + support for multiple source repositories
One of the main blockers for systemd. We now have a new API in pmbootstrap to formalise the patterns for accessing the pmaports directory, and allowing multiple repos to be used.
In addition, special handling is added for the extra-repos dir we use for the systemd packages. For all intents and purposes, pmbootstrap now treats the extra-repos/systemd as it's own source repository, so packages like gnome-shell can be in both pmaports and systemd repos.
There are some issues that will need to be figured out regarding stuff like pmbootstrap lint (which right now will lint both).
As part of the package building rework, built packages will now be kept in a different local binary repo depending on which package repo it's from.
With this, the plasma mobile nightly repo could now be moved it it's own git repository, or into extra-repos - though the latter will require changes to pmbootstrap since there is a lot of systemd hardcoding right now.
The extra repos can also have their own mirrors, currently this is a TypedDict, but that shouldn't prevent arbitrary additional repos being added.
Use apk.static in native chroot and for all apk invocations
We fetch apk static from Alpine, however for systemd we actually use a fork of apk-tools. We now install apk-tools-static in the native chroot and run it from there. This is necessary for the second half since we now use apk.static whenever we install packages in a chroot, since it's much faster than running apk through QEMU.
Don't enforce checksums unless building with strict
The abuild verify() function is overridden by appending a drop-in to the end of the APKBUILD file when copying it into the chroot. This version is modified to not die() when the checksums don't match, instead it sets a flag that pmb uses after building to display a warning (and instructions to generate the checksum).
The should immensely speed up the dev loop case where you're building a package over and over again for testing.
Allow for incremental builds with --src
Make rsync exclude files that are in the .gitignore of the project (like build artifacts). And by default have abuild keep the build artifacts. This way if you're running pmbootstrap build --src=... multiple times then incremental builds will be used.