Gracefully handle packages breaking because of soname bumps

Created by: ollieparanoid

Fixes #893 (closed). I ended up writing a lot here in the description in the hope that this makes it easier to understand the problem and how it's solved. Feedback whether that is good or bad is welcome (but actual reviewing is more important 😉).

Introduction: soname bumps break our packages

Linux programs as packaged in Alpine (and most other distributions) link against dynamic libraries (.so files). These libraries may release incompatible upgrades, which "bump" (increase) the version of the library file (the "soname").

Whenever you build a package with Alpine's abuild (directly or e.g. via pmbootstrap build), and it uses such dynamic libraries, abuild will automatically scan for those and add the exact soname as dependency.

Here's one example. All depends starting with so: are such sonames:

$ pmbootstrap parse_apkindex ~/.local/var/pmbootstrap/packages/x86_64/APKINDEX.tar.gz testapp
{
    "pkgname": "testapp",
    "version": "0-r1",
    "arch": "x86_64",
    "timestamp": "1500000000",
    "depends": [
        "testlib",
        "so:libc.musl-x86_64.so.1",
        "so:libtestlib.so.0"
    ],
    "provides": [
        "cmd:testapp"
    ]
}

postmarketOS uses its own binary package repository together with the ones from Alpine. That means: When a postmarketOS package links against a dynamic library from Alpine, it's currently possible that said library gets a soname bump in Alpine, and then you can't install the binary postmarketOS package anymore because it was built against the old version.

Alpine's apk package manager will write an error - but it is not obvious how the user can work around that if we didn't notice it and fix it in the binary repo: You'd need to mark the postmarketOS aport, which has the outdated dependency, for rebuild. To do it properly, one would increase the pkgrel (that's also how Alpine handles rebuilds for soname upgrades, I've asked in their IRC channel).

For reference, this is what apk says when a dependency had a soname bump:

ERROR: unsatisfiable constraints:
  so:libtestlib.so.1 (missing):
    required by:
                 testapp-1.0-r0[so:libtestlib.so.1]
                 testapp-1.0-r0[so:libtestlib.so.1]

Changes

New command: pmbootstrap pkgrel_bump --auto

Automatically find all packages from the postmarketOS binary repository (disable with pmbootstrap --mirror-pmOS="") as well as the locally built packages, that are broken because of soname bumps. Increase the pkgrel in each package's APKBUILD to mark it for rebuilt.

Variations

• pmbootstrap pkgrel_bump --auto --dry: dry run mode, doesn't change anything and yields a non-zero exit code in case there's a package that would be bumped (this was --auto-dry before and got changed, see below)
• pmbootstrap pkgrel_bump hello-world: bump the pkgrel of one specific package without checking the soname dependencies, useful for related problem #978 (moved)

Automatic check before installing packages

Before any package gets installed (e.g. during pmbootstrap install, which generates a full system image), each package and its dependencies are checked against being broken because of soname bumps. When that is the case, we abort and ask the user to run pmbootstrap pkgrel_bump --auto.

$ pmbootstrap install --add=testapp
*** (1/5) PREPARE NATIVE CHROOT ***
*** (2/5) CREATE DEVICE ROOTFS ("qemu-amd64") ***
NOTE: Run 'pmbootstrap pkgrel_bump --auto' to mark packages with outdated dependencies for rebuild. This will most likely fix this issue (soname bump?).
NOTE: More dependency calculation logging with 'pmbootstrap -v'.
ERROR: Could not find package 'so:libtestlib.so.1' in the aports folder and could not find it in any APKINDEX.
Run 'pmbootstrap log' for details.
See also: <https://postmarketos.org/troubleshooting>

Testcases

• test_soname_bump.py: executes the dry run, fails if it finds broken packages (so we check for breakage in our binary packages repo whenever a new pull request comes in, usually at least once per day, example output).
• test_soname_bump.py: builds the new test aports testlib and testapp from test/testdata/pkgrel_bump/aports, and creates the soname breakage on purpose to verify that pkgrel_bump --auto works as intended.

How to test

After reviewing the code, please create the soname problem locally, and verify that pmbootstrap properly detects it at installation time:

# Copy testlib and testapp to aports, so we can build them
cp -r test/testdata/pkgrel_bump/aports aports/_temp

# Selet the qemu-amd64 device (because it uses x86_64 packages) and delete
# all chroots. Then build testapp and testlib (gets pulled in as dependency)
pmbootstrap zap
pmbootstrap config device qemu-amd64
pmbootstrap build testapp

# Bump the pkgrel of testlib. The package is written in a way that this will
# also increase the soname version.
pmbootstrap pkgrel_bump testlib

# This should rebuild testlib
pmbootstrap build testlib

# Delete the old testlib package to break testapp intentionally
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/testlib-1.0-r0.apk
pmbootstrap index

# Perform a normal installation and ask pmbootstrap to add testapp. It should
# complain that the package depends on the missing "so:libtestlib.so.0" and
# abort the installation
pmbootstrap install --add=testapp

# This should fix the problem by increasing the testapp pkgrel
pmbootstrap pkgrel_bump --auto
pmbootstrap install --add=testapp

# Clean up
rm -r aports/_temp
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/testlib*
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/testapp*
pmbootstrap index