Skip to content

CI: support for integration tests in CI

Replace use of the deprecated "only" keyword with the much more flexible "rules". Introduce support for spawning a dynamic child pipeline to run tests against packages that were built in CI.

The child pipeline initially allows for packages to define what tests and for which devices to run. Initially, this supports the following kinds of tests:

  1. Install tests "pmbootstrap install" is run with the newly built package in the local repo.

  2. Upgrade tests Where a chroot for the device is created, the new package made available via a local repo, and then "apk upgrade -a" is run inside the chroot. This allows for validating the .post-upgrade script in the context of a real rootfs.

This is currently limited to packages that are automatically pulled in as part of a default installation. This is because handling non-default packages requires either spawning jobs per-package, or assuming that multiple packages updated in the same MR can all be installed at once (often not the case). In the end this can't be handled properly without a dependency graph solver, or by imposing some strong restrictions. Hence it is deemed out of scope for this initial implementation.

How this works

Packages in pmaports can define a PMAPORTS-CI header, this declares that when modified they should trigger this additional integration testing. It is only valid for packages that are automatically pulled in for a console UI build for a specific device to opt-in to this testing - optional packages will be ignored. That is to say, the integration testing in its current form totally ignores the issue of package dependency resolution. Hence this is only suggested for core packages like postmarketos-base, boot-deploy, postmarketos-initramfs, etc.

Example header:

# PMAPORTS-CI
# test-upgrade: main oneplus-enchilada lenovo-21bx

The following configuration options are available:

test-upgrade

A list of devices (main being an alias for all main category devices) where we should test that we can upgrade from the previous to the new version of the package.

test-boot (not implemented)

A list of devices where we should validate that they can properly boot (and pass certain tests?) with the new version of the package.


Currently, this implementation will run the install and upgrade tests for all modified packages against the union of the devices specified. That is to say, this MR modifies postmarketos-base and postmarketos-initramfs, with the base package specifying that it should be tested on main devices, and the initramfs package specifying main oneplus-enchilada lenovo-21bx. Thus, both the base and initramfs packages will be tested for main oneplus-enchilada lenovo-21bx.

This is probably not ideal, and a limitation in how I've implemented this. My goal was to avoid creating too many jobs. We could probably solve this by allowing packages to specify other packages they conflict with and then grouping based on that rather than creating a job per package per device.

Issues

I can't figure out how to get the signing key used to build the packages in the build job inside the test job in a way that the chroot doesn't reject with packages with "UNTRUSTED Signature". I have a hacked up version of pmbootstrap that runs apk with "--allow-untrusted" as a temporary workaround. But see the gitlab CI yaml files for more info.

Future plans

per-package tests

Particularly relevant for UI packages, not so much for other software. We should ensure that on supported UI/device combos that the UI boots up and works properly, "looks correct", etc. We can do this with...

Boot testing

Boot testing is not yet implemented, but with the changes made by this MR it becomes much easier to do, either via QEMU or some other mechanism like CDBA.

Edited by Caleb Connolly

Merge request reports