Testing vision notes

This repo fills a critical role right now and we need to keep it working. CI and testing can always be better though and here I'll try to outline some things we can do in the near future.

A key thing is that I think this repository should basically:

• transform into an orchestrator/aggregator of other tests
• Have at least some "simulated prod deployment" testing where we have something that tries to e.g. do upgrades from older releases, run synthetic container workloads etc. More on this below

Other tests

• github.com/bootc-dev/bootc carries tests (in TMT form) and we don't want to duplicate those
• Anaconda carries some testing of image mode; see e.g. https://github.com/rhinstaller/kickstart-tests/blob/16b4e1823e2ac7845f003d8a2d9f13694c7fb648/scripts/defaults-rhel9.sh#L8 - we should be ensuring that their tests work, i.e. most kickstart testing should live in their repository I think
• There's also tests in https://github.com/osbuild/bootc-image-builder/tree/main/test that are only run upstream I think

Shifting left

Bigger picture too a key deliverable for us I think is "shifting left"; ensuring that image mode tests are run on PRs for packages. This is mentioned in https://gitlab.com/fedora/bootc/tracker/-/issues?show=eyJpaWQiOiI0OCIsImZ1bGxfcGF0aCI6ImZlZG9yYS9ib290Yy90cmFja2VyIiwiaWQiOjE1ODAwNjIwNH0%3D at least.

Fixing where our components aren't integrating with dist-git gating

Right now ostree has a lot of tests but we haven't mapped those to tmt. Same with rpm-ostree.