Proposal: Use the Package-and-Test test suite to assess stable branch quality for patch releases
Context
Delivery is working to extend the GitLab maintenance policy to support bug fixes to the two previous monthly releases in addition to the current stable release. This means patch releases will change from releasing a single version package to releasing three package versions.
We currently only have a single Release environment which means we can only deploy and test the current package version.
We're working to increase the number of release environments to allow us to deploy and test all three package versions. To avoid delaying the maintenance policy extension we will also need an interim process to guarantee that the stable branches for all three versions are ready to be tagged and released for self-managed users.
The purpose of this issue is to agree on this interim process.
Interim proposal 1: Use the Package-and-Test pipeline as the quality indicator
The package-and-test is already used to assess backports outside the maintenance policy before release. We could use the same test suite to test all three versions by enforcing its execution on merge requests targeting stable branches and on pipelines running on stable branches.
Part of the work to extend the maintenance policy will open up the stable branches to allow developers to merge bug fixes directly into the stable branch rather than rely on the pick labels. This would give us a process where developers prepare a bug fix merge request and receive feedback of any failing tests from the package-and test
pipeline. Release Managers will continue to tag and release from the working stable branch.
To guarantee that changes to stable branches have been tested the package-and-test
pipeline should not be allowed to fail. We would need help from Quality to handle or reduce test flakiness to allow developers to rely on the test results.
Interim proposal 2: Allow Quality to approve all merge requests targeting stable branches
If the duration and stability of the Package-and-Test
pipeline is a problem we could alternatively allow Quality to approve all merge requests targeting the stable branch. This is a similar process to the one AppSec uses to check security fixes and would mean that each fix is tested independently to give confidence that it is working with all previous changes.
The downside of this approach is increased work for Quality engineers and possible delays to patch releases.
A long-term solution
This proposal is an interim solution until the release environments work is completed. Once we have Release environments available to test each version we will be able to automatically deploy and test the individual packages.
Finalized proposal: Run E2E pipelines on demand.
Details on #2725 (comment 1244036277).
End-to-end pipelines will be executed on merge requests targeting stable branches:
- The
e2e:package-and-test
pipeline will be manual and allowed to fail. A Danger message will be posted to instruct merge request authors to run this pipeline and to ask the SET counterpart for assistance if one of the jobs fails. #2796 (closed) - To guarantee this pipeline is triggered before merging we could also make Danger fail if the
e2e: package-and-test
pipeline hasn't been executed. #2797 (closed)