Document a process for testing release-tools refactorings on branches
Currently, when we refactor or otherwise make sweeping changes to the release-tools code that handles auto-deploys, tags, and security releases, it's often difficult to test without real-world executions. If we merge the changes to
master in order to test them, if something goes wrong our only option is to revert in order to get back to normal.
I think we can work around this by allowing jobs to run on specific branches, before they're merged to
master. Because of our CI variable configuration, and mirroring setup to
ops, these branches have to be protected.
I propose using a specific branch prefix scheme, something like
experimental/, and creating a Protected Branch rule that only allows Delivery to push and merge to those branches.
These experimental branches can get mirrored to ops, and then in order to test we can update the scheduled tasks to use the experimental branch rather than
master. If something goes wrong, we change the task's branch back to
master and re-run it, returning us to the previous, battle-tested behavior.
Exceptions to this strategy would be commands coming from ChatOps, as those execute a trigger with
master explicitly set: https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/release/command.rb#L19. I think we could also add support for this strategy by pulling that from a CI variable.
I think this would be useful for large refactorings like work I'm currently doing for gitlab-org/release-tools#389, and Yorick's work on the API-based releases (which is behind a feature flag already, but this may provide additional safeguards).