Release CE, EE, and Omnibus using the API

This adds support for releasing CE, EE, and Omnibus using just our APIs; instead of cloning a Git repository and using that.

The new code is set up in such a way that release classes are only concerned with the component they belong to, instead of also triggering the release of other components (such as CE triggering Omnibus CE). This makes it easier to opt-out of certain components, test them, maintain them, and understand them.

In this new setup there also isn't a base release class. Instead, there is the module ReleaseTools::PublicRelease::Release, acting as an interface for the release classes. This module requires a few methods to be implemented and provides a few default methods. It does not impose any requirements on the actual steps performed for a release, nor does it offer any hooks of any kind. Instead, release classes define a few required methods (with "execute" being the entry point), and decide for themselves what steps exactly they want to run. The resulting setup is easier to work with, test, maintain, and understand.

Dry-runs are not implemented/supported, as doing so would introduce separate code paths that only run when performing a dry-run. This gives the illusion of being able to test release code, when in reality the real code is not running. Using the API also means that a dry-run would turn (almost) everything into a noop; making a dry-run rather useless. Instead of using a dry-run, testing should be performed by running a real release in an isolated environment of some sort. Issue gitlab-com/gl-infra/delivery#831 discusses a proposal for implementing such a setup.

Security releases are handled by automatically switching to the security project paths, based on the return value of SharedStatus.security_release?.

CE and EE releases are handled by the same class. This is because a CE release always depends on changes introduced by an EE release. The class GitlabRelease is tasked with releasing both CE and EE. This class also makes sure to wait for a Merge Train sync before tagging CE, ensuring we don't tag and release outdated CE code.

See gitlab-com/gl-infra/delivery#843 (closed) for more information

TODO

• Implement code for GitLab EE
• Implement code for Omnibus
• Unit test CE code
• Unit test EE code
• Unit test Omnibus code
• Run CE release using a mirror
• Run EE release using a mirror
• Run Omnibus release using a mirror
• Enable new Omnibus release code using a feature flag
  • We can't use a flag for CE and EE as the old release code also triggers other releases. The API setup in turn can't replicate all this behaviour, at least not without introducing quite a bit of new code (which we'd also have to test).
  • CE, EE and Omnibus have all been tested in isolation, so only enabling Omnibus API code with a flag for now (not using the CE and EE API code) is good enough for the time being
• Update MR description once the changes have been fleshed out better
• Use different project paths for security releases
• Only compile changelogs for X.Y.Z releases; not RCs and auto-deploys
• For Omnibus, don't compile the changelog when releasing EE Omnibus; only when releasing CE Omnibus
• EE, and Omnibus stable branches should be created using a configurable source branch/ref, instead of always using "master"
  • This is needed since releases will be created based on the latest deployed commit, not the latest commit on master
  • Default to "master" if no ref can be found
• Add class/method that given a list of projects returns a Hash mapping the project names to the SHA of the latest production deployments
• Omnibus releases should create the CE and EE tags from the same branch, with the correct VERSION file content
• Add tests for the new implementation of Version#to_normalized_version
• Don't compile changelogs for CE. Due to EE being synced with the merge train, this is not necessary
• CE stable branches must be created from the previous stable branch
  • We must make sure that these branches are updated with the code from the corresponding EE stable branch, before tagging. Right now this sort of works by coincidence because we create stable branches ahead of time
• Re-test Omnibus releases using a fork
• Re-test CE and EE releases using a fork
• Test the logic for waiting for a merge train sync
• Test if the release process is idempotent (e.g. it skips already compiled changelogs)
• When creating stable branches, don't retrieve deployments if the branch already exists. Deployments may not be tracked for security mirrors, so always fetching this data would prevent security releases from happening
  • Alternatively: force the code for getting deployments to always operate on the public projects, even for security releases
• Add logging to the release classes
• Retry API call failures
• Remove changelog date exclusion option