Idea: Composable Releases - Release platform
🛣 Composable Releases - Release platform
🌎 Overview and motivation
As GitLab continues to scale, the need for scalable release automation also becomes necessary. Rather than continuing to build out bespoke tooling tailored to GitLab's specific release procedures, we have an opportunity to build a release platform, with a composable toolset for composing releases, possibly right into the GitLab product.
We are building in the wrong direction. We cannot use release-tools and chatops for anything else. Other teams can make contributions to each, and they do when they really need to, but the projects are not the most approachable. That might be ok for GitLab product releases to an extent. But we should consider what other projects need for releasing.
This is a very early idea with plenty of holes and a lot to understand before it should be considered as a potential direction.
📘 How it might work
A release would be defined and composed using a YAML file. You define things such as:
- Which projects to tag.
- Where to publish packages.
- The cadence or pressure to trigger release types (major, minor, patch).
- Which changes to pull into each release based on label or other rules.
- Creating blog post with a defined template.
- There is a place/way for custom scripts to be inserted between any steps to do things like adding versions to version.gitlab.com via API requests.
The YAML parsing logic could potentially be an extension of ci pipelines with a custom set of rules and definitions in the yaml.
We could imagine something like this (pseudocode):
release_type: major
trigger: cadence
cadence: yearly
rule: May
blog_post:
template: https://gitlab.com/gitlab-org/gitlab/.gitlab/blog_templates/major.erb.html
release_type: minor
trigger: cadence
cadence: monthly
rule: 3rd Thursday
blog_post:
template: https://gitlab.com/gitlab-org/gitlab/.gitlab/blog_templates/minor.erb.html
release_type: patch
trigger: pressure
rule:
quantity: 10
rule:
quantity: 3
labels: ['S2', 'S1']
rule:
quantity: 2
labels ['S1']
blog_post:
template: https://gitlab.com/gitlab-org/gitlab/.gitlab/blog_templates/patch.erb.html
component:
name: GitlabEe
repository: git@gitlab.com:/gitlab-org/gitlab
component:
name: Gitlab-Pages
repository: git@gitlab.com:/gitlab-org/gitlab-pages
component:
name: Gitaly
repository: git@gitlab.com:/gitlab-org/gitaly
component:
name: Omnibus-Gitlab
repository: https://gitlab.com/gitlab-org/omnibus-gitlab
component:
name: CNG
repository: git@gitlab.com:/gitlab-org/build/CNG
build:
trigger: <omnibus pipeline?>
publish_locations:
dockerhub:
url: ...
token: $DOCKERHUB_TOKEN
custom_step:
when: after_publish
script: https://gitlab.com/gitlab-org/release-tools/scripts/add_versions_api.rb
args: '15.4.2, 15.3.6, 15.2.8'
Pros
- More composable releases
- Generic aspects that are true for most releases (in and outside of GitLab) are abstracted away (think tagging and publishing), so only the truly custom aspects are the code that engineers need to pay attention to.
- Creates a marketable product in addition to an internal tool
Cons and risks
- We would be starting from scratch on a tool that is not trivial in design and complexity.
- It could take a long time to build, thus delaying the payoff.
- There is risk that we are investing in the wrong solution.
- As we strive to build a platform style tool, we may find that our customization just won't fit into such a mold.
☁ Unknowns and questions
Note this section is a collection of questions I had for myself when first thinking about this idea.
- How do we model a package?
- What does it mean to generate a release?
- I need to gain a better view of how distribution fits in and functions:
- What triggers a package build
- What goes into the package and why?
- What are the variables?
- What do you need to create a release:
- For GitLab?
- For any generic project?
- What are the common parts, what is unique?
- How does a team configure and maintain a release structure?
- How can it be automated with no manual steps?
- What usually prevents full automation?
📚 Alternatives
🔧 Keep doing what we are doing
Pros
- As long as we build release pipelines with somewhat generic services, we can reuse them for varying release types.
- We are building in a way and with tools our team is comfortable with.
- We already have the foundation built, it's just a matter of continuing forward.
Cons and risks
- If our release process continues to grow and change, the release pipeline can become more brittle with more pieces to maintain.
- While the pipeline lives in a collection of files, tracing it requires looking into a rake task and then collections of services and foundational code that starts to contain lots of Delivery team domain expertise.
- What happens if we need to change direction and our release pipeline no longer can support our needs?