Replace the release instance with GitLab QA update scenario in the release process

Context

Release instance's role in our release process

In the handbook, the release instance (release.gitlab.net) is mentioned with the purpose of "Deploying self-managed releases", where we deploy Final monthly, patch and security releases. So, we test upgrade paths between self-managed releases on the release instance.

In the release runbook, the release instance is mentioned in our release pipeline as below:

  • "An Omnibus pipeline on dev is triggered as a result to build the package, Ubuntu-20.04-staging is the job that triggers the deploy to the release environment:" (note that "release environment" in this context means the release instance)
  • In the release pipelines, there is a job to validate the deployment to the release instance succeeds:

Problems with the release instance

  • Maintenance: the release instance (release.gitlab.net) is a GitLab Omnibus setup running on a single virtual machine (VM). Maintaining it requires additional efforts from the Delivery group to upgrade the host's OS, security review, release tasks to deploy and validate, etc. Some recent events:
  • Deployment: the way the release instance is deployed at the moment does not follow any normal process. The deployments are triggered by an Omnibus package build job, which is very hidden and not well known even by engineers.

Proposal

In Evaluate the role of the release instance (rele... (#21048 - closed), we evaluated the need of the release instance. From the evaluation, we decided that we should replace the release instance with GitLab QA update scenario in the release process (monthly + patch).

I performed a test and it showed that the QA cover all three update paths:

  • patch release -> patch release
  • patch release -> monthly release
  • monthly release -> patch release

So, it is possible to use the GitLab QA update scenario.

Release package test in the release pipeline

The QA step should replace 1:1 to the deployment to the release instance before a release. As you can see in the flowcharts below, the with QA job, now everything is on the release pipeline, give the release managers more visibility into the process.

(The steps in the yellow boxes are for doing one task: test the update path from the current latest version to the upcoming one)

Current:

flowchart LR
    subgraph release pipeline
    A(Omnibus is tagged. \n Tagging pipeline runs)
    end
    A --> B(Ubuntu-20.04-staging job \n in Omnibus pipeline triggers \n a deployer pipeline)
    B --> C(Deploy to release instance)
    C -. asyn .-> D(security_release_tag:verify_release_deploy \n job in release pipeline validates \n the GitLab version on the release instance)
    subgraph release pipeline
    D --> E(Other verification steps)
    E --> F(Publishing)
    end
    style A stroke:#f66,stroke-width:2px
    style F stroke:#f66,stroke-width:2px
    style B stroke:#ff0,stroke-width:2px
    style C stroke:#ff0,stroke-width:2px
    style D stroke:#ff0,stroke-width:2px

Future:

flowchart LR
    subgraph release pipeline
    direction LR
    A(Omnibus is tagged. Tagging pipeline runs)
    A --> B(Run Test::Omnibus::UpdateFromPrevious)
    B --> E(Other verification steps)
    E --> F(Publishing)
    end
    style A stroke:#f66,stroke-width:2px
    style F stroke:#f66,stroke-width:2px
    style B stroke:#ff0,stroke-width:2px

Implementation Tasks

  • Add the QA jobs to the release pipelines

    • Note that, the deployment to the release instance is between tagging and publishing the package, so the QA job should be at the same place. It means, at that time, the new package has not been released yet, i.e. not on DockerHub but on the dev registry (e.g. 17.10.1 Omnibus Docker image is at dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17-10-1-ee-0, or the tag 17.10.1-ee.0 is also accepted)

  • Add a deployment to the release environment using the Deployment API, to adapt this step in our auto-deploy process:

    * `release` is for the final version that is going out for self-managed users. When an MR is deployed to `release`,
  the `released::published` label is applied to the MR to indicate that it will be included
  in the upcoming self-managed release.

  • Update document

  • Deprecate the release instance:

    • Remove the jobs validating the deployment to the release instance
    • Remove the trigger to deploy to the release instance
    • Decommission the instance
    • Remove related infrastructure automation:
      • Terraform
      • Chef
      • etc.
Edited by Dat Tang