Skip to content

Move QA triggers from deployer to release-tools

Context

QA jobs on the deployer pipeline trigger smoke/reliable tests after staging and canary have been updated with the new code.

gstg-qa-stage gprd-cny-qa-stage
Screen_Shot_2021-04-22_at_10.02.35 Screen_Shot_2021-04-22_at_10.02.44

Proposal

As part of &361 and in order to centralize the deployment logic into release-tools, lets move the QA jobs included in gstg-qa and gprd-cny-qa stages from the deployer pipeline to the release-tools pipeline.

Considerations

  • QA jobs on staging should be executed:
    • After the coordinated:deploy:staging stage
    • Only if the deployment to staging was successful
    • Before the Coordinated:finish:staging, notification to Slack should only be sent if QA jobs are green.
  • QA jobs on canary should be executed:
    • After the coordinated:deploy:gprd-cny stage
    • Only if the deployment to gprd-cny was successful
    • Before the coordianted:finish:staging, notification to Slack should only be sent if QA jobs are green.
  • We need to make sure the QA jobs are not executed in the deployer when these are triggered from release-tools
  • We need to make sure that the work done in #1219 (closed) continues to work to link QA issues to test session issues.

Technical details

Staging QA

  1. QA issue is created by gstg-track job on the deployer pipeline. This job triggers a pipeline in release-tools that creates the QA among other things. The QA issue link is required for the QA jobs
Details of QA issue generation
  1. gstg-track-running is executed before the fleet to track a running environment on staging - Example
    • This job triggers a pipeline on release-tools that executes release:track_deployment with DEPLOY_STATUS as running. Example
      • Records a gstg running deployment for GitLab, Omnibus, Gitaly
      • Creates a new gstg deployment in GitLab, Omnibus and Gitaly Security
      • Adds "workflow::staging"
  2. gstg-track is executed after the fleet has been updated. Example
    • This job triggers a pipeline on release-tools that executes release:track_deployment with DEPLOY_STATUS as success. Example
      • Records a successful gstg deployment for GitLab, Omnibus and Gitaly
      • Creates a Sentry deploy
      • Creates QA issue
    • If the deployment failed gstg-track-failure will be executed instead
  1. QA on staging is composed of three jobs:
Job Example Triggers a pipeline on Type Allowed to fail? Variables (example)
gstg-gitlab-qa-full Link gitlab-org/quality/staging Fire and forget Yes - "GITLAB_QA_ISSUE_URL=https://gitlab.com/gitlab-org/release/tasks/-/issues/3075"
- "DEPLOY_ENVIRONMENT=gstg"
- "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"
- "FULL_ONLY"
gstg-gitlab-qa-orchestrated Link gitlab-org/gitlab-qa Fire and forget Yes - "GITLAB_QA_ISSUE_URL=https://gitlab.com/gitlab-org/release/tasks/-/issues/3075"
-"DEPLOY_ENVIRONMENT=gstg"
- "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"
-"NOTIFY_CHANNEL=qa-staging"
- "TOP_UPSTREAM_SOURCE_JOB=https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/5337314"
- "RELEASE=dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"]
gitlab-qa-smoke Link gitlab-org/quality/staging Active waiting No - "GITLAB_QA_ISSUE_URL=https://gitlab.com/gitlab-org/release/tasks/-/issues/3075"
- "DEPLOY_ENVIRONMENT=gstg"
- "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"
- "SMOKE_ONLY"

gprd-cny QA

QA on gprd-cny is composed of three jobs:

Job Example Triggers a pipeline on Type Allowed to fail? Variables (example)
gprd-cny-gitlab-qa-full Link gitlab-org/quality/canary Fire and forget Yes - "GITLAB_QA_ISSUE_URL="
- "DEPLOY_ENVIRONMENT=gprd-cny"
- "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"
- "FULL_ONLY"
gprd-cny-gitlab-qa-smoke Link gitlab-org/quality/canary Active waiting No - GITLAB_QA_ISSUE_URL="
- "DEPLOY_ENVIRONMENT=gprd-cny"
- "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"
- "SMOKE_ONLY"
gprd-cny-gitlab-qa-smoke-main Link gitlab-org/quality/production Active waiting Yes - "GITLAB_QA_ISSUE_URL="
- "DEPLOY_ENVIRONMENT=gprd-cny"
- "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"
- "SMOKE_ONLY"

Questions/considerations:

  • ✅ On the canary QA jobs, what is the difference for gprd-cny-gitlab-qa-smoke and gprd-cny-gitlab-qa-smoke-main? Both jobs trigger a pipeline using the same variables
  • ✅ What is the TOP_UPSTREAM_SOURCE_JOB environment required on gstg-gitlab-qa-orchestrated?
    • It refers to the job that triggered the QA pipeline. On deployer project, it'd be the deployer job and on release-tools, it should be the release-tools job
  • ✅ QA jobs on staging require the QA issue URL (QA_ISSUE_URL), this artifact is generated from the rake/release task. We need to find a way to share it with the coordinated pipeline.
  • ✅ Active waiting QA jobs trigger a pipeline in a quality project and actively wait for this pipeline to finish by waiting X seconds. These jobs sound to me like they could be replaced with a bridge-jobs implementation, although it may be easier to just copy/paste the same implementation.
    • This is a technical detail that should be decided when implementing the changes

Implementation plan

✅ Testing

QA on staging

  • QA full:
    • A pipeline on gitlab-org/quality/staging is triggered with the correct variables
    • The pipeline triggered succeeds
  • Orchestrated
    • A pipeline on gitlab/qa is triggered with the correct variables
    • The pipeline triggered succeeds
  • Smoke
    • A pipeline on gitlab/quality/staging is triggered with the correct variables
    • The pipeline triggered succeeds
  • Coordinated pipeline only waits for the smoke QA pipeline
  • Notifications
Deployment QA Notification
Succeeds Succeeds General notification sent ✅
Succeeds Fails Qa notification sent ✅ Example Link
Fails --- General notification sent ✅

QA on canary

  • QA full:
    • A pipeline on gitlab-org/quality/canary is triggered with the correct variables
    • The pipeline triggered succeeds
  • QA smoke
    • A pipeline on gitlab-org/quality/canary is triggered with the correct variables
    • The pipeline triggered succeeds
  • QA smoke main
    • A pipeline on gitlab-org/quality/production is triggered with the correct variables
    • The pipeline triggered succeeds
  • Coordinated pipeline only waits for the smoke QA pipeline
  • Notifications
Deployment QA Notification
Succeeds Succeeds General notification sent ✅
Succeeds Fails Qa notification sent ✅ Example
Fails --- General notification sent ✅ https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/930742

Clean up

What to do if something goes wrong?

If gitlab-org/release-tools!1628 (merged) hasn't been merged:

  1. Merge gitlab-org/release-tools!1639 (closed)
  2. Modify https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/74/edit to use master
  3. Create a new coordinated pipeline

If gitlab-org/release-tools!1628 (merged) has been merged:

  1. Revert gitlab-org/release-tools!1628 (merged)
  2. Merge gitlab-org/release-tools!1639 (closed)
  3. Create a new coordinated pipeline

Development log

Open up for details

Follow ups

  • Remove feature flag #2157 (closed)
  • Drop the QA::IssueFinder class once the Quality issue is created by quality #2140
  • Trigger QA orchestrated pipeline using bridge-jobs #2153 (closed)
  • Investigate if GITLAB_QA_ISSUE_URL is required for gprd-cny QA jobs #2154
  • Consider automatically retry QA bridge pipelines if they fail #2156 (closed)
Edited by Mayra Cabrera