Move QA triggers from deployer to release-tools
Context
QA jobs on the deployer pipeline trigger smoke/reliable tests after staging and canary have been updated with the new code.
gstg-qa-stage |
gprd-cny-qa-stage |
---|---|
![]() |
![]() |
Proposal
As part of &361 and in order to centralize the deployment logic into release-tools, lets move the QA jobs included in gstg-qa
and gprd-cny-qa
stages from the deployer pipeline to the release-tools pipeline.
- Example of a release-tools pipeline https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/568937
- Example of a deployer pipeline https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/569010
Considerations
- QA jobs on staging should be executed:
- After the
coordinated:deploy:staging
stage - Only if the deployment to staging was successful
- Before the
Coordinated:finish:staging
, notification to Slack should only be sent if QA jobs are green.
- After the
- QA jobs on canary should be executed:
- After the
coordinated:deploy:gprd-cny
stage - Only if the deployment to gprd-cny was successful
- Before the
coordianted:finish:staging
, notification to Slack should only be sent if QA jobs are green.
- After the
- We need to make sure the QA jobs are not executed in the deployer when these are triggered from release-tools
- We need to make sure that the work done in #1219 (closed) continues to work to link QA issues to test session issues.
Technical details
Staging QA
- QA issue is created by
gstg-track
job on the deployer pipeline. This job triggers a pipeline in release-tools that creates the QA among other things. The QA issue link is required for the QA jobs
Details of QA issue generation
-
gstg-track-running
is executed before the fleet to track a running environment on staging - Example- This job triggers a pipeline on release-tools that executes
release:track_deployment
withDEPLOY_STATUS
asrunning
. Example- Records a gstg running deployment for GitLab, Omnibus, Gitaly
- Creates a new
gstg
deployment in GitLab, Omnibus and Gitaly Security - Adds
"workflow::staging"
- This job triggers a pipeline on release-tools that executes
-
gstg-track
is executed after the fleet has been updated. Example- This job triggers a pipeline on release-tools that executes
release:track_deployment
withDEPLOY_STATUS
assuccess
. Example- Records a successful gstg deployment for GitLab, Omnibus and Gitaly
- Creates a Sentry deploy
- Creates QA issue
- If the deployment failed
gstg-track-failure
will be executed instead
- This job triggers a pipeline on release-tools that executes
- QA on staging is composed of three jobs:
Job | Example | Triggers a pipeline on | Type | Allowed to fail? | Variables (example) |
---|---|---|---|---|---|
gstg-gitlab-qa-full |
Link | gitlab-org/quality/staging |
Fire and forget | Yes | - "GITLAB_QA_ISSUE_URL=https://gitlab.com/gitlab-org/release/tasks/-/issues/3075" - "DEPLOY_ENVIRONMENT=gstg" - "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4" - "FULL_ONLY"
|
gstg-gitlab-qa-orchestrated |
Link | gitlab-org/gitlab-qa |
Fire and forget | Yes | - "GITLAB_QA_ISSUE_URL=https://gitlab.com/gitlab-org/release/tasks/-/issues/3075" - "DEPLOY_ENVIRONMENT=gstg" - "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4" - "NOTIFY_CHANNEL=qa-staging" - "TOP_UPSTREAM_SOURCE_JOB=https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/5337314" - "RELEASE=dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:14.5.202111030620-c1ef75cba80.fb7ecd0d5b4"]
|
gitlab-qa-smoke |
Link | gitlab-org/quality/staging |
Active waiting | No | - "GITLAB_QA_ISSUE_URL=https://gitlab.com/gitlab-org/release/tasks/-/issues/3075" - "DEPLOY_ENVIRONMENT=gstg" - "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4" - "SMOKE_ONLY"
|
gprd-cny QA
QA on gprd-cny is composed of three jobs:
Job | Example | Triggers a pipeline on | Type | Allowed to fail? | Variables (example) |
---|---|---|---|---|---|
gprd-cny-gitlab-qa-full |
Link | gitlab-org/quality/canary |
Fire and forget | Yes | - "GITLAB_QA_ISSUE_URL=" - "DEPLOY_ENVIRONMENT=gprd-cny" - "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4" - "FULL_ONLY"
|
gprd-cny-gitlab-qa-smoke |
Link | gitlab-org/quality/canary |
Active waiting | No | - GITLAB_QA_ISSUE_URL=" - "DEPLOY_ENVIRONMENT=gprd-cny" - "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4" - "SMOKE_ONLY"
|
gprd-cny-gitlab-qa-smoke-main |
Link | gitlab-org/quality/production |
Active waiting | Yes | - "GITLAB_QA_ISSUE_URL=" - "DEPLOY_ENVIRONMENT=gprd-cny" - "DEPLOY_VERSION=14.5.202111030620-c1ef75cba80.fb7ecd0d5b4" - "SMOKE_ONLY"
|
Questions/considerations:
-
On the canary QA jobs, what is the difference forgprd-cny-gitlab-qa-smoke
andgprd-cny-gitlab-qa-smoke-main
? Both jobs trigger a pipeline using the same variables- The
*-main
one triggers a job https://ops.gitlab.net/gitlab-org/quality/production, see #1490 (comment 722763056)
- The
-
What is theTOP_UPSTREAM_SOURCE_JOB
environment required ongstg-gitlab-qa-orchestrated
?- It refers to the job that triggered the QA pipeline. On deployer project, it'd be the deployer job and on release-tools, it should be the release-tools job
-
QA jobs on staging require the QA issue URL (QA_ISSUE_URL
), this artifact is generated from therake/release
task. We need to find a way to share it with the coordinated pipeline.- We could save the
QA_ISSUE_URL
in the cache and then fetch it in the coordinated pipeline #1490 (comment 724118612)
- We could save the
-
Active waiting
QA jobs trigger a pipeline in a quality project and actively wait for this pipeline to finish by waitingX
seconds. These jobs sound to me like they could be replaced with a bridge-jobs implementation, although it may be easier to just copy/paste the same implementation.- This is a technical detail that should be decided when implementing the changes
Implementation plan
- Save QA URL into the pipeline cache gitlab-org/release-tools!1605 (merged)
- Introduce quality project classes gitlab-org/release-tools!1608 (merged)
-
Implement a class that allows us to trigger a pipeline in QA projects (staging, canary and production) gitlab-org/release-tools!1606 (merged), gitlab-org/release-tools!1622 (merged)
- Ensure all changes are guarded by a feature flag.
-
Add rake tasks and notifier classes for the
coordinated_pipeline/qa
classes gitlab-org/release-tools!1624 (merged) -
Modify the coordinated pipeline CI to execute the rake tasks for staging and canary. All jobs should be allowed to fail. - gitlab-org/release-tools!1628 (merged)
- This one should include bridge-jobs for triggering smoke QA pipelines.
- Adjust the deployer CI pipeline to skip QA jobs when the deployment is triggered from the release-tools. Note that if the deployment is triggered from ChatOps QA should still be included. https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/441
- Testing
- [-] Update documentation https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#scheduled-qa-test-pipelines (if required) => Not required.

Testing

-
Create a new branch with the CI configuration
add-qa-ci-modifications
in ops https://ops.gitlab.net/gitlab-org/release/tools/-/tree/add-qa-ci-modifications - Protect the branch so it can reuse the protected environment variables.
-
Modify the
tag
pipeline schedule to useadd-qa-ci-modifications
https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/74/edit - Enable feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/209/edit
-
Tag coordinated pipeline based on the new branch (make sure to restore it back to
master
, once the pipeline is created) - Validate the following scenarios
QA on staging
-
QA full:
- A pipeline on
gitlab-org/quality/staging
is triggered with the correct variables - The pipeline triggered succeeds
- A pipeline on
-
Orchestrated
- A pipeline on
gitlab/qa
is triggered with the correct variables - The pipeline triggered succeeds
- A pipeline on
-
Smoke
- A pipeline on
gitlab/quality/staging
is triggered with the correct variables - The pipeline triggered succeeds
- A pipeline on
- Coordinated pipeline only waits for the smoke QA pipeline
- Notifications
Deployment | QA | Notification | |
---|---|---|---|
Succeeds | Succeeds | General notification sent | ![]() |
Succeeds | Fails | Qa notification sent |
![]() |
Fails | --- | General notification sent | ![]() |
QA on canary
-
QA full:
- A pipeline on
gitlab-org/quality/canary
is triggered with the correct variables - The pipeline triggered succeeds
- A pipeline on
-
QA smoke
- A pipeline on
gitlab-org/quality/canary
is triggered with the correct variables - The pipeline triggered succeeds
- A pipeline on
-
QA smoke main
- A pipeline on
gitlab-org/quality/production
is triggered with the correct variables - The pipeline triggered succeeds
- A pipeline on
- Coordinated pipeline only waits for the smoke QA pipeline
- Notifications
Deployment | QA | Notification | |
---|---|---|---|
Succeeds | Succeeds | General notification sent | ![]() |
Succeeds | Fails | Qa notification sent |
![]() |
Fails | --- | General notification sent |
![]() |
Clean up
- Remove code that triggers full pipelines - gitlab-org/release-tools!1640 (merged)
- Remove the protected branch from ops
- Delete the branch from ops
What to do if something goes wrong?
If gitlab-org/release-tools!1628 (merged) hasn't been merged:
- Merge gitlab-org/release-tools!1639 (closed)
- Modify https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/74/edit to use
master
- Create a new coordinated pipeline
If gitlab-org/release-tools!1628 (merged) has been merged:
- Revert gitlab-org/release-tools!1628 (merged)
- Merge gitlab-org/release-tools!1639 (closed)
- Create a new coordinated pipeline
Development log
Open up for details
- November 3rd - State of art was updated and a broad implementation plan was suggested
-
November 4th - Different strategies about how to share the
QA_ISSUE_URL
were analyzed. To start with, it might be easier to save the information into the cache -
November 8th
- An MR to save the QA URL was submitted for review gitlab-org/release-tools!1605 (merged)
- An MR introducing classes to trigger quality pipelines was opened gitlab-org/release-tools!1606 (merged)
- Quality confirmed QA projects are agnostic to the upstream pipeline so the transition from deployer to release-tools shouldn't impose any problem #1490 (comment 727103671)
-
November 9th
- gitlab-org/release-tools!1605 (merged) was merged
- MR that introduces quality project classes as submitted and merged gitlab-org/release-tools!1608 (merged)
-
November 23rd
- Classes that trigger quality pipelines were refactored to a simpler approach gitlab-org/release-tools!1606 (merged).
- An MR that extends quality project classes was opened and submitted for review gitlab-org/release-tools!1622 (merged)
-
November 24th
- gitlab-org/release-tools!1622 (merged) was merged
- gitlab-org/release-tools!1606 (merged) was submitted for review
-
December 1st
- Strategy for fetching QA issues changed: Instead of relying in QA, a class was added to search the QA issue based on
DEPLOY_VERSION
and save that information into an artifact. Later QA classes will reuse that artifact to trigger pipelines
- Strategy for fetching QA issues changed: Instead of relying in QA, a class was added to search the QA issue based on
-
December 2nd
- gitlab-org/release-tools!1606 (merged) was merged
- gitlab-org/release-tools!1634 (merged) was reverted
-
December 3th
- gitlab-org/release-tools!1624 (merged) was submitted for review
-
December 6th
- gitlab-org/release-tools!1624 (merged) was merged
- gitlab-org/release-tools!1628 (merged) was modified based on the latest changes
- December 8th
-
December 9th
- Based on the test's positive results, CI changes were merged. No major issue has been reported
-
December 10th
- Testing completed.
Follow ups
- Remove feature flag #2157 (closed)
- Drop the
QA::IssueFinder
class once the Quality issue is created by quality #2140 - Trigger QA orchestrated pipeline using bridge-jobs #2153 (closed)
- Investigate if
GITLAB_QA_ISSUE_URL
is required for gprd-cny QA jobs #2154 - Consider automatically retry QA bridge pipelines if they fail #2156 (closed)