Test usage of default branch on release-toolings

3rd step of &383 (closed)

#1357 (closed) plans to change gitlab-org/gitlab default branch from master to main, ahead of this upcoming change, we need to make sure our processes are still functional.

Purpose of this issue is to test the following processes:

• Daily deployments
• Release candidates
• Security releases

General requirements

Following steps should be completed before the testing

• On release-tools, add a default_branch method on Project::GitlabEE and Project::GitlabCe to use master or main based on a feature flag - gitlab-org/release-tools!1346 (merged)
• Create the feature flag on ops.gitlab.net - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit
• Wait until %13.8 and the regular security release have been completed. Tentative date to start testing: February 2nd 2021
• Create a temp branch on gitlab-org/gitlab and gitlab-org/gitlab-foss called main (This branch is going to be deleted after the testing is completed)
  • gitlab-org/gitlab - https://gitlab.com/gitlab-org/gitlab/-/commits/main
  • gitlab-org/gitlab-foss - https://gitlab.com/gitlab-org/gitlab-foss/-/commits/main
• Protect main branch on Canonical and Security: Only Delivery team members should be allowed to push/merge

Canonical gitlab-org/gitlab	Canonical gitlab-org/gitlab-foss

Security gitlab-org/gitlab	Security gitlab-org/gitlab-foss

• For gitlab-org/gitlab ensure main is propagated to Security and Dev
  • Security: https://gitlab.com/gitlab-org/security/gitlab/-/commits/main
  • Dev: https://dev.gitlab.org/gitlab/gitlab-ee/-/commits/main
• For gitlab-org/gitlab-foss ensure main is propagated to Security and Dev
  • Security: https://gitlab.com/gitlab-org/security/gitlab-foss/-/commits/main
  • Dev: https://dev.gitlab.org/gitlab/gitlabhq/-/commits/main

Testing daily deployments

• Make sure the content of main is up to date with master
• Pause the auto_deploy:prepare task https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/73/edit
• Include a commit on main that is not included on master: Generate a fake changelog and remove a comment from app/models/project.rb so a pipeline is triggered gitlab-org/gitlab@6916f873
• Ensure Security and Dev are synced
• Enable the feature flag that makes release-tools using the main branch. https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit
• Manually trigger the auto_deploy:prepare branch https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/2998935
• Ensure auto-deploy branch for gitlab-org/gitlab are created from main and include the commit added to main
  • Note: Auto-deploy branch was created from main but it did not include the commit. See #1441 (comment 499831206) for reasoning
• Ensure auto-deploy branch for Omnibus, Charts and CNG are created from master
  • Omnibus auto-deploy branch https://gitlab.com/gitlab-org/security/omnibus-gitlab/commits/13-9-auto-deploy-2021020217
  • CNG auto-deploy branch https://gitlab.com/gitlab-org/security/charts/components/images/commits/13-9-auto-deploy-2021020217
  • Charts auto-deploy branch https://gitlab.com/gitlab-org/security/charts/gitlab/commits/13-9-auto-deploy-2021020217
• Ensure packages are built correctly
  • https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/184920
• Ensure packages are deployed successfully to production https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/450255

After the testing has been completed:

• Disable the feature flag
• Re-enable the auto_deploy:prepare task

Testing RC's

• Confirm with Release Managers that is a good time to start testing. Once we have coordination, let's proceed.
• Enable the feature flag that makes release-tools using the main branch - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit
• Create an RC version to ensure it builds correctly
  • <code data-sourcepos="67:9-67:48">11c7a28acfcc05e7e2db041eb5119cce777e2e40</code> only exists on main and not master

/chatops run release tag 13.9.0-rc41 --gitlab-sha=11c7a28acfcc05e7e2db041eb5119cce777e2e40 

Job https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3087659

• Ensure that a stable branch was created for gitlab-org/gitlab and that the content matches the one from main branch: https://gitlab.com/gitlab-org/gitlab/-/commits/13-9-stable-ee.
• Since the stable branch will be deleted, add a screenshot of the commits page for historical purposes.
• Ensure that a stable branch was created for gitlab-org/gitlab-foss: https://gitlab.com/gitlab-org/gitlab-foss/-/commits/13-9-stable.
• Since the stable branch will be deleted, add a screenshot of the commits page for historical purposes.
• Ensure stable branches are propagated to Security and Dev

Once the testing has been completed:

• Delete stable branches from Canonical (for both gitlab-org/gitlab and gitlab-org/gitlab-foss)
• Delete stable branches from Security (for both gitlab-org/gitlab and gitlab-org/gitlab-foss)
• Delete stable branches from Dev (for both gitlab-org/gitlab and gitlab-org/gitlab-foss)
• Disable the feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit

Testing security releases

Preparation

• Create a fake security release tracking issue - https://gitlab.com/gitlab-org/gitlab/-/issues/320810.
• Create two fake security issues and associate them to the fake security release tracking issue
  • https://gitlab.com/gitlab-org/security/gitlab/-/issues/341
  • https://gitlab.com/gitlab-org/security/gitlab/-/issues/343
• Ensure the security issues are ready to be processed
  • Each issue needs to have 4 MRs for this: One targeting main and the proper backports
• Ensure the fake release tracking issue is picked up by our tooling by
  • Adding ~"upcoming security release" label
  • Adding a due date (earlier than the actual security release)
• Enable the feature flag that makes release-tools using the main branch. https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit
• Assign all the merge requests to the @gitlab-release-tools-bot

Validation

• Trigger the security:validate scheduled task https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/75/edit
  • https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3112968
• Ensure merge requests targeting main branch are accepted by our tooling

Early merge process

• Ensure our tooling recognizes the fake security release process by executing the command on dry-run mode (we don't want to accidentally merge actual security fixes) - https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3113016

/chatops run release merge --security --default-branch --dry-run

• Proceed to merge the fake security MRs https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3113071

/chatops run release merge --security --default-branch

• Ensure merge requests are set to MWPS

Security Issue A	Security Issue B
https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1236	https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1241

• Ensure merge requests are merged
  • https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1236
  • https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1241

Tagging a security release

NOTE: Tagging a security release involves compiling the changelog on the default branch, updating stable branches, and creating tags. This can cause inconvenience on some teams and a manual cleanup was required. If we ever need to test again for gitlab-org/gitlab, we need to 1) Skip the release execution of other projects, probably with a feature flag, and 2) Notify the other teams about it, so they don't be alarmed if they see a tag. See #1441 (comment 508245213) for more details

• Create a fake stable branch 42-1-stable-ee on gitlab-org/gitlab- https://gitlab.com/gitlab-org/gitlab/-/commits/42-1-stable-ee
• Ensure this branch is propagated to Security and dev
  • Security: https://gitlab.com/gitlab-org/security/gitlab/-/commits/42-1-stable-ee
  • Dev: https://dev.gitlab.org/gitlab/gitlab-ee/-/commits/42-1-stable-ee
• Create a fake stable branch 42-1-stable-ee on gitlab-org/gitlab-foss- https://gitlab.com/gitlab-org/gitlab-foss/-/commits/42-1-stable
• Ensure this branch is propagated to Security and dev
  • Security: https://gitlab.com/gitlab-org/security/gitlab-foss/-/commits/42-1-stable
  • Dev: https://dev.gitlab.org/gitlab/gitlabhq/-/commits/42-1-stable
• Merge merge requests targeting 42-1-stable-ee
  • https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1237
  • https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1242
• Tag a security release: https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3114458

/chatops run release tag --security 42.1.1 # Or some unrealistic version

• Ensure tags and changelog were created correctly.

Tags	Changelog on 42-1-stable-ee	Changelog on master

Note: See here for the explanation about why the changelog was generated on master and not in main

Testing security commands

Ensure the following commands consider the main branch

• /chatops run release status --security
• /chatops run mirror status
• /chatops run release sync_remotes --security https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3114615
• /chatops run release close_issues --security https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3114776
• Disable the feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit

Clean up

Once the testing is completed and satisfactory:

• Close the security issues and merge requests
• Close the fake security release tracking issue

Albeit fake, a new security release was tagged, which triggered a compilation changelog and a new tag on GitLab, GitLab FOSS, Gitaly, Omnibus, CNG, and Charts. We need to clean those up

Steps to clean up GitLab satellite projects

GitLab

• On Canonical, Security, and dev remove 42-1-stable-ee branch
• On Canonical, Security, and dev remove 42.1.1 tag
• Send a merge request to remove the Changelog compilation gitlab-org/gitlab!54101 (merged)

GitLab FOSS

• On Canonical, Security, and dev remove 42-1-stable branch
• On Canonical, Security, and dev remove 42.1.1 tag

Omnibus

• On Canonical, Security, and dev remove 42-1-stable branch
• On Canonical, Security, and dev remove 42.1.1 tag
• Send a merge request to remove the Changelog compilation gitlab-org/omnibus-gitlab!5011 (merged)

Gitaly

• On Canonical, Security, and dev remove 42-1-stable branch
• On Canonical, Security, and dev remove 42.1.1 tag
• Send a merge request to remove the Changelog compilation gitlab-org/gitaly!3130 (merged)

CNG

• On Canonical, Security, and dev, remove 42-1-stable branch
• On Canonical, Security, and dev, remove 42.1.1 tag

Charts

• On Canonical, Security, and dev, remove 5-0-stable branch
• On Canonical, Security, and dev, remove 42.1.1 tag
• Revert version mapping, Chart version and changelog
  • Version mapping gitlab-org/charts/gitlab@08b05ea0
  • Chart version gitlab-org/charts/gitlab@4f9b8186
  • Changelog gitlab-org/charts/gitlab@b9d6088f

Clean up

After all testing has been completed:

• Unprotect main for gitlab-org/gitlab on Canonical, Security and Dev
• Unprotect main for gitlab-org/gitlab-foss on Canonical, Security and Dev
• Delete main from gitlab-org/gitlab on Canonical, Security and Dev
• Delete main from gitlab-org/gitlab-foss on Canonical, Security and Dev

Follow-ups

• main branch was created as a protected branch, however, no pipelines were triggered for this branch. Based on our CI config rules, pipelines are only executed for master, auto-deploy, and stable branches. Before transitioning to main (step 5 of #1357 (closed)), we need to make sure pipelines are also executed on this branch - gitlab-org/gitlab#320794 (closed)