Test usage of default branch on release-toolings
3rd step of &383 (closed)
#1357 (closed) plans to change gitlab-org/gitlab
default branch from master
to main
, ahead of this upcoming change, we need to make sure our processes are still functional.
Purpose of this issue is to test the following processes:
General requirements
Following steps should be completed before the testing
-
On release-tools, add a default_branch
method onProject::GitlabEE
andProject::GitlabCe
to usemaster
ormain
based on a feature flag - gitlab-org/release-tools!1346 (merged) -
Create the feature flag on ops.gitlab.net - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit -
Wait until %13.8 and the regular security release have been completed. Tentative date to start testing: February 2nd 2021 -
Create a temp branch on gitlab-org/gitlab
andgitlab-org/gitlab-foss
calledmain
(This branch is going to be deleted after the testing is completed)-
gitlab-org/gitlab
- https://gitlab.com/gitlab-org/gitlab/-/commits/main -
gitlab-org/gitlab-foss
- https://gitlab.com/gitlab-org/gitlab-foss/-/commits/main
-
-
Protect main
branch on Canonical and Security: Only Delivery team members should be allowed to push/merge
Canonical gitlab-org/gitlab |
Canonical gitlab-org/gitlab-foss |
---|---|
![]() |
![]() |
Security gitlab-org/gitlab |
Security gitlab-org/gitlab-foss |
---|---|
![]() |
![]() |
-
For gitlab-org/gitlab
ensuremain
is propagated to Security and Dev -
For gitlab-org/gitlab-foss
ensuremain
is propagated to Security and Dev
Testing daily deployments
-
Make sure the content of main
is up to date withmaster
-
Pause the auto_deploy:prepare
task https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/73/edit -
Include a commit on main
that is not included onmaster
: Generate a fake changelog and remove a comment fromapp/models/project.rb
so a pipeline is triggered gitlab-org/gitlab@6916f873 -
Ensure Security and Dev are synced -
Enable the feature flag that makes release-tools
using themain
branch. https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit -
Manually trigger the auto_deploy:prepare
branch https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/2998935 -
Ensure auto-deploy branch for gitlab-org/gitlab
are created frommain
and include the commit added tomain
- Note: Auto-deploy branch was created from
main
but it did not include the commit. See #1441 (comment 499831206) for reasoning
- Note: Auto-deploy branch was created from
-
Ensure auto-deploy branch for Omnibus, Charts and CNG are created from master
- Omnibus auto-deploy branch https://gitlab.com/gitlab-org/security/omnibus-gitlab/commits/13-9-auto-deploy-2021020217
- CNG auto-deploy branch https://gitlab.com/gitlab-org/security/charts/components/images/commits/13-9-auto-deploy-2021020217
- Charts auto-deploy branch https://gitlab.com/gitlab-org/security/charts/gitlab/commits/13-9-auto-deploy-2021020217
-
Ensure packages are built correctly -
Ensure packages are deployed successfully to production https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/450255
After the testing has been completed:
-
Disable the feature flag -
Re-enable the auto_deploy:prepare
task
Testing RC's
-
Confirm with Release Managers that is a good time to start testing. Once we have coordination, let's proceed. -
Enable the feature flag that makes release-tools
using themain
branch - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit -
Create an RC version to ensure it builds correctly -
<code data-sourcepos="67:9-67:48">11c7a28acfcc05e7e2db041eb5119cce777e2e40</code> only exists on
main
and notmaster
-
<code data-sourcepos="67:9-67:48">11c7a28acfcc05e7e2db041eb5119cce777e2e40</code> only exists on
/chatops run release tag 13.9.0-rc41 --gitlab-sha=11c7a28acfcc05e7e2db041eb5119cce777e2e40
Job https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3087659
-
Ensure that a stable branch was created for gitlab-org/gitlab
and that the content matches the one frommain
branch: https://gitlab.com/gitlab-org/gitlab/-/commits/13-9-stable-ee. -
Since the stable branch will be deleted, add a screenshot of the commits page for historical purposes. -
Ensure that a stable branch was created for gitlab-org/gitlab-foss
: https://gitlab.com/gitlab-org/gitlab-foss/-/commits/13-9-stable. -
Since the stable branch will be deleted, add a screenshot of the commits page for historical purposes. -
Ensure stable branches are propagated to Security and Dev
Once the testing has been completed:
-
Delete stable branches from Canonical (for both gitlab-org/gitlab
andgitlab-org/gitlab-foss
) -
Delete stable branches from Security (for both gitlab-org/gitlab
andgitlab-org/gitlab-foss
) -
Delete stable branches from Dev (for both gitlab-org/gitlab
andgitlab-org/gitlab-foss
) -
Disable the feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit
Testing security releases
Preparation
-
Create a fake security release tracking issue - https://gitlab.com/gitlab-org/gitlab/-/issues/320810. -
Create two fake security issues and associate them to the fake security release tracking issue -
Ensure the security issues are ready to be processed - Each issue needs to have 4 MRs for this: One targeting
main
and the proper backports
- Each issue needs to have 4 MRs for this: One targeting
-
Ensure the fake release tracking issue is picked up by our tooling by - Adding ~"upcoming security release" label
- Adding a due date (earlier than the actual security release)
-
Enable the feature flag that makes release-tools
using themain
branch. https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit -
Assign all the merge requests to the @gitlab-release-tools-bot
Validation
-
Trigger the security:validate
scheduled task https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/75/edit -
Ensure merge requests targeting main
branch are accepted by our tooling
Early merge process
-
Ensure our tooling recognizes the fake security release process by executing the command on dry-run mode (we don't want to accidentally merge actual security fixes) - https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3113016
/chatops run release merge --security --default-branch --dry-run
-
Proceed to merge the fake security MRs https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3113071
/chatops run release merge --security --default-branch
-
Ensure merge requests are set to MWPS
Security Issue A | Security Issue B |
---|---|
https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1236 | https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1241 |
![]() |
![]() |
-
Ensure merge requests are merged
Tagging a security release
NOTE: Tagging a security release involves compiling the changelog on the default branch, updating stable branches, and creating tags. This can cause inconvenience on some teams and a manual cleanup was required. If we ever need to test again for gitlab-org/gitlab
, we need to 1) Skip the release execution of other projects, probably with a feature flag, and 2) Notify the other teams about it, so they don't be alarmed if they see a tag. See #1441 (comment 508245213) for more details
-
Create a fake stable branch 42-1-stable-ee
ongitlab-org/gitlab
- https://gitlab.com/gitlab-org/gitlab/-/commits/42-1-stable-ee -
Ensure this branch is propagated to Security
anddev
-
Create a fake stable branch 42-1-stable-ee
ongitlab-org/gitlab-foss
- https://gitlab.com/gitlab-org/gitlab-foss/-/commits/42-1-stable -
Ensure this branch is propagated to Security
anddev
-
Merge merge requests targeting 42-1-stable-ee
-
Tag a security release: https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3114458
/chatops run release tag --security 42.1.1 # Or some unrealistic version
-
Ensure tags and changelog were created correctly.
Tags | Changelog on 42-1-stable-ee
|
Changelog on master
|
---|---|---|
![]() |
![]() |
![]() |
Note: See here for the explanation about why the changelog was generated on master
and not in main
Testing security commands
Ensure the following commands consider the main
branch
-
/chatops run release status --security
-
/chatops run mirror status
-
/chatops run release sync_remotes --security
https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3114615 -
/chatops run release close_issues --security
https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/3114776 -
Disable the feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/188/edit
Clean up
Once the testing is completed and satisfactory:
-
Close the security issues and merge requests -
Close the fake security release tracking issue
Albeit fake, a new security release was tagged, which triggered a compilation changelog and a new tag on GitLab, GitLab FOSS, Gitaly, Omnibus, CNG, and Charts. We need to clean those up
Steps to clean up GitLab satellite projects
GitLab
-
On Canonical, Security, and dev remove 42-1-stable-ee
branch -
On Canonical, Security, and dev remove 42.1.1
tag -
Send a merge request to remove the Changelog compilation gitlab-org/gitlab!54101 (merged)
GitLab FOSS
-
On Canonical, Security, and dev remove 42-1-stable
branch -
On Canonical, Security, and dev remove 42.1.1
tag
Omnibus
-
On Canonical, Security, and dev remove 42-1-stable
branch -
On Canonical, Security, and dev remove 42.1.1
tag -
Send a merge request to remove the Changelog compilation gitlab-org/omnibus-gitlab!5011 (merged)
Gitaly
-
On Canonical, Security, and dev remove 42-1-stable
branch -
On Canonical, Security, and dev remove 42.1.1
tag -
Send a merge request to remove the Changelog compilation gitlab-org/gitaly!3130 (merged)
CNG
-
On Canonical, Security, and dev, remove 42-1-stable
branch -
On Canonical, Security, and dev, remove 42.1.1
tag
Charts
-
On Canonical, Security, and dev, remove 5-0-stable
branch -
On Canonical, Security, and dev, remove 42.1.1
tag -
Revert version mapping, Chart version and changelog - Version mapping gitlab-org/charts/gitlab@08b05ea0
- Chart version gitlab-org/charts/gitlab@4f9b8186
- Changelog gitlab-org/charts/gitlab@b9d6088f
Clean up
After all testing has been completed:
-
Unprotect main
forgitlab-org/gitlab
on Canonical, Security and Dev -
Unprotect main
forgitlab-org/gitlab-foss
on Canonical, Security and Dev -
Delete main
fromgitlab-org/gitlab
on Canonical, Security and Dev -
Delete main
fromgitlab-org/gitlab-foss
on Canonical, Security and Dev
Follow-ups
-
main
branch was created as a protected branch, however, no pipelines were triggered for this branch. Based on our CI config rules, pipelines are only executed formaster
, auto-deploy, and stable branches. Before transitioning tomain
(step 5 of #1357 (closed)), we need to make sure pipelines are also executed on this branch - gitlab-org/gitlab#320794 (closed)