Release 16.0
First steps
-
Change #f_upcoming_release
topic with/topic 16.0.0: <link_to_this_issue>
-
Check whether the auto-deploy branch schedule fits the Release Managers' working hours. Adjust if needed and make sure any changes are updated in the handbook. -
Consider planned PCLs and modify the dates below to accurately reflect the plan of action. There are none -
If there is a Family & Friends day this month, add notes for pausing deployments before the day starts, and unpausing them before the next business day. See the documentation. There is none scheduled during this rotation -
Update this issue with three planned dates for recurring Staging rollback practice. Consider spreading these across timezones to share the knowledge. -
Set Due Date for this Issue to the first practice session
-
-
Check for any deprecations and see if we are possibly affected in our k8s or chef config. with removal: '16.0'
that might affect our configuration - otherwise the next auto-deploy after a monthly release might fail with a deprecation failure -
Confirm the due date set in the upcoming security release is achievable. Work with AppSec if the date needs to change
First Staging Rollback Practice
2023-04-26 2023-04-27
Date to be Completed: -
Merge https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/522 as requested in gitlab-com/gl-infra/delivery#2872 (comment 1357537210) before beginning the rollback -
Perform Staging Rollback Practice -
Set Due date for this issue to the date of the second practice session
Second Staging Rollback Practice
2023-05-03 2023-05-05
Date to be Completed: -
Perform Staging Rollback Practice -
Set Due date for this issue to the date of the third practice session
Third Staging Rollback Practice
Date to be Completed: 2023-05-11
-
Perform Staging Rollback Practice -
Set Due date for this issue to the date of the release day
26th of April
Deploy a stable build to pre
(optional?)
-
/chatops run deploy 16.0.202304251320-fbe15b731d3.f5235220b62 pre
28th of April
- [-]
Pause auto deploys at AMER end of day in preparation for CR to move the ops instance(gitlab-com/gl-infra/production#8590 (closed)):/chatops run auto_deploy pause
This CR was delayed to prioritize deployments.
1st May
- [-]
At 06:30 UTC, confirm that CR gitlab-com/gl-infra/production#8590 (closed) has been completed or rolled back. - [-]
Unpause auto deploys:/chatops run auto_deploy unpause
. This CR was delayed to prioritize deployments.
3rd
Deploy a stable build to pre
-
/chatops run deploy 16.0.202305030020-08cc692e46a.27d1bff8675 pre
5th May
-
Pause auto deploys in preparation for gitlab-com/gl-infra/production#10855 (closed) and gitlab-com/gl-infra/production#8590 (closed): /chatops run auto_deploy pause
8th May
-
Unpause auto deploys if they were paused: /chatops run auto_deploy unpause
10th
Deploy a stable build to pre
-
/chatops run deploy 16.0.202305101100-5e900b162b1.9163b99801c pre
Up until the 11th
- Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
- Push any successful deploy to canary into production after some time has passed (preferably 1h).
- Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
- Ask for permission to promote the release in #production - provide the necessary context to the Engineer
- If permission is granted, utilize the following command to initiate an overridden promotion:
/chatops run deploy <VERSION> gprd --ignore-production-checks 'deployment approved by on call SRE'
- This will post a comment into this issue and begin the deployment
- Ask the SRE On-Call to respond to the comment with their approval for auditing purposes
11th
If this date is on a weekend, do this work on the next working day
-
Pause autodeploy by 02:00 UTC for gitlab-com/gl-infra/production#9630 (closed) /chatops run auto_deploy pause
-
Unpause autodeploy when the CR is complete (should be by 18:00 UTC). /chatops run auto_deploy unpause
-
Find the latest sha
that made it into production successfully:5e900b162b1
-
Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute
. -
Notify Engineering Managers and developers that this is the sha
that is guaranteed to be released on the 22nd:/chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/commits/5e900b162b1. You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.0` Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
12th 15th
If this date is on a weekend, do this work on the last Friday before the 18th.
-
Log latest auto-deploy branch: 16.0.202305151105-44865a2c9af.50c10502dcb
-
Ensure this build makes it through into production -
Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute
. -
Grab the sha
from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidatesha
for the release:/chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/-/commits/44865a2c9af You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.0` Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
16th
-
11:00 UTC - Pause auto-deploy or lock gstg-cny
to prevent migrations/deployments for gitlab-com/gl-infra/production#9630 (closed). -
~18:00 UTC - Unpause auto-deploy or unlock gstg-cny
During AMER time, choose the final deployment, deploy it to pre
.
-
/chatops run deploy 16.0.202305160805-4e860c046f2.b3511eb83a9 pre
two three working days before the release
17th: If this date is on a Sunday, do this work on the last Friday before the 20th. If it falls on a Friday or Saturday, move it to Thursday.
-
Determine what is the last auto deploy branch to have deployed to productionpre and add it here:16.0.202305160805-4e860c046f2.b3511eb83a9
-
If you plan to use the latest commit deployed to production to create the RC, make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute
. -
Create a RC version to ensure that the final version builds correctly # In Slack: /chatops run release tag 16.0.0-rc42
This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:
/chatops run release tag 16.0.0-rc42 --gitlab-sha=4e860c046f2
This will then use XXX
as the SHA to create the GitLab stable branches.
NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.
-
Verify that the CE stable branch contains the right commits - There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
- The sync commit will have the message "Add latest changes from gitlab-org/gitlab@16-0-stable-ee"
-
Verify that the pipelines are green # In Slack: /chatops run release status 16.0.0-rc42
-
Notify Engineering Managers and developers that final candidate has been created: /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/-/commits/16-0-stable-ee You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.0` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
-
Verify that the RC has been deployed to the pre environment - Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged.
A notification will be sent to the
#announcements
channel in Slack when it starts. - If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.
- Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged.
A notification will be sent to the
19th 18th: one two day before the release, AMER hours
If this date is on a weekend, do this work on the Friday before that weekend.
-
Confirm that final RC version has passed automated tests -
Ensure tests are green on CE stable branch -
Ensure tests are green on EE stable branch -
Ensure tests are green on Omnibus -
Ensure default and stable branches are synced: /chatops run mirror status
-
-
Tag 16.0.0
:# In Slack: /chatops run release tag 16.0.0
-
Check progress of EE packages build and CE packages build
-
-
Validate 16.0.0
has been deployed to the release environment
Instructions for manual deploy
```sh
# In Slack:
/chatops run deploy 16.0.0-ee.0 release
```
-
Validate 16.0.0
has been passed automated QA by ensuring therelease-gitlab-qa-smoke
job from the release deploy pipeline is green.
Past this point, no new code can be added to the release that was not included in the final RC.
19th
-
If tagging of 16.0 has completed, and there are no incidents or other blockers, consider doing a 15.11 patch release.
22nd: release day
Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.
-
At 13:00 UTC, post an update about the package building status in #f_upcoming_release
:mega: Packages for 16.0.0 are built and will be published at 13:10UTC
- At 13:10 UTC:
-
⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time⚠ -
Publish the packages via ChatOps: # In Slack: /chatops run publish 16.0.0
- If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
-
- At 14:10 UTC:
-
Verify the check-packages
job completes successfully on the EE Pipeline -
Verify the check-packages
job completes successfully on the CE Pipeline -
Verify that Docker images appear on hub.docker.com
: EE / CE -
Post an update about the status in #f_upcoming_release
:mega: 16.0.0 is published and publicly available
-
Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC -
Adjust the protected branch rules for the current version -
Unprotect the 15-11-stable-ee
branch -
Add a protected branch rule to allow maintainers to merge into the 16-0-stable-ee
branch
-
-
Create the 16.0.0
version on version.gitlab.com
-
Release Certification
The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security