Release 14.10
First steps
-
Change #f_upcoming_release
topic with/topic 14.10.0: <link_to_this_issue>
-
Adjust the auto-deploy branch schedule based on the Release Manager's working hours. -
Consider planned PCLs and modify the dates below to accurately reflect the plan of action. -
If there is a Family & Friends day this month, add notes for pausing deployments before the day starts, and unpausing them before the next business day. See the documentation. -
Update this issue with three planned dates for recurring Staging rollback practice. Consider spreading these across timezones to share the knowledge. -
Set Due Date for this Issue to the first practice session
-
-
Check for any deprecations and see if we are possibly affected in our k8s or chef config. with removal: '14.10'
that might affect our configuration - otherwise the next auto-deploy after a monthly release might fail with a deprecation failure
First Staging Rollback Practice - Week of 28th March (Reuben / Graeme)
Date to be Completed:
-
Perform Staging Rollback Practice - Completed by Graeme. -
Set Due date for this issue to the date of the second practice session
Second Staging Rollback Practice - Week of 4th April (John Skarbek)
Date to be Completed:
-
Perform Staging Rollback Practice -
Set Due date for this issue to the date of the third practice session
Third Staging Rollback Practice - Week of 11th April (Reuben)
Date to be Completed:
-
Perform Staging Rollback Practice -
Set Due date for this issue to the date of the release day
23rd March
-
No deployments to gstg and gprd due to soft PCL - gitlab-com/www-gitlab-com!101147 (merged).
24th March
-
Soft PCL ends at 9:00 am UTC on 24th March, so deployments to gstg and gprd can be done after that. -
Pause deployments at the end of day due to Family & Friends day on 25th March. /chatops run auto_deploy pause
28th March
-
Unpause deployments at start of day. Deployments were paused due to F&F day on 25th March. /chatops run auto_deploy unpause
8th April
-
Pause deployments at the end of day due to Family & Friends day on 11th April. /chatops run auto_deploy pause
12th April
-
Unpause deployments at start of day. Deployments were paused due to F&F day on 11th April. /chatops run auto_deploy unpause
Up until the 15th
- Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
- Push any successful deploy to canary into production after some time has passed (preferably 1h).
- Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
- Ask for permission to promote the release in #production - provide the necessary context to the Engineer
- If permission is granted, utilize the following command to initiate an overridden promotion:
/chatops run deploy <VERSION> --production --ignore-production-checks 'deployment approved by on call SRE'
- This will post a comment into this issue and begin the deployment
- Ask the SRE On-Call to respond to the comment with their approval for auditing purposes
15th
If this date is on a weekend, do this work on the next working day
-
Find the latest sha
that made it into production successfully:sha
-
Notify Engineering Managers and developers that this is the sha
that is guaranteed to be released on the 22nd:/chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA>. You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 14.10` Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
19th
If this date is on a weekend, do this work on the last Friday before the 18th.
-
Log latest auto-deploy branch: 14-10-auto-deploy-2022041912 -
Ensure this build makes it through into production -
Grab the sha
from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidatesha
for the release:/chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA> You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 14.10` Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
20th: two working days before the release
If this date is on a Sunday, do this work on the last Friday before the 20th. If it falls on a Friday or Saturday, move it to Thursday.
-
Determine what is the last auto deploy branch to have deployed to production and add it here: 14-10-auto-deploy-2022042000
-
Create a RC version to ensure that the final version builds correctly # In Slack: /chatops run release tag 14.10.0-rc42
This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:
/chatops run release tag 14.10.0-rc42 --gitlab-sha=XXX
This will then use XXX
as the SHA to create the GitLab stable branches.
NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.
-
Verify that the CE stable branch contains the right commits - There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
- The sync commit will have the message "Add latest changes from gitlab-org/gitlab@14-10-stable-ee"
-
Notify Engineering Managers and developers that final candidate has been created: /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/-/commits/14-10-stable-ee You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 14.10` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
-
Verify that the RC has been deployed to the pre environment - Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged.
A notification will be sent to the
#announcements
channel in Slack when it starts. - If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.
- Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged.
A notification will be sent to the
21st: one day before the release
If this date is on a weekend, do this work on the Friday before that weekend.
-
Confirm that final RC version has passed automated tests -
Ensure tests are green on CE stable branch -
Ensure tests are green on EE stable branch -
Ensure tests are green on Omnibus -
Ensure default and stable branches are synced: /chatops run mirror status
-
-
Tag 14.10.0
:# In Slack: /chatops run release tag 14.10.0
-
Check progress of EE packages build and CE packages build
-
-
Validate 14.10.0
has been deployed to the release environment
Instructions for manual deploy
```sh
# In Slack:
/chatops run deploy 14.10.0-ee.0 --release
```
-
Validate 14.10.0
has been passed automated QA by ensuring therelease-gitlab-qa-smoke
job from the release deploy pipeline is green.
Past this point, no new code can be added to the release that was not included in the final RC.
22nd: release day
Final release is tagged, so any changes will have to initiate a patch release.
-
At 13:00 UTC, post an update about the package building status in #f_upcoming_release
:mega: Packages for 14.10.0 are built and will be published at 13:30UTC
- At 13:30 UTC:
-
⚠ Make sure that neither packages nor the blog post get published earlier than 13:30UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time⚠ -
Publish the packages via ChatOps: # In Slack: /chatops run publish 14.10.0
- If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
-
- At 14:10 UTC:
-
Verify that EE packages appear on packages.gitlab.com
: EE (should contain 14 packages) -
Verify that CE packages appear on packages.gitlab.com
: CE (should contain 13 packages) -
Verify that Docker images appear on hub.docker.com
: EE / CE -
Post an update about the status in #f_upcoming_release
:mega: 14.10.0 is published and publicly available
-
Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC -
Create the 14.10.0
version on version.gitlab.com
-
Release Certification
The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security