Release 14.3
First steps
-
Change #f_upcoming_release
topic with/topic 14.3.0: <link_to_this_issue>
-
Modify the dates below to accurately reflect the plan of action -
If there is a Family & Friends day this month, add notes for pausing deployments before the day starts, and unpausing them before the next business day. See the documentation.
Aug 26 - end of AMER shift
-
Pause auto_deploy for Family & Friends day on Aug 27
Aug 29 (Sunday)
-
unpause auto_deploy
Up until the 15th
- Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
- Push any successful deploy to canary into production after some time has passed (preferably 1h).
- Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
- Ask for permission to promote the release in #production - provide the necessary context to the Engineer
- If permission is granted, utilize the following command to initiate an overridden promotion:
/chatops run deploy <VERSION> --production --ignore-production-checks 'deployment approved by on call SRE'
- This will post a comment into this issue and begin the deployment
- Ask the SRE On-Call to respond to the comment with their approval for auditing purposes
15th
If this date is on a weekend, do this work on the next working day
-
Find the latest sha
that made it into production successfully:sha
-
Notify Engineering Managers and developers that this is the sha
that is guaranteed to be released on the 22nd:/chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA> Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`"
16th
If this date is on a weekend, do this work on the last Friday before the 18th.
-
Log latest auto-deploy branch: 14-3-auto-deploy-2021091616
-
Ensure this build makes it through into production -
Grab the sha
from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidatesha
for the release:/chatops run notify ":mega: This is the _candidate_ commit to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/commits/<SHA> You can check the status of an MR using the following chatops command: `/chatops run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/12345 14.3` Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means: * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release` * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
-
IMPORTANT lock canary for deployments because we have a hard PCL on 17th and over the weekend.
20th: two working days before the release
If this date is on a Sunday, do this work on the last Friday before the 20th. If it falls on a Friday or Saturday, move it to Thursday.
** As on 17th we had a soft PCL, we didn't deploy for 3 days now, so expect a big diff to be deployed EMEA morning and try to make sure it gets through as soon as possible. **
-
unlock canary first thing in the morning -
Determine what is the last auto deploy branch to have deployed to production and add it here: 14-3-auto-deploy-2021092003 -
Create a RC version to ensure that the final version builds correctly # In Slack: /chatops run release tag 14.3.0-rc42
This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:
/chatops run release tag 14.3.0-rc42 --gitlab-sha=XXX
This will then use XXX
as the SHA to create the GitLab stable branches.
NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.
-
Verify that the CE stable branch contains the right commits - There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
- The sync commit will have the message "Add latest changes from gitlab-org/gitlab@14-3-stable-ee"
-
Notify Engineering Managers and developers that final candidate has been created: /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 22nd. https://gitlab.com/gitlab-org/security/gitlab/-/commits/14-3-stable-ee You can check the status of an MR using the following chatops command: `/chatops run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/12345 14.3` * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"
-
Verify that the RC has been deployed to the pre environment
21st: one day before the release
If this date is on a weekend, do this work on the Friday before that weekend.
-
Confirm that final RC version has passed automated tests -
Ensure tests are green on CE stable branch -
Ensure tests are green on EE stable branch -
Ensure tests are green on Omnibus -
Ensure default and stable branches are synced: /chatops run mirror status
-
-
Tag 14.3.0
:# In Slack: /chatops run release tag 14.3.0
-
Check progress of EE packages build and CE packages build
-
-
Validate 14.3.0
has been deployed to the release environment
Instructions for manual deploy
```sh
# In Slack:
/chatops run deploy 14.3.0-ee.0 --release
```
-
Validate 14.3.0
has been passed automated QA by ensuring therelease-gitlab-qa-smoke
job from the release deploy pipeline is green.
Past this point, no new code can be added to the release that was not included in the final RC.
22nd: release day
Final release is tagged, so any changes will have to initiate a patch release.
-
At 13:00 UTC, post an update about the package building status in #f_upcoming_release
:mega: Packages for 14.3.0 are built and will be published at 13:30UTC
- At 13:30 UTC:
-
⚠ Make sure that neither packages nor the blog post get published earlier than 13:30UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time⚠ -
Publish the packages via ChatOps: # In Slack: /chatops run publish 14.3.0
- If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
-
- At 14:10 UTC:
-
Verify that EE packages appear on packages.gitlab.com
: EE (should contain 14 packages) -
Verify that CE packages appear on packages.gitlab.com
: CE (should contain 13 packages) -
Verify that Docker images appear on hub.docker.com
: EE / CE -
Post an update about the status in #f_upcoming_release
:mega: 14.3.0 is published and publicly available
-
Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC -
Create the 14.3.0
version on version.gitlab.com
-