Skip to content

Removes dynamic_release_date FF from monthly template

Mayra Cabrera requested to merge remove-dynamic-release-date-ff-from-monthly-template into master

What does this MR do and why?

Removes dynamic_release_date FF from monthly template

Starting with 16.6, the release date will be a dynamic one, instead of being on the 22nd it is going to be on the third Thursday of each month. This commit removes the old logic from the monthly template, and makes it always use the dynamic one.

Related to gitlab-com/gl-infra/delivery#19638 (closed)

Testing

  • 16.6
Click to expand

First steps

First Staging Rollback Practice

Date to be Completed:

Second Staging Rollback Practice

Date to be Completed:

Hot patching production practice

Date to be Completed:

The work here is to be done by any delivery team member, excluding the current release managers

  • @name
  • Perform hot patching Practice
  • Hot Patching practice is documented: <link to comment with screenshots>
  • Set Due date for this issue to the date of the release day

Up until Friday, Nov 10

  • Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, set the OVERRIDE_PRODUCTION_CHECKS_REASON as a variable in the manual promote job. The value of the variable should be the reason why production checks are being overridden. See gitlab-com-deployer.md#skipping-production-promotion-checks for more info.
    3. This will post a comment into this issue and begin the deployment
    4. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

Release Preparation

Initial preparation day: Friday, Nov 10

  • Find the latest sha that made it into production successfully: sha
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 16th: 
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 16th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.6`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

Candidate announcement day: Monday, Nov 13

  • Log latest auto-deploy branch: BRANCH_NAME
  • Ensure this build makes it through into production
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release: 
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 16th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.6`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

RC tag day: Tuesday, Nov 14

  • Determine what is the last auto deploy branch to have deployed to production and add it here: BRANCH

  • If you plan to use the latest commit deployed to production to create the RC, make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Create a RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 16.6.0-rc42

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 16.6.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

  • Verify that the CE stable branch contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@16-6-stable-ee"

  • Verify that the pipelines are green

    # In Slack:
/chatops run release status 16.6.0-rc42

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 16th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/16-6-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.6`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

Tag day: Wednesday, Nov 15

Instructions for manual deploy 
    ```sh
    # In Slack:
    /chatops run deploy 16.6.0-ee.0 release
    ```
  • Validate 16.6.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

Release day: Thursday, Nov 16

Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

  • At 13:00 UTC, post an update about the package building status in #f_upcoming_release 
    :mega: Packages for 16.6.0 are built and will be published at 13:10UTC
  • At 13:10 UTC:
    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • Publish the packages via ChatOps: 
      # In Slack:
/chatops run publish 16.6.0
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
  • At 14:10 UTC:
    • Verify the check-packages job completes successfully on the EE Pipeline
    • Verify the check-packages job completes successfully on the CE Pipeline
    • Verify that Docker images appear on hub.docker.com: EE / CE
    • Post an update about the status in #f_upcoming_release 
    :mega: 16.6.0 is published and publicly available

    • Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC

    • Adjust the protected branch rules for the current version

    • Create the 16.6.0 version on version.gitlab.com

    • Create a new release environment for the stable branch in the release environments repository. This is done by opening a merge request to create a file at environments/16-6-stable with contents the same as the previous monthly release environment. The next time someone merges to the stable branch, that environment will be updated. Make sure another release manager reviews and merges the MR.

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security

  • 16.7
Click to expand

First steps

First Staging Rollback Practice

Date to be Completed:

Second Staging Rollback Practice

Date to be Completed:

Hot patching production practice

Date to be Completed:

The work here is to be done by any delivery team member, excluding the current release managers

  • @name
  • Perform hot patching Practice
  • Hot Patching practice is documented: <link to comment with screenshots>
  • Set Due date for this issue to the date of the release day

Up until Friday, Dec 15

  • Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, set the OVERRIDE_PRODUCTION_CHECKS_REASON as a variable in the manual promote job. The value of the variable should be the reason why production checks are being overridden. See gitlab-com-deployer.md#skipping-production-promotion-checks for more info.
    3. This will post a comment into this issue and begin the deployment
    4. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

Release Preparation

Initial preparation day: Friday, Dec 15

  • Find the latest sha that made it into production successfully: sha
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 21st: 
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 21st.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.7`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

Candidate announcement day: Monday, Dec 18

  • Log latest auto-deploy branch: BRANCH_NAME
  • Ensure this build makes it through into production
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release: 
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 21st.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.7`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

RC tag day: Tuesday, Dec 19

  • Determine what is the last auto deploy branch to have deployed to production and add it here: BRANCH

  • If you plan to use the latest commit deployed to production to create the RC, make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Create a RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 16.7.0-rc42

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 16.7.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

  • Verify that the CE stable branch contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@16-7-stable-ee"

  • Verify that the pipelines are green

    # In Slack:
/chatops run release status 16.7.0-rc42

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 21st.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/16-7-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.7`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

Tag day: Wednesday, Dec 20

Instructions for manual deploy 
    ```sh
    # In Slack:
    /chatops run deploy 16.7.0-ee.0 release
    ```
  • Validate 16.7.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

Release day: Thursday, Dec 21

Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

  • At 13:00 UTC, post an update about the package building status in #f_upcoming_release 
    :mega: Packages for 16.7.0 are built and will be published at 13:10UTC
  • At 13:10 UTC:
    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • Publish the packages via ChatOps: 
      # In Slack:
/chatops run publish 16.7.0
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
  • At 14:10 UTC:
    • Verify the check-packages job completes successfully on the EE Pipeline
    • Verify the check-packages job completes successfully on the CE Pipeline
    • Verify that Docker images appear on hub.docker.com: EE / CE
    • Post an update about the status in #f_upcoming_release 
    :mega: 16.7.0 is published and publicly available

    • Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC

    • Adjust the protected branch rules for the current version

    • Create the 16.7.0 version on version.gitlab.com

    • Create a new release environment for the stable branch in the release environments repository. This is done by opening a merge request to create a file at environments/16-7-stable with contents the same as the previous monthly release environment. The next time someone merges to the stable branch, that environment will be updated. Make sure another release manager reviews and merges the MR.

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security

  • 16.8
Click to expand

First steps

First Staging Rollback Practice

Date to be Completed:

Second Staging Rollback Practice

Date to be Completed:

Hot patching production practice

Date to be Completed:

The work here is to be done by any delivery team member, excluding the current release managers

  • @name
  • Perform hot patching Practice
  • Hot Patching practice is documented: <link to comment with screenshots>
  • Set Due date for this issue to the date of the release day

Up until Friday, Jan 12

  • Ensure any deploys that do not make it to canary are investigated. Disable canary if necessary.
  • Push any successful deploy to canary into production after some time has passed (preferably 1h).
  • Should any deployment blockers prevent automatic promotions to production, this requires approval by the SRE On-Call.
    1. Ask for permission to promote the release in #production - provide the necessary context to the Engineer
    2. If permission is granted, set the OVERRIDE_PRODUCTION_CHECKS_REASON as a variable in the manual promote job. The value of the variable should be the reason why production checks are being overridden. See gitlab-com-deployer.md#skipping-production-promotion-checks for more info.
    3. This will post a comment into this issue and begin the deployment
    4. Ask the SRE On-Call to respond to the comment with their approval for auditing purposes

Release Preparation

Initial preparation day: Friday, Jan 12

  • Find the latest sha that made it into production successfully: sha
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Notify Engineering Managers and developers that this is the sha that is guaranteed to be released on the 18th: 
    /chatops run notify ":mega: This is the most recent commit running on GitLab.com and this is guaranteed to be released on the 18th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>.
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.8`
Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

Candidate announcement day: Monday, Jan 15

  • Log latest auto-deploy branch: BRANCH_NAME
  • Ensure this build makes it through into production
  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.
  • Grab the sha from this new auto-deploy branch and notify Engineering Managers and developers that this is the candidate sha for the release: 
    /chatops run notify ":mega: This is the _candidate_ commit to be released on the 18th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/<SHA>
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.8`
Further deployments may result in the final commit being different from the candidate. Please see the following documentation on what this means:
  * `https://about.gitlab.com/handbook/engineering/releases/#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release`
  * `https://about.gitlab.com/handbook/engineering/releases/#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-into-the-monthly-release`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

RC tag day: Tuesday, Jan 16

  • Determine what is the last auto deploy branch to have deployed to production and add it here: BRANCH

  • If you plan to use the latest commit deployed to production to create the RC, make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Create a RC version to ensure that the final version builds correctly

    # In Slack:
/chatops run release tag 16.8.0-rc42

This will use the latest commit deployed to production for the various components that we release. If a different commit is necessary for a component, such as GitLab, you should run the following instead:

/chatops run release tag 16.8.0-rc42 --gitlab-sha=XXX

This will then use XXX as the SHA to create the GitLab stable branches.

NOTE: this SHA is only used if the stable branch has yet to be created. If it already exists, the branch is left as-is.

  • Verify that the CE stable branch contains the right commits

    • There should be at least two commits: the last commit from the previous stable branch (usually a version update), and the sync commit created by the merge train.
    • The sync commit will have the message "Add latest changes from gitlab-org/gitlab@16-8-stable-ee"

  • Verify that the pipelines are green

    # In Slack:
/chatops run release status 16.8.0-rc42

  • Notify Engineering Managers and developers that final candidate has been created:

    /chatops run notify ":mega: The stable branch has been created and the release candidate is tagged. Barring any show-stopping issues, this is the final commit to be released on the 18th.
https://gitlab.com/gitlab-org/security/gitlab/-/commits/16-8-stable-ee
You can check if an MR made the cut by using the following ChatOps command: `/chatops run release check [MR_URL] 16.8`
  * Documentation about `release check` chatops command: `https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/auto-deploy.md#status-of-a-merged-mr-with-respect-to-monthly-releases`"

  • Verify that the RC has been deployed to the pre environment

    • Deployment to pre will start automatically. It can take 2 hours to start once the RC is tagged. A notification will be sent to the #announcements channel in Slack when it starts.
    • If required to deploy manually, follow the steps in pre-and-release-environments.md#manual-deployments.

Tag day: Wednesday, Jan 17

Instructions for manual deploy 
    ```sh
    # In Slack:
    /chatops run deploy 16.8.0-ee.0 release
    ```
  • Validate 16.8.0 has been passed automated QA by ensuring the release-gitlab-qa-smoke job from the release deploy pipeline is green.

Past this point, no new code can be added to the release that was not included in the final RC.

Release day: Thursday, Jan 18

Final release is tagged, so any changes will have to initiate a patch release. Reminder: We have a soft PCL today, coordinate with the EOC before deploying to production. Consider not running the post-deploy migration pipeline to keep rollback options available.

  • At 13:00 UTC, post an update about the package building status in #f_upcoming_release 
    :mega: Packages for 16.8.0 are built and will be published at 13:10UTC
  • At 13:10 UTC:
    • ⚠ Make sure that neither packages nor the blog post get published earlier than 13:10UTC without approval by the messaging lead of the release post. Mind that you don't need their approval if you're on time ⚠
    • Publish the packages via ChatOps: 
      # In Slack:
/chatops run publish 16.8.0
    • If anything goes wrong and the release is delayed, ping the release post manager on Slack to make them aware of the issue. Cross-post the slack message to the #marketing channel to notify them too
  • At 14:10 UTC:
    • Verify the check-packages job completes successfully on the EE Pipeline
    • Verify the check-packages job completes successfully on the CE Pipeline
    • Verify that Docker images appear on hub.docker.com: EE / CE
    • Post an update about the status in #f_upcoming_release 
    :mega: 16.8.0 is published and publicly available

    • Once all packages are available publicly and GitLab.com is up and running on the release version, ping the release post manager on Slack (#release-post channel) to give them a go to merge the release post at ~14:20 UTC, so that it will be live at 15:00 UTC

    • Adjust the protected branch rules for the current version

    • Create the 16.8.0 version on version.gitlab.com

    • Create a new release environment for the stable branch in the release environments repository. This is done by opening a merge request to create a file at environments/16-8-stable with contents the same as the previous monthly release environment. The next time someone merges to the stable branch, that environment will be updated. Make sure another release manager reviews and merges the MR.

Release Certification

The release certification process may apply to this release. cc @gitlab-com/gl-security/federal-application-security

Author Check-list

  • Has documentation been updated?
Edited by Mayra Cabrera

Merge request reports