Skip to content

Adjust security release template

Reuben Pereira requested to merge rp/adapt-security-release-template into master

What does this MR do and why?

Describe in detail what your merge request does and why.

  • In the security patch issue used by RMs, remove explicit dates and instead use relative language like "Two days before due date".
  • Do not set due date on the security release tracking issue.
  • Both these changes are behind the dynamic_release_date feature flag.

Security issue template changes

Security release issue with `dynamic_release_date` FF enabled

Security patch release: 1.1.1, 1.1.2, 1.1.3

First steps

  • Set the Due date on this issue with the planned Security publish date

  • Disable Omnibus nightly builds by setting the schedules to inactive: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules. This prevents us accidentally revealing vulnerabilities before the release.

  • Ensure that Canonical, Security and Build repositories are synced:

# In Slack
/chatops run mirror status
  • Post a message on the #quality Slack channel to notify the Quality team that a security release is in progress:

Hello team, the security release has started (<link_to_this_issue>) and Omnibus nightly builds are now disabled. The GitLab ChatOps bot will post a notification to this channel when the security release is complete.

  • Post a comment on https://gitlab.com/gitlab-jh/gitlab-jh-enablement/-/issues/112 to notify JiHU of the upcoming security release.

  • Post a message on the #g_engineering_productivity channel to let them know that the secuirty release preperation has started. EP will use this information to quickly respond to pipeline failures to keep us unblocked

  • Post a message on the #g_runner Slack channel to notify the Runner team that a security release is in progress and that it will be published on the due date.

  • Verify pipelines on the GitLab projects are green: [gitlab-org/security/charts/components/images, gitlab-org/security/gitaly, gitlab-org/security/gitlab-pages, gitlab-org/security/omnibus-gitlab]

  • Check if the security release tracking issue contains any linked issues for projects under GitLab managed versioning (cng-ee, gitaly, gitlab-pages, omnibus-gitlab-ee).

    • If there are any linked issues, follow the release manager instructions and adjust this issue to include any additional steps needed. This is to synchronize the GitLab and the GitLab runner security release in case there is one planned.

  • Modify the dates below to accurately reflect the plan of action.

    For example, if the planned due date is 28th, update the section titled "One day before due date" to "On 27th (One day before due date)".

Early-merge phase

Up until one day before the Security Release due date

  • Merge the merge requests targeting default branches

    # In Slack
/chatops run release merge --security --default-branch

  • Verify if a Gitaly security fix is included in the upcoming security release, if it is, follow the How to deal with Gitaly security fixes guide.

Three days before due date

  • Check which security issues are not ready, and post a comment listing the issues that will be removed, why they will be removed (the reason returned by the run release merge command), by when they need to be ready (next day), and ping the authors. This serves as a warning to authors who weren't aware that their security issue was not in an acceptable state, and should hopefully result in less security issues needing to be unlinked.

One day before due date

If this date is on a weekend, do this work on the next working day.

  • Determine the security release manager from the schedule. Look for the security release manager of the latest released monthly version

  • Post the following message to #sec-appsec in Slack: <security-release-manager> We are starting the [security release](<link to this issue>), aiming for release tomorrow. Please create a blog post MR on gitlab-org/security/www-gitlab-com.

  • Once the blog post MR has been created by the security release manager, add a link to it here: https://gitlab.com/gitlab-org/security/www-gitlab-com/-/merge_requests/

  • Fetch the list of non-security MRs that will be included in these releases:

    /chatops run release pending_backports
    • Crosslink the list to #sec-appsec letting the security release manager these should be included in the blog post.

  • Check that all MRs merged into the default branch have been deployed to production:

    # In Slack:
/chatops run auto_deploy security_status

    NOTE: This only checks gitlab-org/security/gitlab. If other projects have security MRs you should verify those manually.

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute

  • Unlink any security issues that are not ready from the security release tracking issue (in gitlab-org/gitlab), and post a comment listing the issues that have been removed, why they have been removed (the reason returned by the run release merge command), and ping the authors. Make sure to complete this step before merging backports to avoid picking up unexpected changes

  • Merge backports and any other merge request pending:

    # In Slack:
/chatops run release merge --security

  • Ensure all Merge Requets associated with Implementation Issues with label reduced backports are merged

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • If all the security issues have been deployed to production, consider tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security

For the next task: Waiting between pipelines is necessary as they may otherwise fail to concurrently push changes to the same project/branch.

  • Tag the 1.1.1 security release, and wait for the pipeline to finish: /chatops run release tag --security 1.1.1

  • Tag the 1.1.2 security release, and wait for the pipeline to finish: /chatops run release tag --security 1.1.2

  • Tag the 1.1.3 security release, and wait for the pipeline to finish: /chatops run release tag --security 1.1.3

  • Check that EE and CE packages are built:

Deploy

Release

Consider communicating with the AppSec counterpart before publishing to sync on the time of releasing the blog post. Emails to the security mailing list are normally handled as a follow up task and should not delay release tasks

  • Publish 1.1.1 via ChatOps:

    /chatops run publish --security 1.1.1

  • Publish 1.1.2 via ChatOps:

    /chatops run publish --security 1.1.2

  • Publish 1.1.3 via ChatOps:

    /chatops run publish --security 1.1.3

  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/

  • Verify that the check-packages job completes:

  • Verify that Docker images appear on hub.docker.com: EE / CE

  • Deploy the blog post

  • Create the versions:

    • Create 1.1.1 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 1.1.2 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 1.1.3 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.

Final steps

  • Sync default branches for GitLab Foss, Omnibus GitLab and Gitaly, via ChatOps:

    # In Slack
/chatops run release sync_remotes --security

  • Close the security implementation issues

    # In Slack
/chatops run release close_issues --security

  • Close the old security release tracking issue and create a new one:

    # In Slack
/chatops run release tracking_issue --security

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules

  • In case it was disabled, enable the Gitaly update task.

  • Notify engineers the security release is out (blog post link needs to be replaced with the actual link):

/chatops run notify ":mega: GitLab Security Release: 1.1.1, 1.1.2, 1.1.3 has just been released: <blog post link>! Share this release blog post with your network to ensure broader visibility across our community."

  • Check all new tags have synced to Canonical

  • Ping the next set of release managers on the upcoming security release issue and ask them to set the intended security release due date. If needed, suggest a possible due date based on the current release activities.

  • Link the new security release tracking issue on the topic of the #releases channel, next to Next Security Release.

  • Sync the GitLab default branch by using the merge-train project:

  • If after 5 times the sync by the merge train continues to fail, use the previous strategy to sync the GitLab project:

    • Disable the merge_train_to_canonical feature flag on ops.
    • Enable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master pipeline schedule on the merge-train.
    • Execute the sync_remotes task on Slack: /chatops run release sync_remotes --security. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

Security issue with `dynamic_release_date` disabled

Security patch release: 1.1.1, 1.1.2, 1.1.3

First steps

  • Set the Due date on this issue with the planned Security publish date

  • Disable Omnibus nightly builds by setting the schedules to inactive: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules. This prevents us accidentally revealing vulnerabilities before the release.

  • Ensure that Canonical, Security and Build repositories are synced:

# In Slack
/chatops run mirror status
  • Post a message on the #quality Slack channel to notify the Quality team that a security release is in progress:

Hello team, the security release has started (<link_to_this_issue>) and Omnibus nightly builds are now disabled. The GitLab ChatOps bot will post a notification to this channel when the security release is complete.

  • Post a comment on https://gitlab.com/gitlab-jh/gitlab-jh-enablement/-/issues/112 to notify JiHU of the upcoming security release.

  • Post a message on the #g_engineering_productivity channel to let them know that the secuirty release preperation has started. EP will use this information to quickly respond to pipeline failures to keep us unblocked

  • Post a message on the #g_runner Slack channel to notify the Runner team that a security release is in progress and that it will be published on the due date.

  • Verify pipelines on the GitLab projects are green: [gitlab-org/security/charts/components/images, gitlab-org/security/gitaly, gitlab-org/security/gitlab-pages, gitlab-org/security/omnibus-gitlab]

  • Check if the security release tracking issue contains any linked issues for projects under GitLab managed versioning (cng-ee, gitaly, gitlab-pages, omnibus-gitlab-ee).

    • If there are any linked issues, follow the release manager instructions and adjust this issue to include any additional steps needed. This is to synchronize the GitLab and the GitLab runner security release in case there is one planned.

  • Modify the dates below to accurately reflect the plan of action.

Early-merge phase

Up until one day before the Security Release due date

  • Merge the merge requests targeting default branches

    # In Slack
/chatops run release merge --security --default-branch

  • Verify if a Gitaly security fix is included in the upcoming security release, if it is, follow the How to deal with Gitaly security fixes guide.

On the 25th (three days before due date)

  • Check which security issues are not ready, and post a comment listing the issues that will be removed, why they will be removed (the reason returned by the run release merge command), by when they need to be ready (next day), and ping the authors. This serves as a warning to authors who weren't aware that their security issue was not in an acceptable state, and should hopefully result in less security issues needing to be unlinked.

On the 27th (one day before due date)

If this date is on a weekend, do this work on the next working day.

  • Determine the security release manager from the schedule. Look for the security release manager of the latest released monthly version

  • Post the following message to #sec-appsec in Slack: <security-release-manager> We are starting the [security release](<link to this issue>), aiming for release tomorrow. Please create a blog post MR on gitlab-org/security/www-gitlab-com.

  • Once the blog post MR has been created by the security release manager, add a link to it here: https://gitlab.com/gitlab-org/security/www-gitlab-com/-/merge_requests/

  • Fetch the list of non-security MRs that will be included in these releases:

    /chatops run release pending_backports
    • Crosslink the list to #sec-appsec letting the security release manager these should be included in the blog post.

  • Check that all MRs merged into the default branch have been deployed to production:

    # In Slack:
/chatops run auto_deploy security_status

    NOTE: This only checks gitlab-org/security/gitlab. If other projects have security MRs you should verify those manually.

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute

  • Unlink any security issues that are not ready from the security release tracking issue (in gitlab-org/gitlab), and post a comment listing the issues that have been removed, why they have been removed (the reason returned by the run release merge command), and ping the authors. Make sure to complete this step before merging backports to avoid picking up unexpected changes

  • Merge backports and any other merge request pending:

    # In Slack:
/chatops run release merge --security

  • Ensure all Merge Requets associated with Implementation Issues with label reduced backports are merged

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • If all the security issues have been deployed to production, consider tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security

For the next task: Waiting between pipelines is necessary as they may otherwise fail to concurrently push changes to the same project/branch.

  • Tag the 1.1.1 security release, and wait for the pipeline to finish: /chatops run release tag --security 1.1.1

  • Tag the 1.1.2 security release, and wait for the pipeline to finish: /chatops run release tag --security 1.1.2

  • Tag the 1.1.3 security release, and wait for the pipeline to finish: /chatops run release tag --security 1.1.3

  • Check that EE and CE packages are built:

Deploy

Release

Consider communicating with the AppSec counterpart before publishing to sync on the time of releasing the blog post. Emails to the security mailing list are normally handled as a follow up task and should not delay release tasks

  • Publish 1.1.1 via ChatOps:

    /chatops run publish --security 1.1.1

  • Publish 1.1.2 via ChatOps:

    /chatops run publish --security 1.1.2

  • Publish 1.1.3 via ChatOps:

    /chatops run publish --security 1.1.3

  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/

  • Verify that the check-packages job completes:

  • Verify that Docker images appear on hub.docker.com: EE / CE

  • Deploy the blog post

  • Create the versions:

    • Create 1.1.1 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 1.1.2 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 1.1.3 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.

Final steps

  • Sync default branches for GitLab Foss, Omnibus GitLab and Gitaly, via ChatOps:

    # In Slack
/chatops run release sync_remotes --security

  • Close the security implementation issues

    # In Slack
/chatops run release close_issues --security

  • Close the old security release tracking issue and create a new one:

    # In Slack
/chatops run release tracking_issue --security

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules

  • In case it was disabled, enable the Gitaly update task.

  • Notify engineers the security release is out (blog post link needs to be replaced with the actual link):

/chatops run notify ":mega: GitLab Security Release: 1.1.1, 1.1.2, 1.1.3 has just been released: <blog post link>! Share this release blog post with your network to ensure broader visibility across our community."

  • Check all new tags have synced to Canonical

  • Ping the next set of release managers on the upcoming security release issue and ask them to set the intended security release due date. If needed, suggest a possible due date based on the current release activities.

  • Link the new security release tracking issue on the topic of the #releases channel, next to Next Security Release.

  • Sync the GitLab default branch by using the merge-train project:

  • If after 5 times the sync by the merge train continues to fail, use the previous strategy to sync the GitLab project:

    • Disable the merge_train_to_canonical feature flag on ops.
    • Enable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master pipeline schedule on the merge-train.
    • Execute the sync_remotes task on Slack: /chatops run release sync_remotes --security. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

Critical security release

Critical security patch release: 16.3.2, 16.2.6, 16.1.6

First steps

  • Set the Due date on this issue with the planned Security publish date

  • Disable Omnibus nightly builds by setting the schedules to inactive: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules. This prevents us accidentally revealing vulnerabilities before the release.

  • Ensure that Canonical, Security and Build repositories are synced:

# In Slack
/chatops run mirror status
  • Post a message on the #quality Slack channel to notify the Quality team that a security release is in progress:

Hello team, the security release has started (<link_to_this_issue>) and Omnibus nightly builds are now disabled. The GitLab ChatOps bot will post a notification to this channel when the security release is complete.

  • Post a comment on https://gitlab.com/gitlab-jh/gitlab-jh-enablement/-/issues/112 to notify JiHU of the upcoming security release.

  • Post a message on the #g_engineering_productivity channel to let them know that the secuirty release preperation has started. EP will use this information to quickly respond to pipeline failures to keep us unblocked

  • Post a message on the #g_runner Slack channel to notify the Runner team that a security release is in progress and that it will be published on the due date.

  • Verify pipelines on the GitLab projects are green: [gitlab-org/security/charts/components/images, gitlab-org/security/gitaly, gitlab-org/security/gitlab-pages, gitlab-org/security/omnibus-gitlab]

  • Check if the security release tracking issue contains any linked issues for projects under GitLab managed versioning (cng-ee, gitaly, gitlab-pages, omnibus-gitlab-ee).

  • Post a message to #g_dedicated-team channel in Slack to inform about this critical security release. Dedicated will need to make a plan to patch their installations.

One day before the due date

  • Determine the security release manager from the schedule. Look for the security release manager of the latest released monthly version
  • Post the following message to #sec-appsec in Slack: <security-release-manager> We are starting the [security release](<link to this issue>), aiming for release tomorrow. Please create a blog post MR on gitlab-org/security/www-gitlab-com.
  • Once the blog post MR has been created by the security release manager, add a link to it here: https://gitlab.com/gitlab-org/security/www-gitlab-com/-/merge_requests/
  • Fetch the list of non-security MRs that will be included in these releases: 
    /chatops run release pending_backports
    • Crosslink the list to #sec-appsec letting the security release manager these should be included in the blog post.
  • Merge critical security merge requests using the UI.
  • Follow the process for [speed the auto-deploy process for security merge requests]
  • Deploy all the fixes to production.
  • Merge all backports into the appropriate stable branches (if required). Stable branches will diverge at this point. Any further fixes must target security branches.
  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security
  • If all the security issues have been deployed to production, consider tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security

For the next task: Waiting between pipelines is necessary as they may otherwise fail to concurrently push changes to the same project/branch.

  • Tag the 16.3.2 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.3.2

  • Tag the 16.2.6 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.2.6

  • Tag the 16.1.6 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.1.6

  • Check that EE and CE packages are built:

Deploy

Release

Consider communicating with the AppSec counterpart before publishing to sync on the time of releasing the blog post. Emails to the security mailing list are normally handled as a follow up task and should not delay release tasks

  • Publish 16.3.2 via ChatOps:

    /chatops run publish --security 16.3.2

  • Publish 16.2.6 via ChatOps:

    /chatops run publish --security 16.2.6

  • Publish 16.1.6 via ChatOps:

    /chatops run publish --security 16.1.6

  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/

  • Verify that the check-packages job completes:

  • Verify that Docker images appear on hub.docker.com: EE / CE

  • Deploy the blog post

  • Create the versions:

    • Create 16.3.2 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 16.2.6 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 16.1.6 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.

Final steps

  • Sync default branches for GitLab Foss, Omnibus GitLab and Gitaly, via ChatOps:

    # In Slack
/chatops run release sync_remotes --security

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules

  • In case it was disabled, enable the Gitaly update task.

  • Notify engineers the security release is out (blog post link needs to be replaced with the actual link):

/chatops run notify ":mega: GitLab Critical Security Release: 16.3.2, 16.2.6, 16.1.6 has just been released: <blog post link>! Share this release blog post with your network to ensure broader visibility across our community."

  • Check all new tags have synced to Canonical

  • Ping the next set of release managers on the upcoming security release issue and ask them to set the intended security release due date. If needed, suggest a possible due date based on the current release activities.

  • Link the new security release tracking issue on the topic of the #releases channel, next to Next Security Release.

  • Sync the GitLab default branch by using the merge-train project:

  • If after 5 times the sync by the merge train continues to fail, use the previous strategy to sync the GitLab project:

    • Disable the merge_train_to_canonical feature flag on ops.
    • Enable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master pipeline schedule on the merge-train.
    • Execute the sync_remotes task on Slack: /chatops run release sync_remotes --security. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

  • Close the critical security tracking issue

gitlab-com/gl-infra/delivery#19533 (closed)

Author Check-list

  • Has documentation been updated?
Edited by Reuben Pereira

Merge request reports