Skip to content

Update auto-deploy steps on security template

Mayra Cabrera requested to merge update-auto-deploy-steps-on-security-template into master

What does this MR do?

For regular security releases, we now merge all the security merge requests at the same time, and cherry-pick the security fix directly into master. Because of this, some of the auto-deploy steps should be executed before the preparation steps.

This commits re-organizes the security release steps in a logical order.

Examples

Regular Security Release

Security patch release: 13.0.1, 12.10.7, 12.9.8

Preparation

Preparation steps should ideally be completed within one day

  • Temporarily disable the scheduled auto-deploy tasks via ChatOps:

    # In Slack
/chatops run auto_deploy pause

  • Disable Omnibus nightly builds by setting the schedules to inactive: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules.

  • Ensure latest auto-deploy branches are synced across Canonical, Security, and Build:

    # In Slack
/chatops run mirror status

  • Merge security merge requests using ChatOps

    # In Slack:
/chatops run release merge --security

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure security fixes are included in the auto-deploy branch. Fixes are automatically cherry-picked into the auto-deploy branch after they're merged. If they were not chery-picked, you can use the cherry pick script to do it manually.

    • For GitLab
    • For Omnibus GitLab

Auto Deploy tag and deploy to Staging

  • Tag a new auto-deploy version via ChatOps (no need to wait for green build), to create a deployer pipeline that will deploy to staging and create a QA issue: 
    # In Slack
/chatops run auto_deploy tag --security

QA

  • On the deployer pipeline, manually cancel the gprd-cny-change-lock from the deployer pipeline. This will prevent the automatic promotion to Canary.

  • Once the deploy to staging is completed, notify the @appsec-team member so they can validate the security fixes on staging.

  • Once the security fixes been validated on staging, retry the gprd-cny-change-lock job to promote to Canary.

  • If there are no issues reported on canary, proceed to promote the deployment to production.

Packaging

  • Ping the Security Engineers so they can get started with the blog post. The blog post should be done on https://dev.gitlab.org/gitlab/www-gitlab-com

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • Tag the security release:

    # In Slack:
  /chatops run release tag --security 13.0.1
  /chatops run release tag --security 12.10.7
  /chatops run release tag --security 12.9.8

  • Check that EE and CE packages are built:

Deploy

  • Verify that pre.gitlab.com is running the latest patch version

Release

  • This section should be done in coordination with the Security team, so make sure to confirm with them before proceeding

    # In Slack
@appsec-team - We are ready to publish the security release packages for 13.0.1, please let us know if the blog post is ready.

  • Publish the packages via ChatOps:

     # In Slack:
  /chatops run publish 13.0.1
  /chatops run publish 12.10.7
  /chatops run publish 12.9.8

  • Create the versions:

  • Merge the blog post on https://gitlab.com/gitlab-com/www-gitlab-com

  • In the #content-updates channel, share a link to the blog post.

Sync

  • Push security/gitlab master, and auto-deploy branches to all remotes.

  • Push security/gitlab-foss master and auto-deploy branches to all remotes.

  • Push security/omnibus-gitlab master and auto-deploy branches to all remotes.

  • Push security/gitaly master branch to all remotes.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

Auto-Deploy

  • Re-enable the scheduled auto-deploy tasks via ChatOps:

    # In Slack
/chatops run auto_deploy unpause

  • Create a new auto-deploy branch from master via ChatOps:

    # In Slack
/chatops run auto_deploy prepare

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules

Critical Security Release

Security patch release: 13.0.1, 12.10.7, 12.9.8

Preparation

Preparation steps should ideally be completed within one day

  • Temporarily disable the scheduled auto-deploy tasks via ChatOps:

    # In Slack
/chatops run auto_deploy pause

  • Disable Omnibus nightly builds by setting the schedules to inactive: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules.

  • Ensure latest auto-deploy branches are synced across Canonical, Security, and Build:

    # In Slack
/chatops run mirror status

  • Merge critical security merge requests using the UI.

    • Enable "Squash commits" option when merging.

  • Cherry-pick the security fixes into the auto-deploy branch.

Auto Deploy tag and deploy to Staging

  • Tag a new auto-deploy version via ChatOps (no need to wait for green build), to create a deployer pipeline that will deploy to staging and create a QA issue: 
    # In Slack
/chatops run auto_deploy tag --security

QA

  • On the deployer pipeline, manually cancel the gprd-cny-change-lock from the deployer pipeline. This will prevent the automatic promotion to Canary.

  • Once the deploy to staging is completed, notify the @appsec-team member so they can validate the security fixes on staging.

  • Once the security fixes been validated on staging, retry the gprd-cny-change-lock job to promote to Canary.

  • If there are no issues reported on canary, proceed to promote the deployment to production.

Packaging

  • Ping the Security Engineers so they can get started with the blog post. The blog post should be done on https://dev.gitlab.org/gitlab/www-gitlab-com

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • Tag the security release:

    # In Slack:
  /chatops run release tag --security 13.0.1
  /chatops run release tag --security 12.10.7
  /chatops run release tag --security 12.9.8

  • Check that EE and CE packages are built:

Deploy

  • Verify that pre.gitlab.com is running the latest patch version

Release

  • This section should be done in coordination with the Security team, so make sure to confirm with them before proceeding

    # In Slack
@appsec-team - We are ready to publish the security release packages for 13.0.1, please let us know if the blog post is ready.

  • Publish the packages via ChatOps:

     # In Slack:
  /chatops run publish 13.0.1
  /chatops run publish 12.10.7
  /chatops run publish 12.9.8

  • Create the versions:

  • Merge the blog post on https://gitlab.com/gitlab-com/www-gitlab-com

  • In the #content-updates channel, share a link to the blog post.

Sync

  • Push security/gitlab master, and auto-deploy branches to all remotes.

  • Push security/gitlab-foss master and auto-deploy branches to all remotes.

  • Push security/omnibus-gitlab master and auto-deploy branches to all remotes.

  • Push security/gitaly master branch to all remotes.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

Auto-Deploy

  • Re-enable the scheduled auto-deploy tasks via ChatOps:

    # In Slack
/chatops run auto_deploy unpause

  • Create a new auto-deploy branch from master via ChatOps:

    # In Slack
/chatops run auto_deploy prepare

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules

Edited by Mayra Cabrera

Merge request reports