Skip to content

Consider runner security releases

Mayra Cabrera requested to merge fix-security-template into master

What does this MR do and why?

Adjust the security template to consider runner security releases. GitLab runner and GitLab security releases can be run independently, but they should be released at the same time to make communications easier.

As a corrective action of https://gitlab.com/gitlab-com/runner-group/team-tasks/-/issues/167#note_1012198894

Example

Security patch release: 15.2.4, 15.1.4, 15.0.10

First steps

  • Set the Due date on this issue wiht the planned Security publish date
  • Disable Omnibus nightly builds by setting the schedules to inactive: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules. This prevents us accidently revealing vulnerabilities before the release.
  • Ensure that Canonical, Security and Build repositories are synced: 
    # In Slack
/chatops run mirror status
  • Post a comment on https://gitlab.com/gitlab-jh/gitlab-jh-enablement/-/issues/112 to notify JiHU of the upcoming security release
  • Post a message on the #g_runner Slack channel to notify the Runner team that a security release is in progress and that it will be published on the due date. This is to synchronize the GitLab and the GitLab runner security release in case there is one planned.
  • Modify the dates below to accurately reflect the plan of action.

Early-merge phase

Up until the 27th, or one day before the Security Release due date

  • Merge the merge requests targeting default branches

    # In Slack
/chatops run release merge --security --default-branch

  • Verify if a Gitaly security fix is included in the upcoming security release, if it is, follow the How to deal with Gitaly security fixes guide.

On the 27th (one day before due date)

If this date is on a weekend, do this work on the next working day.

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute.

  • Determine the security release manager from the schedule. Look for the security release manager of the latest released monthly version.

  • Post the following message to #sec-appsec in Slack: <security-release-manager> We are starting the [security release](<link to this issue>), aiming for release tomorrow. Please create a blog post MR on gitlab-org/security/www-gitlab-com.

  • Once the blog post MR has been created by the security release manager, add a link to it here: https://gitlab.com/gitlab-org/security/www-gitlab-com/-/merge_requests/

  • Merge security merge requests targeting default branches

    # In Slack:
/chatops run release merge --security --default-branch

  • Check that all MRs merged into the default branch have been deployed to production:

    # In Slack:
/chatops run auto_deploy security_status

    NOTE: This only checks gitlab-org/security/gitlab. If other projects have security MRs you should verify those manually.

  • Merge backports and any other merge request pending:

    # In Slack:
/chatops run release merge --security

  • Ensure all Merge Requets associated with Implementation Issues with label reduced backports are merged

  • Unlink any security issues that are not ready from the security release tracking issue (in gitlab-org/gitlab), and post a comment listing the issues that have been removed, why they have been removed (the reason returned by the run release merge command), and ping the authors.

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • If there are no Omnibus security issues associated, consider tagging today already.

    • Omnibus default branch MRs are only merged together with backport MRs and need to be deployed to gprd first before tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security

For the next task: Waiting between pipelines is necessary as they may otherwise fail to concurrently push changes to the same project/branch.

  • Tag the 15.2.4 security release, and wait for the pipeline to finish: /chatops run release tag --security 15.2.4

  • Tag the 15.1.4 security release, and wait for the pipeline to finish: /chatops run release tag --security 15.1.4

  • Tag the 15.0.10 security release, and wait for the pipeline to finish: /chatops run release tag --security 15.0.10

  • Check that EE and CE packages are built:

Deploy

Release

  • Publish packages via ChatOps: 
    /chatops run publish --security
  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/
  • Verify that EE packages appear on packages.gitlab.com: EE (should contain 15 packages)
  • Verify that CE packages appear on packages.gitlab.com: CE (should contain 13 packages)
Package Counts

Inside of the omnibus-gitlab pipelines, the Package-and-image-release stage contain many jobs. For all jobs that are part of an Operating System Package, count each of them one time, with the exception of CentOS 7, count this one 3 times.

For example, for 14.9.X-ee:

  • AmazonLinux-2-arm64-release
  • AmazonLinux-2-release
  • CentOS-7-release
  • CentOS-8-arm64-release
  • CentOS-8-release
  • Debian-9-release
  • Debian-10-arm-release
  • Debian-10-release
  • Debian-11-arm-release
  • Debian-11-release
  • OpenSUSE-15.3-arm64-release
  • OpenSUSE-15.3-release
  • SLES-12.5-release
  • SLES-15.2-release
  • Ubuntu-18.04-release
  • Ubuntu-20.04-arm-release
  • Ubuntu-20.04-release

The above would produce 19 total packages.

Points of importance:

  • The amount and type of these jobs changes over time, therefore the number of packages created may change with each release
  • Reach out to the Distribution team for any questions related to needing to confirm packages
  • CE normally has fewer packages
  • Verify that Docker images appear on hub.docker.com: EE / CE
  • Deploy the blog post
  • Create the versions:

Final steps

  • Sync default branches for GitLab, GitLab Foss, Omnibus GitLab and Gitaly, via ChatOps:

    # In Slack
/chatops run release sync_remotes --security

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

  • Close the security implementation issues

    # In Slack
/chatops run release close_issues --security

  • Notify engineers the security release is out (blog post link needs to be replaced with the actual link):

    /chatops run notify ":mega: GitLab Security Release: 15.2.4, 15.1.4, 15.0.10 has just been released: <blog post link>! Share this release blog post with your network to ensure broader visibility across our community."

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules

  • In case it was disabled, enable the Gitaly update task.

  • Close the old security release tracking issue and create a new one:

    # In Slack
/chatops run release tracking_issue --security

  • Link the new security release tracking issue on the topic of the #releases channel, next to Next Security Release.

Author Check-list

  • [-] Has documentation been updated?
Edited by Mayra Cabrera

Merge request reports