Skip to content

Add blog post job to the security pipeline

Mayra Cabrera requested to merge add-release-blog-post-to-security-pipeline into master

What does this MR do and why?

A dedicated CI configuration was added to the security releasee pipeline for the steps triggered one day before the due date. This CI configuration includes two jobs:

  • security_release:release_preparation:start: A notification job to broadcast the generation of the blog post in the f_upcoming_release channel.
  • security_release:release_preparation:blog_post: A job to trigger the creation of the security and patch blog post.

The security template was also modified to include a step to trigger these jobs and disregard some when the combined_blog_post feature flag is enabled.

Related to gitlab-com/gl-infra/delivery#19705 (closed)

Testing

Pipelines

Pipeline creation

With SECURITY_RELEASE_PIPELINE as true With SECURITY_RELEASE_PIPELINE as release_preparation
Screenshot_2023-10-18_at_13.59.26 Screenshot_2023-10-18_at_14.01.59
https://gitlab.com/gitlab-org/release-tools/-/pipelines/1041626674 https://gitlab.com/gitlab-org/release-tools/-/pipelines/1041632867

Pipeline dry-run run-up

Pre execution After the release_preparation stage is triggered Slack notifications
Screenshot_2023-10-18_at_14.10.52 Screenshot_2023-10-18_at_14.12.11 Screenshot_2023-10-18_at_14.26.44

Template

Before

Click to expand

Security patch release: 16.4.2, 16.3.6, 16.2.9

First steps

  • Set the Due date on this issue with the planned Security publish date

  • Start the security_release_prepare:start job in the security release pipeline: foo

    • Ensure the security_release:prepare stage completes before continuing to the next section.

  • Modify the dates below to accurately reflect the plan of action.

    For example, if the planned due date is 28th, update the section titled "One day before due date" to "On 27th (One day before due date)".

Two days before due date

  • Check if the security release tracking issue contains any linked issues for projects under GitLab managed versioning (cng-ee, gitaly, gitlab-pages, omnibus-gitlab-ee).
  • Check the issues linked to the security release tracking issue. If there are any that DO NOT have the security-target label applied, check to make sure they are expected to be included, otherwise unlink them and point the assignees to the correct process.
  • Merge the merge requests targeting default branches 
    # In Slack
/chatops run release merge --security --default-branch
  • Verify that the table of issues in the security release tracking issue has been updated showing the default MRs have been merged or set to Merge When Pipeline Succeeds (MWPS/auto-merge).

Two days before due date

One day before due date

If this date is on a weekend, do this work on the next working day.

  • Determine the security release manager from the schedule. Look for the security release manager of the latest released monthly version

  • Post the following message to #sec-appsec in Slack: <security-release-manager> We are starting the [security release](<link to this issue>), aiming for release tomorrow. Please create a blog post MR on gitlab-org/security/www-gitlab-com.

  • Once the blog post MR has been created by the security release manager, add a link to it here: https://gitlab.com/gitlab-org/security/www-gitlab-com/-/merge_requests/

  • Fetch the list of non-security MRs that will be included in these releases:

    /chatops run release pending_backports
    • Crosslink the list to #sec-appsec letting the security release manager these should be included in the blog post.

  • Check that all MRs merged into the default branch have been deployed to production:

    # In Slack:
/chatops run auto_deploy security_status

    NOTE: This only checks gitlab-org/security/gitlab. If other projects have security MRs you should verify those manually.

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute

  • Unlink any security issues that are not ready from the security release tracking issue (in gitlab-org/gitlab), and post a comment listing the issues that have been removed, why they have been removed (the reason returned by the run release merge command), and ping the authors. Make sure to complete this step before merging backports to avoid picking up unexpected changes

  • Merge backports and any other merge request pending:

    # In Slack:
/chatops run release merge --security

  • Ensure all Merge Requets associated with Implementation Issues with label reduced backports are merged

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • If all the security issues have been deployed to production, consider tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security

For the next task: Waiting between pipelines is necessary as they may otherwise fail to concurrently push changes to the same project/branch.

  • Tag the 16.4.2 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.4.2

  • Tag the 16.3.6 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.3.6

  • Tag the 16.2.9 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.2.9

  • Check that EE and CE packages are built:

Deploy

Release

Consider communicating with the AppSec counterpart before publishing to sync on the time of releasing the blog post. Emails to the security mailing list are normally handled as a follow up task and should not delay release tasks

  • Publish 16.4.2 via ChatOps:

    /chatops run publish --security 16.4.2

  • Publish 16.3.6 via ChatOps:

    /chatops run publish --security 16.3.6

  • Publish 16.2.9 via ChatOps:

    /chatops run publish --security 16.2.9

  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/

  • Verify that the check-packages job completes:

  • Verify that Docker images appear on hub.docker.com: EE / CE

  • Deploy the blog post

  • Create the versions:

    • Create 16.4.2 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 16.3.6 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 16.2.9 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.

Final steps

  • Start the security_release_finalize:start job in the security release pipeline: foo

  • Sync the GitLab default branch by using the merge-train project:

  • If after 5 times the sync by the merge train continues to fail, use the previous strategy to sync the GitLab project:

    • Disable the merge_train_to_canonical feature flag on ops.
    • Enable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master pipeline schedule on the merge-train.
    • Execute the sync_remotes task on Slack: /chatops run release sync_remotes --security. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

After

Click to expand

Security patch release: 16.4.2, 16.3.6, 16.2.9

First steps

  • Set the Due date on this issue with the planned Security publish date

  • Start the security_release_prepare:start job in the security release pipeline: foo

    • Ensure the security_release:prepare stage completes before continuing to the next section.

  • Modify the dates below to accurately reflect the plan of action.

    For example, if the planned due date is 28th, update the section titled "One day before due date" to "On 27th (One day before due date)".

Two days before due date

  • Check if the security release tracking issue contains any linked issues for projects under GitLab managed versioning (cng-ee, gitaly, gitlab-pages, omnibus-gitlab-ee).
  • Check the issues linked to the security release tracking issue. If there are any that DO NOT have the security-target label applied, check to make sure they are expected to be included, otherwise unlink them and point the assignees to the correct process.
  • Merge the merge requests targeting default branches 
    # In Slack
/chatops run release merge --security --default-branch
  • Verify that the table of issues in the security release tracking issue has been updated showing the default MRs have been merged or set to Merge When Pipeline Succeeds (MWPS/auto-merge).

Two days before due date

One day before due date

If this date is on a weekend, do this work on the next working day.

  • Start the security_release_blog_post:start job of the security release pipeline: foo

  • Check that all MRs merged into the default branch have been deployed to production:

    # In Slack:
/chatops run auto_deploy security_status

    NOTE: This only checks gitlab-org/security/gitlab. If other projects have security MRs you should verify those manually.

  • Make sure to execute the post-deploy migration pipeline to ensure that all post-deploy migrations have been executed: /chatops run post_deploy_migrations execute

  • Unlink any security issues that are not ready from the security release tracking issue (in gitlab-org/gitlab), and post a comment listing the issues that have been removed, why they have been removed (the reason returned by the run release merge command), and ping the authors. Make sure to complete this step before merging backports to avoid picking up unexpected changes

  • Merge backports and any other merge request pending:

    # In Slack:
/chatops run release merge --security

  • Ensure all Merge Requets associated with Implementation Issues with label reduced backports are merged

  • If any merge requests could not be merged, investigate what needs to be done to resolve the issues. Do not proceed unless it has been determined safe to do so.

  • Ensure tests are green in CE and green in EE

    # In Slack:
/chatops run release status --security

  • If all the security issues have been deployed to production, consider tagging.

On the Due Date

Packaging

  • Ensure tests are green in CE and green in EE 
    # In Slack:
/chatops run release status --security

For the next task: Waiting between pipelines is necessary as they may otherwise fail to concurrently push changes to the same project/branch.

  • Tag the 16.4.2 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.4.2

  • Tag the 16.3.6 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.3.6

  • Tag the 16.2.9 security release, and wait for the pipeline to finish: /chatops run release tag --security 16.2.9

  • Check that EE and CE packages are built:

Deploy

Release

Consider communicating with the AppSec counterpart before publishing to sync on the time of releasing the blog post. Emails to the security mailing list are normally handled as a follow up task and should not delay release tasks

  • Publish 16.4.2 via ChatOps:

    /chatops run publish --security 16.4.2

  • Publish 16.3.6 via ChatOps:

    /chatops run publish --security 16.3.6

  • Publish 16.2.9 via ChatOps:

    /chatops run publish --security 16.2.9

  • Notify AppSec counterparts they can submit the blog post to https://gitlab.com/gitlab-com/www-gitlab-com/

  • Verify that the check-packages job completes:

  • Verify that Docker images appear on hub.docker.com: EE / CE

  • Deploy the blog post

  • Create the versions:

    • Create 16.4.2 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 16.3.6 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.
    • Create 16.2.9 version on version.gitlab.com. Be sure to mark it as a security release. The Vulnerability Type column should indicate "No" for the new version.

Final steps

  • Start the security_release_finalize:start job in the security release pipeline: foo

  • Sync the GitLab default branch by using the merge-train project:

  • If after 5 times the sync by the merge train continues to fail, use the previous strategy to sync the GitLab project:

    • Disable the merge_train_to_canonical feature flag on ops.
    • Enable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master pipeline schedule on the merge-train.
    • Execute the sync_remotes task on Slack: /chatops run release sync_remotes --security. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

Author Check-list

  • [-] Has documentation been updated?
Edited by Mayra Cabrera

Merge request reports