Skip to content

Add task into security pipeline to enable omnibus builds

Mayra Cabrera requested to merge re-enable-omnibus-tasks into master

What does this MR do and why?

As part of automating the final steps of a security release, a task has been included to re-enable the omnibus builds. Logic inside the Security::Prepare::OmnibusNightly was updated to account for two status: enable and disable.

Related to gitlab-com/gl-infra/delivery#19440 (closed)

Testing

Prework

Security release pipeline

Security release pipeline
Screenshot_2023-07-17_at_14.05.13

Job log:

2023-07-17 20:06:59.784823 D ReleaseTools::GitlabClient -- [HTTParty] [2023-07-17 20:06:59 +0000] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-tools/remote_mirrors" - 
2023-07-17 20:07:09.644094 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:07:09 +0000] 200 "GET https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules" - 
2023-07-17 20:07:09.644408 I ReleaseTools::Security::Prepare::OmnibusNightly -- Updating omnibus pipeline schedule -- {:description=>"EE nightly", :action=>:disable}
2023-07-17 20:07:09.644459 I ReleaseTools::Security::Prepare::OmnibusNightly -- Taking ownership of the pipeline schedule -- {:pipeline_schedule=>5}
2023-07-17 20:07:15.867375 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:07:15 +0000] 201 "POST https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/5/take_ownership" 809 
2023-07-17 20:07:21.008035 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:07:21 +0000] 200 "PUT https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/5" - 
2023-07-17 20:07:21.008189 I ReleaseTools::Security::Prepare::OmnibusNightly -- Updating omnibus pipeline schedule -- {:description=>"CE nightly", :action=>:disable}
2023-07-17 20:07:21.008209 I ReleaseTools::Security::Prepare::OmnibusNightly -- Taking ownership of the pipeline schedule -- {:pipeline_schedule=>4}
2023-07-17 20:07:26.405581 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:07:26 +0000] 201 "POST https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/4/take_ownership" 809 
2023-07-17 20:07:30.897996 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:07:30 +0000] 200 "PUT https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/4" - 
2023-07-17 20:07:30.898191 I ReleaseTools::Slack::Security::Notifier -- Posting slack message -- {:job_type=>"Omnibus nightly builds", :status=>:success}
2023-07-17 20:08:40.763239 D ReleaseTools::GitlabClient -- [HTTParty] [2023-07-17 20:08:40 +0000] 200 "GET https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-tools/remote_mirrors" - 
2023-07-17 20:08:41.754773 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:08:41 +0000] 200 "GET https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules" - 
2023-07-17 20:08:41.755014 I ReleaseTools::Security::Prepare::OmnibusNightly -- Updating omnibus pipeline schedule -- {:description=>"EE nightly", :action=>:enable}
2023-07-17 20:08:41.755041 I ReleaseTools::Security::Prepare::OmnibusNightly -- Taking ownership of the pipeline schedule -- {:pipeline_schedule=>5}
2023-07-17 20:08:42.535724 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:08:42 +0000] 200 "POST https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/5/take_ownership" - 
2023-07-17 20:08:43.204262 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:08:43 +0000] 200 "PUT https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/5" - 
2023-07-17 20:08:43.204491 I ReleaseTools::Security::Prepare::OmnibusNightly -- Updating omnibus pipeline schedule -- {:description=>"CE nightly", :action=>:enable}
2023-07-17 20:08:43.204528 I ReleaseTools::Security::Prepare::OmnibusNightly -- Taking ownership of the pipeline schedule -- {:pipeline_schedule=>4}
2023-07-17 20:08:43.869474 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:08:43 +0000] 200 "POST https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/4/take_ownership" - 
2023-07-17 20:08:44.444219 D ReleaseTools::GitlabDevClient -- [HTTParty] [2023-07-17 20:08:44 +0000] 200 "PUT https://dev.gitlab.org/api/v4/projects/gitlab%2Fomnibus-gitlab/pipeline_schedules/4" - 
2023-07-17 20:08:44.444428 I ReleaseTools::Slack::Security::Notifier -- Posting slack message -- {:job_type=>"Omnibus nightly builds", :status=>:success}

Omnibus pipeline schedules:

Before the security release pipeline After the 'disable job' was triggered After the 'enable job' was triggered
Screenshot_2023-07-17_at_14.02.32 Screenshot_2023-07-17_at_14.08.03 Screenshot_2023-07-17_at_14.09.03

Slack notifications

Disabling omnibus pipeline schedules Enable omnibus pipeline schedules
Screenshot_2023-07-17_at_14.11.41 Screenshot_2023-07-17_at_14.11.50
Link Link

Security template

  • With security_pipeline enabled
Click to expand

Final steps

  • Start the security_release_finalize:start job in the security release pipeline: foo

  • Sync the GitLab default branch by using the merge-train project:

    • Disable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master [pipeline schedule on the merge-train].
    • Trigger the gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master [pipeline schedule on the merge-train] and wait until it finishes. This pipeline will attempt to sync the GitLab default branch.
    • If the sync fails, repeat the above step.

  • If after 5 times the sync by the merge train continues to fail, use the previous strategy to sync the GitLab project:

    • Disable the merge_train_to_canonical [feature flag on ops].
    • Enable the gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master [pipeline schedule on the merge-train].
    • Execute the sync_remotes task on Slack: /chatops run release sync_remotes --security. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

  • Verify all remotes are synced:

    # In Slack
/chatops run mirror status

    If conflicts are found, manual intervention will be needed to sync the repositories.

  • Notify engineers the security release is out (blog post link needs to be replaced with the actual link):

    /chatops run notify ":mega: GitLab Security Release: 16.1.3, 16.0.8, 15.11.13 has just been released: <blog post link>! Share this release blog post with your network to ensure broader visibility across our community."

  • In case it was disabled, enable the Gitaly update task.

  • Close the old security release tracking issue and create a new one:

    # In Slack
/chatops run release tracking_issue --security

  • Ping the [next set of release managers] on the [upcoming security release] issue and ask them to set the intended security release due date. If needed, suggest a possible due date based on the current release activities.

  • Check all new tags have synced to Canonical

  • Link the new security release tracking issue on the topic of the #releases channel, next to Next Security Release.

  • With security_pipeline disabled
Click to expand

Final steps

  • Sync default branches for GitLab Foss, Omnibus GitLab and Gitaly, via ChatOps:

    # In Slack
/chatops run release sync_remotes --security

  • Close the security implementation issues

    # In Slack
/chatops run release close_issues --security

  • Enable Omnibus nightly builds by setting the schedules to active https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules

Author Check-list

  • Has documentation been updated?
Edited by Mayra Cabrera

Merge request reports