Move Omnibus security release from dev.gitlab.org to GitLab Security

When we migrated the security development to private groups on &121 (closed), we only moved the GitLab repo itself but not the GitLab components. Omnibus security release is highly tied to the GitLab Security release, so to continue automating our security release processes, we need to move the Omnibus security release process from dev.gitlab.org to the private fork on GitLab security.

This issue is to track what it's required to do this transition. Let's tentatively aim for the security release of 12.9.

Plan.

1. Modify files on omnibus-gitlab

• .gitlab/merge_request_templates/Security Release.md - gitlab-org/omnibus-gitlab!3993 (merged)
• .gitlab/issue_templates/Security developer workflow.md - gitlab-org/omnibus-gitlab!3993 (merged)
• scripts/security-harness - After this migration, dev.gitlab.org will no longer be used to prepare security MR's - gitlab-org/omnibus-gitlab!3998 (merged)
  • This was reverted as we're only mirroring protected branches from Security to Dev gitlab-org/omnibus-gitlab!4011 (merged)

2.  Modify release-tools scripts

• templates/security_patch.rb - gitlab-org/release-tools!910 (merged)
• lib/release_tools/securit/merge_requests_validator.rb - gitlab-org/release-tools!907 (merged)

3. Modify files on gitlab

• scripts/security-harness - After this migration, dev.gitlab.org will no longer be used to prepare security MR's - gitlab-org/gitlab!27121 (diffs)
• gitlab/issue_templates/Security%20Release.md - gitlab-org/gitlab!27349 (merged)

4. Ensure omnibus-gitlab on Security is ready

• Enable issues on security/omnibus-gitlab - http://gitlab.com/gitlab-org/security/omnibus-gitlab/-/issues (Unsure why they are not enabled at the moment)
• Ensure master, *-stable and auto-deploy branches are protected
• Ensure only release managers can merge and push to the protected branches.
• Ensure it's using similar issue and security templates to the ones on the GitLab project - gitlab-org/omnibus-gitlab!4012 (merged)
• Enable tracking of security releases - gitlab-org/release-tools!891 (diffs)
  • gitlab-org/release-tools!921 (merged)

5. Test Omnibus GitLab on Security

• Create security MR's and validate them.
  • Security issue was included and the merge requests were validated and merged https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/issues/1

6. Communicate to engineers using multi-modal communication

• Prepare an announcement - #743 (closed)
• Post the announcement on Slack: #g_distribution, #releases, #development, #backend, #security, #app-sec
• Post the announcement on Engineering Week-In-Review

added security-release teamDelivery labels

changed the description

From the infrastructure side, we already have everything set: a private group that is updated through push mirroring from the canonical repo. I'm not familiar with Omnibus security release process, from past releases it seems very similar (if not the same) to the GitLab process, so perhaps the ensure the process is the same.

changed the description

@balasankarc, @ibaum, @twk3 Interested in your thoughts for this one. Do you see any downside of moving the security release process from Omnibus to GitLab Security (https://gitlab.com/gitlab-org/security)

I think moving it will be good, as it allows us to properly align with the security release documentation.

Most of the changes for omnibus will need to take place in our gitlab-ci yaml, as we already have three mirrors with different jobs run, and this adds a forth.

• gitlab.com gitlab-org/omnibus-gitlab - only runs danger/spec tests
• gitlab.com gitlab-org/build/omnibus-gitlab-mirror - builds and end-to-end tests a package for a branch
• dev.gitlab.org gitlab/omnibus-gitlab - builds all the packages, but runs no test other than compliance related. This is also the only repo with release tasks and the ci settings to support them

I'm not familiar with the security forks, but I think it would be ideal for the first iteration if we were able to do the MRs in the security fork, but still have the releases all happen in dev.

moving the security release process from Omnibus to GitLab Security

This essentially changes where MRs gets opened and merged. For release, corresponding branches will get synced to the dev mirror, and that is where tags are created, right? If so, I think moving is perfectly fine.

Most of the changes for omnibus will need to take place in our gitlab-ci yaml, as we already have three mirrors with different jobs run, and this adds a forth.

In the current scenario, the only thing possible for a developer regarding security MRs is ensuring package gets built on all OSs (because MRs happen in dev). They can't run tests, or do an end-to-end QA run since they can't push the branch to com. By moving to security fork, we can give them the ability to run specs at least. For ensuring packages gets built they (or the reviewer) will have to push the branch to dev - this is not something new, but is what we currently do for regular MRs (we even have it in the MR template as a checkbox). I think we can implement the ability for an end-to-end QA run for security MRs as next iteration. For this iteration, just modify the only/except rules for spec jobs to run in security fork too.

@balasankarc, @twk3

I'm not familiar with the security forks, but I think it would be ideal for the first iteration if we were able to do the MRs in the security fork, but still have the releases all happen in dev.

This essentially changes where MRs gets opened and merged. For release, corresponding branches will get synced to the dev mirror, and that is where tags are created, right? If so, I think moving is perfectly fine.

Yes, in this case, it seems the only part of the process that will change is that MR's will be opened and merged on Omnibus GitLab Security, the rest of the security process for Omnibus will remain as-is.

For ensuring packages gets built they (or the reviewer) will have to push the branch to dev - this is not something new, but is what we currently do for regular MRs (we even have it in the MR template as a checkbox).

In our current setup, Security projects are automatically synced to dev via push mirror, so perhaps that step won't be necessary anymore after the migration.

---

I'll prepare a list of files to be modified on Omnibus and detailed steps to be taken to ensure a smooth migration.

cc @marin, @rspeicher, @yorickpeterse

mentioned in issue #689 (closed)

mentioned in merge request gitlab-org/gitlab!25901 (merged)

assigned to @mayra-cabrera

High-level plan

1. Modify files on omnibus-gitlab

• .gitlab/merge_request_templates/Security Release.md
• .gitlab/issue_templates/Security developer workflow.md
• scripts/security-harness - After this migration, dev.gitlab.org will no longer be used to prepare security MR's

2. Modify release-tools scripts

• templates/security_patch.rb
• lib/release_tools/securit/merge_requests_validator.rb

3. Modify files on gitlab

• scripts/security-harness - After this migration, dev.gitlab.org will no longer be used to prepare security MR's
• gitlab/issue_templates/Security%20Release.md

4. Ensure omnibus-gitlab on Security is ready

• Enable issues on security/omnibus-gitlab - http://gitlab.com/gitlab-org/security/omnibus-gitlab/-/issues (Unsure why they are not enabled at the moment)
• Enable tracking of security releases - gitlab-org/release-tools!891 (diffs)

5. Communicate to engineers using multi-modal communication

• Prepare an announcement
• Post the message on Slack: #releases, #g_distribution, #development, #backend, #quality, #security
• Post the message on Engineering Week-In-Review

@balasankarc, @twk3 Could you please take a look at the above steps for the migration and let me know your thoughts?

Since we're only migrating where MRs gets opened and merged, it seems the number of files to modify on omnibus-gitlab is minimum, could you please double-check those? Thanks!

@mayra-cabrera I think part of step one should include getting pipelines working on the security forks.

@twk3 Could you please be more specific about which pipelines are you referring?

@mayra-cabrera ignore me, the pipelines look fine: https://gitlab.com/gitlab-org/security/omnibus-gitlab/pipelines Or at least certainly good enough for a the first iteration. I wrongly assumed we had more jobs locked behind an 'only' clause.

We need to prevent developers to push to this repo.

@mayra-cabrera Confirming: This should be "prevent developers to push to stable/security branches", right? For omnibus-gitlab, dev was not only used for security development, but for regular developments too. This is where packages gets built for all OSs. So, developers need to push their feature branches to dev every now and then (unless we are mirroring all the branches from com to dev through security fork, which I think is not what we will be doing).

@balasankarc

unless we are mirroring all the branches from com to dev through security fork, which I think is not what we will be doing).

We're only mirroring the protected branches from Canonical -> Dev - > Security. Currently, our infrastructure setup looks like this:

Assuming:

• Canonical: https://gitlab.com/gitlab-org/omnibus-gitlab
• Security: gitlab.com/gitlab-org/security/omnibus-gitlab
• Build: https://dev.gitlab.org/gitlab/omnibus-gitlab

Then:

1. Protected branches from Canonical are synced to Security via push mirror
2. Branches from Security are synced to Build via push mirror.

graph LR
  classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px
  classDef security fill:#e086bd,stroke:#333,stroke-width:1px

  subgraph Canonical
    c-master(master)
    c-auto-deploy(12-8-auto-deploy-20200120)
    c-stable(12-8-stable)
    c-feature(feature/some-new-feature)
  end

  subgraph Security
    s-master(master)
    s-auto-deploy(12-8-auto-deploy-20200120)
    s-stable(12-8-stable)
  end

  subgraph Build
    b-master(master)
    b-auto-deploy(12-8-auto-deploy-20200120)
    b-stable(12-8-stable)
  end

  c-master -->|push mirror| s-master
  s-master -->|push mirror| b-master

  c-stable -->|push mirror| s-stable
  s-stable -->|push mirror| b-stable
  
  c-auto-deploy -->|push mirror | s-auto-deploy
  s-auto-deploy -->|push mirror | b-auto-deploy

  class c-master,c-stable,c-auto-deploy canonical
  class b-master,b-stable,b-auto-deploy canonical
  class s-master,s-stable,s-auto-deploy security

For omnibus-gitlab, dev was not only used for security development, but for regular developments too.

Interesting! @balasankarc I guess this could be solved by either:

1. Mirroring all the branches from Canonical to Security (not only the protected branches) or,
2. Use Omnibus Security repo for regular development too, or,
3. Continue using dev for regular development.

I'm inclined towards the first one, as Security repo should only be used for security development and dev should only be used for building packages, but I'm open to suggestions.

@rspeicher What do you think of this one? Do you foresee any undesired consequence about mirroring all branches from Canonical to Security

Cost. Mirroring all branches will create a huge number of pipelines and large number of unnecessary packages.

Is the actual problem you are highlighting here due to diverged branches between mirrors? If yes, having security in the middle between Canonical and Build will not change this workflow if I remember correctly.

Cost. Mirroring all branches will create a huge number of pipelines and large number of unnecessary packages.

Ah, in that case and to not complicate things, it should be fine to continue using dev for regular developments as this is already the case. I don't think there's any change that needs to be done in dev, so I've removed that part from the steps.

I agree. For now, let's leave omnibus-gitlab in dev as is. It already has stable, security, and auto-deploy branches protected, and most of the time only Distribution team pushes feature branches there.

mentioned in epic &153 (closed)

changed the description

mentioned in issue #714 (closed)

added priority1 label

marked the checklist item Enable issues on security/omnibus-gitlab - http://gitlab.com/gitlab-org/security/omnibus-gitlab/-/issues (Unsure why they are not enabled at the moment) as completed

changed milestone to %12.9

mentioned in commit gitlab-org/omnibus-gitlab@530e5923

mentioned in merge request gitlab-org/omnibus-gitlab!3993 (merged)

mentioned in commit gitlab-org/omnibus-gitlab@4d09b517

changed the description

mentioned in commit gitlab-org/gitlab@136e174c

mentioned in commit gitlab-org/omnibus-gitlab@546870f2

mentioned in merge request gitlab-org/gitlab!27121 (merged)

changed the description

mentioned in commit gitlab-org/omnibus-gitlab@360531b9

mentioned in commit gitlab-org/omnibus-gitlab@cf9e0392

mentioned in merge request gitlab-org/omnibus-gitlab!3998 (merged)

marked the checklist item .gitlab/merge_request_templates/Security Release.md - gitlab-org/omnibus-gitlab!3993 (merged) as completed

marked the checklist item .gitlab/issue_templates/Security developer workflow.md - gitlab-org/omnibus-gitlab!3993 (merged) as completed

mentioned in commit gitlab-org/release-tools@9512d600

mentioned in merge request gitlab-org/release-tools!907 (merged)

mentioned in commit gitlab-org/release-tools@66c63148

mentioned in commit gitlab-org/release-tools@94c73a40

mentioned in commit gitlab-org/release-tools@4cea4bf7

marked the checklist item scripts/security-harness - After this migration, dev.gitlab.org will no longer be used to prepare security MR's as completed

marked the checklist item scripts/security-harness - After this migration, dev.gitlab.org will no longer be used to prepare security MR's - gitlab-org/gitlab!27121 (diffs) as completed

changed the description

marked the checklist item lib/release_tools/securit/merge_requests_validator.rb - gitlab-org/release-tools!907 (merged) as completed

mentioned in commit gitlab-org/release-tools@b8d53709

mentioned in commit gitlab-org/release-tools@f3c13afa

mentioned in merge request gitlab-org/release-tools!910 (merged)

changed the description

mentioned in commit gitlab-org/gitlab@6cb5d109

mentioned in merge request gitlab-org/gitlab!27349 (merged)

mentioned in commit gitlab-org/gitlab@f43bab04

mentioned in commit gitlab-org/gitlab@244fb431

marked the checklist item templates/security_patch.rb - gitlab-org/release-tools!910 (merged) as completed

changed the description

marked the checklist item gitlab/issue_templates/Security%20Release.md - gitlab-org/gitlab!27349 (merged) as completed

changed the description

mentioned in commit gitlab-org/gitlab@1ccd8fc9

mentioned in merge request gitlab-org/gitlab!27427 (merged)

changed the description

changed the description

marked the checklist item Ensure master, *-stable and auto-deploy branches are protected as completed

marked the checklist item Ensure only release managers can merge and push to the protected branches. as completed

mentioned in commit gitlab-org/omnibus-gitlab@f785126d

mentioned in merge request gitlab-org/omnibus-gitlab!4012 (merged)

changed the description

mentioned in commit gitlab-org/omnibus-gitlab@fa4f387a

mentioned in commit gitlab-org/omnibus-gitlab@2221b70d

mentioned in commit gitlab-org/omnibus-gitlab@39765fa6

mentioned in commit gitlab-org/gitlab@d5d13785

mentioned in commit gitlab-org/gitlab@e799a222

mentioned in commit gitlab-org/omnibus-gitlab@53d01bb5

mentioned in commit gitlab-org/omnibus-gitlab@9d6f4960

mentioned in commit gitlab-org/omnibus-gitlab@f7ac211e

mentioned in commit gitlab-org/omnibus-gitlab@87130c8e

marked the checklist item Ensure it's using similar issue and security templates to the ones on the GitLab project - gitlab-org/omnibus-gitlab!4012 (merged) as completed

marked the checklist item Create security MR's and validate them. as completed

changed the description

mentioned in commit gitlab-org/release-tools@aa6c5367

mentioned in merge request gitlab-org/release-tools!921 (merged)

changed the description

marked the checklist item Enable tracking of security releases - gitlab-org/release-tools!891 (diffs) as completed

changed the description

mentioned in issue #743 (closed)

marked the checklist item Prepare an announcement - #743 (closed) as completed

changed the description

marked the checklist item Post the announcement on Slack: #g_distribution, #releases, #development, #backend, #security, #app-sec as completed

marked the checklist item Post the announcement on Engineering Week-In-Review as completed

changed the description

added to epic &153 (closed)

Since all the items have been completed and Omnibus security fixes were implemented on GitLab Security for the last security release, I'm closing this one. Thanks everyone!

closed

mentioned in issue #766 (closed)

added DeliveryP1 label and removed priority1 label

mentioned in merge request gitlab-org/release/docs!416 (merged)