Scaling the Security Release process on GitLab.com
Master epic for tracking progress towards:
* Moving development of security patches from dev.gitlab.org to private projects on gitlab.com.
* Automate and improve security tasks
* Include security releases as part of the auto-deploy processes
References:
- [Original issue](https://gitlab.com/gitlab-org/gitlab-foss/issues/55648)
- [Design document](https://about.gitlab.com/handbook/engineering/infrastructure/library/security-releases-development/)
## Overview
This epic is split into multiple epics and issue boards
### On-going
| Epic | Issue board | Status |
| ------ | ------ | ---- |
| [Improvements - Security Release](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/153) | [Issue board](https://gitlab.com/groups/gitlab-org/-/boards/1526231?milestone_title=12.8&scope=all&utf8=%E2%9C%93&state=opened&label_name%5B%5D=security-release&label_name%5B%5D=team%3A%3ADelivery&label_name%5B%5D=security) | :hourglass_flowing_sand: |
| [Include Security Releases as part of the auto-deploy](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/109) | TBD | :hourglass_flowing_sand: |
### Completed
| Epic | Issue board | Status |
| ------ | ------ | ---- |
| [Move the security development process as-is to Security](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/121) | --- | :white_check_mark: |
---
<details>
<summary>Backup 1: First definitions</summary>
### Mirroring and syncing
_All protected branches_ from Canonical are synced to Build via push mirroring.
:white_check_mark: Existing setup is used.
_Specific protected branches_ from Canonical (`master`, `X-Y-stable`,
`X-Y-auto-deploy-YYYYMMDD`) should be synced to Security via pull mirror,
prefixing all branches with `security/` (e.g., `master` on Canonical becomes
`security/master` on Security): https://gitlab.com/gitlab-com/gl-infra/delivery/issues/533, https://gitlab.com/gitlab-com/gl-infra/delivery/issues/548
_All protected branches_ from Security are synced to Build via push mirroring: https://gitlab.com/gitlab-com/gl-infra/delivery/issues/473
Because both Canonical branches and Security branches (prefixed with
`security/`) are synced to Build, we can perform a regular or security
release at any time; one does not block the other.
### Normal release workflow
:white_check_mark: Nothing changes! :tada:
Normal releases pull from Build using the `X-Y-stable` or
`X-Y-auto-deploy-YYYYMMDD` branch, depending on what we're releasing.
During release, `master`, stable branches, and tags get pushed to
Canonical and Build by release-tools.
release-tools will:
1. Pull the Build repository using `master`
1. Check and out and pull `X-Y-stable[-ee]` branch
1. Compile the changelog on `X-Y-stable[-ee]` and `master`
1. Update the `VERSION` and component version files
1. Push `X-Y-stable[-ee]` and `master` to the Canonical and Build repositories
### Security release workflow
:hourglass: This is basically the exact same as the non-security
release process, but all branches, even `master`, are prefixed with
`security/`, and we push to Security and Build (rather than Canonical and Build).
release-tools will:
1. Pull the Build repository using `security/master`
* :white_check_mark: https://gitlab.com/gitlab-org/release-tools/merge_requests/726
1. Check and out and pull `security/X-Y-stable[-ee]` branch
* :white_check_mark: https://gitlab.com/gitlab-org/release-tools/merge_requests/726
1. Compile the changelog on `security/X-Y-stable[-ee]` and `security/master`
1. Update the `VERSION` and component version files :white_check_mark:
1. Push `security/X-Y-stable[-ee]` and `security/master` to the Security and Build repositories.
* :warning: https://gitlab.com/gitlab-com/gl-infra/delivery/issues/532
### Publication
Security releases pull from Build using the `security/X-Y-stable` or
`security/X-Y-auto-deploy-YYYYMMMDD` branch, depending on what we're
releasing.
After publication, the `security/*` branches (including `security/master`)
are merged into their normal counterparts (e.g., `security/12-3-stable` gets
pulled into `12-3-stable`) and pushed to the Canonical repository, bringing
all three repositories back in sync.
:warning: `SyncRemoteService` needs to be updated for the `security/` prefix.
```mermaid
graph LR
subgraph Canonical
c-master[master]
c-stable[12-3-stable]
c-auto-deploy[12-3-auto-deploy-20190911]
end
subgraph Build
bs-master[security/master]
bs-stable[security/12-3-stable]
bs-auto-deploy[security/12-3-auto-deploy-20190911]
end
bs-stable -->|publish| c-stable
bs-master -->|publish| c-master
bs-auto-deploy -->|publish| c-auto-deploy
```
All tags are synced from Build to Canonical.
```mermaid
graph LR
subgraph Canonical
c-tag[v12.3.2]
end
subgraph Build
bs-tag[v12.3.2]
end
bs-tag -->|publish| c-tag
```
## Components
These are the projects that make up the GitLab product.
- `gitlab-ce`
- [`.gitlab/issue_templates/Security Release.md`](https://gitlab.com/gitlab-org/gitlab-ce/blob/663b7bb4771f3261d7451b6e1d74c778fd0e3589/.gitlab/issue_templates/Security%20Release.md)
- This gets greatly simplified through the use of associated issues and MRs.
- [`bin/secpick`](https://gitlab.com/gitlab-org/gitlab-ce/blob/663b7bb4771f3261d7451b6e1d74c778fd0e3589/bin/secpick#L15)
- <https://gitlab.com/gitlab-org/gitlab-ce/blob/663b7bb4771f3261d7451b6e1d74c778fd0e3589/bin/secpick#L64-70>
- <https://gitlab.com/gitlab-org/gitlab-ce/blob/663b7bb4771f3261d7451b6e1d74c778fd0e3589/bin/secpick#L110-112>
- [`scripts/security-harness`](https://gitlab.com/gitlab-org/gitlab-ce/blob/663b7bb4771f3261d7451b6e1d74c778fd0e3589/scripts/security-harness)
- We'll need to check for something like `security` in the remote value rather than `dev.gitlab.org`
- `gitlab-ee`
- Identical to `gitlab-ce`
- `gitlab-workhorse`
- [`_support/tag.sh`](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/33a5eb7d171e9357335d4605e7c6b4209fd5a18c/_support/tag.sh#L10-11)
## Tooling
These projects orchestrate a release of the GitLab product.
- `omnibus-gitlab`
- [`.gitlab/issue_templates/Security developer workflow.md`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/fd3274cd6283f300413fc2b7a19a875328a8055b/.gitlab/issue_templates/Security%20developer%20workflow.md)
- This entire document will need to be revised as part of [Process changes](#process-changes).
- [`.gitlab/merge_request_templates/Security Release.md`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/fd3274cd6283f300413fc2b7a19a875328a8055b/.gitlab/merge_request_templates/Security%20Release.md)
- This entire document will need to be revised as part of [Process changes](#process-changes).
- [`scripts/security-harness`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/fd3274cd6283f300413fc2b7a19a875328a8055b/scripts/security-harness)
- We'll need to check for something like `security` in the remote value rather than `dev.gitlab.org`
- `release-tools`
- ~~[`lib/release_tools/gitlab_dev_client.rb`](https://gitlab.com/gitlab-org/release-tools/blob/c45103a916305bb4a80d8ea9a2762a94d0e39322/lib/release_tools/gitlab_dev_client.rb)~~
- [`lib/release_tools/passing_build.rb#L141`](https://gitlab.com/gitlab-org/release-tools/blob/c45103a916305bb4a80d8ea9a2762a94d0e39322/lib/release_tools/passing_build.rb#L141)
- ~~[`lib/release_tools/project/*`](https://gitlab.com/gitlab-org/release-tools/tree/master/lib/release_tools/project)~~
- ~~[`lib/release_tools/qa/issue_presenter.rb#L101-145`](https://gitlab.com/gitlab-org/release-tools/blob/c45103a916305bb4a80d8ea9a2762a94d0e39322/lib/release_tools/qa/issue_presenter.rb#L101-145)~~
- ~~[`lib/release_tools/security/client.rb#L7-11`](https://gitlab.com/gitlab-org/release-tools/blob/c45103a916305bb4a80d8ea9a2762a94d0e39322/lib/release_tools/security/client.rb#L7-11)~~ => https://gitlab.com/gitlab-org/release-tools/merge_requests/737
- ~~Remove ["allowed to fail" jobs](https://gitlab.com/gitlab-org/release-tools/blob/c45103a916305bb4a80d8ea9a2762a94d0e39322/lib/release_tools/security/pipeline.rb#L6-16)~~ => https://gitlab.com/gitlab-org/release-tools/merge_requests/737
- [`templates/security_patch.md.erb`](https://gitlab.com/gitlab-org/release-tools/blob/master/templates/security_patch.md.erb)
- Most of this document will need to be changed as part of [Process changes](#process-changes).
## Nothing to do
- `gitlab-elasticsearch-indexer`
- No mentions of `dev.gitlab.org`
- `gitlab-monitor`
- None found
- `gitlab-shell`
- No mentions of `dev.gitlab.org`
## Process changes
- [gitlab-org/release/docs](https://gitlab.com/gitlab-org/release/docs)
- [ ] https://gitlab.com/gitlab-org/release/docs/issues/31
- [ ] https://gitlab.com/gitlab-org/release/docs/merge_requests/119
- [gitlab-org/gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce)
- [ ] `.gitlab/merge_request_templates/Security Release.md` => https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32613
- `gitaly`
- [ ] [`doc/PROCESS.md`](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/PROCESS.md)
- `gitlab-pages`
- [ ] [`.gitlab/merge_request_templates/Security Release.md`](https://gitlab.com/gitlab-org/gitlab-pages/blob/c391d5251d67c68f89b54c54375dacd971584ead/.gitlab/merge_request_templates/Security%20Release.md)
- [ ] [`PROCESS.md`](https://gitlab.com/gitlab-org/gitlab-pages/blob/c391d5251d67c68f89b54c54375dacd971584ead/PROCESS.md)
</details>
<details>
<summary> Backup 2- Development workflow, not germane to discussion here</summary>
### Day-to-day development workflow
:white_check_mark: Nothing changes! :tada:
Features and bug fixes get developed in Canonical, merging to the `master`
branch.
If a merge request needs to be included in the current auto-deploy branch, it
receives the `Pick into auto-deploy` label, which we cherry-pick on a regular
schedule.
If a merge request needs go into a specific stable branch to be included in a
patch release, it receives the `Pick into X.Y` label, which we cherry-pick
during the patch release process.
```mermaid
graph LR
subgraph Canonical
c-master(master)
c-stable(12-3-stable)
c-auto-deploy(12-3-auto-deploy-20190911)
c-feature(feature/some-new-feature)
c-bug(15015-some-bug-fix)
c-feature --> c-master
c-bug --> c-master
c-bug -.->|Pick into 12.3| c-stable
c-bug -.->|Pick into auto-deploy| c-auto-deploy
end
```
### Security development workflow
A merge request should be opened in Security, referencing the confidential
vulnerability issue from Canonical, targeting the `security/master` branch.
The `security/master` merge request goes through review and, when approved, gets
reassigned to the `@gitlab-release-tools-bot` for merging.
```mermaid
graph TD
subgraph Security
s-master(security/master)
s-fix-master(security/15015-fix-mermaid-xss)
s-fix-master --> s-master
end
subgraph Canonical
c-issue((#15015 XSS in Mermaid))
end
s-fix-master -.-> c-issue
```
Once the `security/master` merge request is reviewed, approved, and assigned to the bot,
backport merge requests can be opened in Security targeting the previous three
stable branches. These backports go through a simplified review process and then
get assigned to the `@gitlab-release-tools-bot` user for merging.
```mermaid
graph TD
subgraph Security
s-fix-12-2(security/15015-fix-mermaid-xss-12-2)
s-fix-12-1(security/15015-fix-mermaid-xss-12-1)
s-fix-12-0(security/15015-fix-mermaid-xss-12-0)
s-stable-12-2(security/12-2-stable)
s-stable-12-1(security/12-1-stable)
s-stable-12-0(security/12-0-stable)
s-fix-12-2 --> s-stable-12-2
s-fix-12-1 --> s-stable-12-1
s-fix-12-0 --> s-stable-12-0
end
subgraph Canonical
c-issue((#15015 XSS in Mermaid))
end
s-fix-12-2 -.-> c-issue
s-fix-12-1 -.-> c-issue
s-fix-12-0 -.-> c-issue
```
Because all branches in Security are prefixed with `security/`, security merge
requests can be merged as soon as they are deemed ready.
#### Security MR validation
* :white_check_mark: Don't allow jobs to fail when using Security => https://gitlab.com/gitlab-org/release-tools/merge_requests/737
* :white_check_mark: Check different project paths when using Security => https://gitlab.com/gitlab-org/release-tools/merge_requests/737
</details>
epic