Scaling the Security Release process on GitLab.com
Master epic for tracking progress towards:
- Moving development of security patches from dev.gitlab.org to private projects on gitlab.com.
- Automate and improve security tasks
- Include security releases as part of the auto-deploy processes
References:
Overview
This epic is split into multiple epics and issue boards
On-going
Epic | Issue board | Status |
---|---|---|
Improvements - Security Release | Issue board | ![]() |
Include Security Releases as part of the auto-deploy | TBD | ![]() |
Completed
Epic | Issue board | Status |
---|---|---|
Move the security development process as-is to Security | --- | ![]() |
Backup 1: First definitions
Mirroring and syncing
All protected branches from Canonical are synced to Build via push mirroring.
Specific protected branches from Canonical (master
, X-Y-stable
,
X-Y-auto-deploy-YYYYMMDD
) should be synced to Security via pull mirror,
prefixing all branches with security/
(e.g., master
on Canonical becomes
security/master
on Security): gitlab-com/gl-infra/delivery#533 (closed), gitlab-com/gl-infra/delivery#548 (closed)
All protected branches from Security are synced to Build via push mirroring: gitlab-com/gl-infra/delivery#473 (closed)
Because both Canonical branches and Security branches (prefixed with
security/
) are synced to Build, we can perform a regular or security
release at any time; one does not block the other.
Normal release workflow
Normal releases pull from Build using the X-Y-stable
or
X-Y-auto-deploy-YYYYMMDD
branch, depending on what we're releasing.
During release, master
, stable branches, and tags get pushed to
Canonical and Build by release-tools.
release-tools will:
- Pull the Build repository using
master
- Check and out and pull
X-Y-stable[-ee]
branch - Compile the changelog on
X-Y-stable[-ee]
andmaster
- Update the
VERSION
and component version files - Push
X-Y-stable[-ee]
andmaster
to the Canonical and Build repositories
Security release workflow
master
, are prefixed with
security/
, and we push to Security and Build (rather than Canonical and Build).
release-tools will:
- Pull the Build repository using
security/master
- Check and out and pull
security/X-Y-stable[-ee]
branch - Compile the changelog on
security/X-Y-stable[-ee]
andsecurity/master
- Update the
VERSION
and component version files - Push
security/X-Y-stable[-ee]
andsecurity/master
to the Security and Build repositories.
Publication
Security releases pull from Build using the security/X-Y-stable
or
security/X-Y-auto-deploy-YYYYMMMDD
branch, depending on what we're
releasing.
After publication, the security/*
branches (including security/master
)
are merged into their normal counterparts (e.g., security/12-3-stable
gets
pulled into 12-3-stable
) and pushed to the Canonical repository, bringing
all three repositories back in sync.
SyncRemoteService
needs to be updated for the security/
prefix.
graph LR
subgraph Canonical
c-master[master]
c-stable[12-3-stable]
c-auto-deploy[12-3-auto-deploy-20190911]
end
subgraph Build
bs-master[security/master]
bs-stable[security/12-3-stable]
bs-auto-deploy[security/12-3-auto-deploy-20190911]
end
bs-stable -->|publish| c-stable
bs-master -->|publish| c-master
bs-auto-deploy -->|publish| c-auto-deploy
All tags are synced from Build to Canonical.
graph LR
subgraph Canonical
c-tag[v12.3.2]
end
subgraph Build
bs-tag[v12.3.2]
end
bs-tag -->|publish| c-tag
Components
These are the projects that make up the GitLab product.
-
gitlab-ce
-
.gitlab/issue_templates/Security Release.md
- This gets greatly simplified through the use of associated issues and MRs.
-
bin/secpick
-
scripts/security-harness
- We'll need to check for something like
security
in the remote value rather thandev.gitlab.org
- We'll need to check for something like
-
-
gitlab-ee
- Identical to
gitlab-ce
- Identical to
-
gitlab-workhorse
Tooling
These projects orchestrate a release of the GitLab product.
-
omnibus-gitlab
-
.gitlab/issue_templates/Security developer workflow.md
- This entire document will need to be revised as part of Process changes.
-
.gitlab/merge_request_templates/Security Release.md
- This entire document will need to be revised as part of Process changes.
-
scripts/security-harness
- We'll need to check for something like
security
in the remote value rather thandev.gitlab.org
- We'll need to check for something like
-
-
release-tools
lib/release_tools/gitlab_dev_client.rb
lib/release_tools/passing_build.rb#L141
lib/release_tools/project/*
lib/release_tools/qa/issue_presenter.rb#L101-145
-
=> gitlab-org/release-tools!737 (merged)lib/release_tools/security/client.rb#L7-11
-
Remove "allowed to fail" jobs=> gitlab-org/release-tools!737 (merged) -
templates/security_patch.md.erb
- Most of this document will need to be changed as part of Process changes.
Nothing to do
-
gitlab-elasticsearch-indexer
- No mentions of
dev.gitlab.org
- No mentions of
-
gitlab-monitor
- None found
-
gitlab-shell
- No mentions of
dev.gitlab.org
- No mentions of
Process changes
- gitlab-org/release/docs
-
gitlab-org/gitlab-ce
-
.gitlab/merge_request_templates/Security Release.md
=> https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32613
-
-
gitaly
-
gitlab-pages
Backup 2- Development workflow, not germane to discussion here
Day-to-day development workflow
Features and bug fixes get developed in Canonical, merging to the master
branch.
If a merge request needs to be included in the current auto-deploy branch, it
receives the Pick into auto-deploy
label, which we cherry-pick on a regular
schedule.
If a merge request needs go into a specific stable branch to be included in a
patch release, it receives the Pick into X.Y
label, which we cherry-pick
during the patch release process.
graph LR
subgraph Canonical
c-master(master)
c-stable(12-3-stable)
c-auto-deploy(12-3-auto-deploy-20190911)
c-feature(feature/some-new-feature)
c-bug(15015-some-bug-fix)
c-feature --> c-master
c-bug --> c-master
c-bug -.->|Pick into 12.3| c-stable
c-bug -.->|Pick into auto-deploy| c-auto-deploy
end
Security development workflow
A merge request should be opened in Security, referencing the confidential
vulnerability issue from Canonical, targeting the security/master
branch.
The security/master
merge request goes through review and, when approved, gets
reassigned to the @gitlab-release-tools-bot
for merging.
graph TD
subgraph Security
s-master(security/master)
s-fix-master(security/15015-fix-mermaid-xss)
s-fix-master --> s-master
end
subgraph Canonical
c-issue((#15015 XSS in Mermaid))
end
s-fix-master -.-> c-issue
Once the security/master
merge request is reviewed, approved, and assigned to the bot,
backport merge requests can be opened in Security targeting the previous three
stable branches. These backports go through a simplified review process and then
get assigned to the @gitlab-release-tools-bot
user for merging.
graph TD
subgraph Security
s-fix-12-2(security/15015-fix-mermaid-xss-12-2)
s-fix-12-1(security/15015-fix-mermaid-xss-12-1)
s-fix-12-0(security/15015-fix-mermaid-xss-12-0)
s-stable-12-2(security/12-2-stable)
s-stable-12-1(security/12-1-stable)
s-stable-12-0(security/12-0-stable)
s-fix-12-2 --> s-stable-12-2
s-fix-12-1 --> s-stable-12-1
s-fix-12-0 --> s-stable-12-0
end
subgraph Canonical
c-issue((#15015 XSS in Mermaid))
end
s-fix-12-2 -.-> c-issue
s-fix-12-1 -.-> c-issue
s-fix-12-0 -.-> c-issue
Because all branches in Security are prefixed with security/
, security merge
requests can be merged as soon as they are deemed ready.
Security MR validation
-
Don't allow jobs to fail when using Security => gitlab-org/release-tools!737 (merged) -
Check different project paths when using Security => gitlab-org/release-tools!737 (merged)
- Show closed items