process.md 10.1 KB
Newer Older
1 2
# Security Releases general information

3 4
Security vulnerabilities in GitLab and its dependencies are to be addressed with
the highest priority.
5

6 7 8 9
Security releases are naturally very similar to [patch releases](../patch/process.md), but
on a much shorter timeline. The goal is to make a security release available as
soon as possible, while ensuring that the security issue is properly addressed
and that the fix does not introduce regressions.
10

11 12 13 14
At GitLab, we have two types of security releases:

 * [critical](#critical-security-release): immediate patch and mitigation is required for a single issue
 * [non-critical](#non-critical-security-releases): monthly release of planned issue
15

16 17
Also see [the GitLab Maintenance Policy](https://docs.gitlab.com/ee/policy/maintenance.html)
for the information on determining supported releases and assigning versions.
18

19
## General process overview
20

21 22
Release manager process is [described here](release-manager.md). The release
manager also makes sure that all the deadlines (described below) are
23
respected. Any delay needs to be escalated to Sr. Manager of Infrastructure.
24

25
Security engineers need to follow the process [defined here](security-engineer.md).
26

John Jarvis's avatar
John Jarvis committed
27
Developers need to follow the process [defined here](developer.md).
28

29 30
Quality engineers need to follow the process [defined here](quality.md).

31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
### Terminology

1. **Security vulnerability issue** - A confidential issue in a GitLab Canonical project (`gitlab-org/some-project`), detailing a security vulnerability and steps to reproduce.

   Usually created via HackerOne.

   Example: [gitlab-org/gitlab#193100](https://gitlab.com/gitlab-org/gitlab/issues/193100).

1. **Security release tracking issue** - A confidential issue in `gitlab-org/gitlab` which is the high-level overview of an entire security release (critical or otherwise).

   Created via [issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Security%20Release.md).

   Example: [gitlab-org/gitlab#209019](https://gitlab.com/gitlab-org/gitlab/-/issues/209019).

1. **Security implementation issue** - An issue in a GitLab Security project (`gitlab-org/security/some-project`), outlining the versions affected, detailing the steps a developer needs to take to remediate the vulnerability, and associating backports.

   Created via [issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Security%20developer%20workflow.md).

   Example: [gitlab-org/security/gitlab#80](https://gitlab.com/gitlab-org/security/gitlab/-/issues/80).

1. **Security merge request (MR)** - A merge request in a GitLab Security project (`gitlab-org/security/some-project`), resolving a vulnerability in a specific branch.

   Created via [merge request template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Security%20Release.md).

   Example: [gitlab-org/security/gitlab!314](https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/314).

1. **Release task issue** - Used by [release managers] as a checklist of steps to complete the release.

   Created via ChatOps (`/chatops run release prepare --security`).

   Example: [gitlab-org/release/tasks#1272](https://gitlab.com/gitlab-org/release/tasks/-/issues/1272).

#### Issue association

```mermaid
graph RL
  classDef canonical fill:#74bd3d
  classDef security fill:#ff7272

  secVuln(Security vulnerability Issue)
  secRelease(Security release Issue)
  secFix(Security implementation issue)
  task{{Release task issue}}

  class secVuln canonical
  class secRelease canonical
  class secFix security

  secFix -->|references| secVuln
  secFix ---|related| secRelease

  task -.->|references| secRelease
```

#### Merge request association

```mermaid
graph RL
  classDef canonical fill:#74bd3d
  classDef security fill:#ff7272

  secFix(Security implementation issue)
  secMR1(["Security MR 1 (master)"])
  secMR2(["Security MR 2 (backport)"])
  secMR3(["Security MR 3 (backport)"])
  secMR4(["Security MR 4 (backport)"])

  class secFix security
  class secMR1,secMR2,secMR3,secMR4 security

  secMR1 -->|closes| secFix
  secMR2 & secMR3 & secMR4 -.->|references| secFix
```
#### Merging

```mermaid
graph TD
  classDef security fill:#ff7272

  coMerge{{"/chatops run release merge --security"}}
  coMergeMaster{{"/chatops run release merge --security --master"}}

  secMR1(["Security MR 1 (master)"])
  secMR2(["Security MR 2 (backport)"])
  secMR3(["Security MR 3 (backport)"])
  secMR4(["Security MR 4 (backport)"])

  autodeploy(auto-deploy)

  stable129(12-9-stable-ee)
  stable128(12-8-stable-ee)
  stable127(12-7-stable-ee)

  coMerge --> secMR2
  coMerge --> secMR3
  coMerge --> secMR4

  secMR2 --> |merge| stable129
  secMR3 --> |merge| stable128
  secMR4 --> |merge| stable127

  coMergeMaster --> secMR1
  secMR1 --> |merge| master
  master -.-> |cherry-pick| autodeploy

  class secMR1,secMR2,secMR3,secMR4 security
  class stable129,stable128,stable127 security
  class master,autodeploy security
```

*NOTE:* The `--master` flag is not mutually exclusive and would also merge the backports if not already merged.

[release managers]: https://about.gitlab.com/community/release-managers/

## Non-critical Security Releases
146

147
Around the 28th of the month, we are releasing patch versions of GitLab containing
148 149
non-critical security fixes. This date has been chosen as it gives enough
time to release regular patch releases after standard release on the 22nd of
150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170
the month. 

### Security Release deadlines

Just as with patch releases, deadlines for non-critical security releases are at
the discretion of release managers, the 28th of the month is used as default due date.

Final deadline will depend on a variety of factors:
* GitLab.com availability.
* Workload on Release Managers.
* Stability of auto-deployments.

In the runup to the due date, release managers will start processing the
security implementation issues associated with the Security Release Tracking issue.

To guarantee the security fix is included on the Security Release, all merge requests
associated should be ready to be processed 48h before the due date established in the
Security Release Tracking Issue.

Fixes not ready by that date may not be considered for the security release
and they'll have to go into the next one.
171

172
## Critical Security Releases
173

174
Depending on the severity and impact of the vulnerability, an
175 176
immediate patch release consisting of just the security fix may be warranted.

177 178 179
The critical release process is a superset of a non-critical release, and for
completeness the process is written here:

180 181 182 183 184 185 186
1. The vulnerability has been reported and discussed between a [Security
   Engineer] and a [Developer]. When the timeline of a possible fix is
   established, the Security Engineer informs a [Release Manager] of a need
   for the critical security release.
1. A release manager proposes a timeline for the critical security release based
   on the other release tasks, and informs a security engineer and a [Quality
   Engineer] in the designated release issue.
187 188 189 190 191 192
1. Depending on the nature of the issue, a security engineer may decide that
   a temporary mitigation strategy is required. If a patch is required,
   the security engineer works with the developer and infrastructure engineer
   following the [post-deployment patch process](../deploy/post-deployment-patches.md).
   In other cases, the security engineer will work with infrastructure
   engineers on remediations.
Antony Saba's avatar
Antony Saba committed
193 194
1. The security engineer creates a [security operations issue](https://gitlab.com/gitlab-com/gl-security/operations/issues/new?issuable_template=User%20Impact%20Investigation)
   using the "User Impact Investigation Template" for the
195 196
   security operations team to track further investigations and other post-remediation
   activities, such as determining and notifying affected users.
197 198 199 200
1. A security engineer works on a blog post to pre-announce the critical
   security release. They also work with the marketing team to inform the
   customers of an upcoming release. The security engineer also prepares the
   second blog post which will be published at release time.
201 202 203
1. The developer follows the [Developer process](developer.md) to create the
   artifacts necessary for a release.  When complete, the developer assigns
   merge requests to the release managers.
204
1. A release manager prepares the release with all backports, and deploys to
205 206 207 208 209
   staging (if applicable). If there are any non security releases pending, such
   as a release candidate, release managers may decide to release those first.
1. If any post-deployment patches were applied, an on-call Security engineer
   should verify that the applied patches are still working; unless the newly
   deployed version removes the need for these patches.
210 211 212
1. A quality engineer prepares environments and executes QA tasks together with
   the security engineer to verify that all released versions have successfully
   resolved the vulnerability.
213
1. When the QA is completed, release manager deploys to the other environments.
214 215 216 217
1. When all environments contain the fix, any temporary mitigation strategy is
   reverted.
1. Following deployment, a release manager coordinates with a security engineer
   on the exact timing of a release.
218 219 220 221 222 223 224 225 226 227 228
1. A release manager will promote the packages at the designated time, and merge
   the release blog post.
1. A security engineer works with the marketing team to send notification emails
   to any affected users.
1. A release manager closes all release issues.
1. A security engineer keeps the issues where the vulnerabilities were reported
   open for the next 30 days.
1. A security engineer prepares a blog post that explains the vulnerability in
   detail and releases it approximately 30 days after the original release.
1. Once the final blog post is released, a security engineer removes the
   confidentiality from the issues and closes them.
229 230 231

Each involved role should follow their own guide, and create separate issues
linking to the main release issue.
232

233
## Guides by role
234

235 236 237 238
- [Release Manager]
- [Security Engineer]
- [Developer]
- [Quality Engineer]
239 240 241 242

---

[Return to Guides](../README.md)
243 244 245 246 247

[Release Manager]: release-manager.md
[Security Engineer]: security-engineer.md
[Developer]: developer.md
[Quality Engineer]: quality.md