process.md 10.1 KB
Newer Older
1
2
# Security Releases general information

3
4
Security vulnerabilities in GitLab and its dependencies are to be addressed with
the highest priority.
5

6
7
8
9
Security releases are naturally very similar to [patch releases](../patch/process.md), but
on a much shorter timeline. The goal is to make a security release available as
soon as possible, while ensuring that the security issue is properly addressed
and that the fix does not introduce regressions.
10

11
12
At GitLab, we have two types of security releases:

Mark Florian's avatar
Mark Florian committed
13
 * [critical](#critical-security-releases): immediate patch and mitigation is required for a single issue
14
 * [non-critical](#non-critical-security-releases): monthly release of planned issue
15

16
17
Also see [the GitLab Maintenance Policy](https://docs.gitlab.com/ee/policy/maintenance.html)
for the information on determining supported releases and assigning versions.
18

19
## Guides by role
20

21
22
23
24
- [Release Manager]
- [Security Engineer]
- [Developer]
- [Quality Engineer]
25

26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
### Terminology

1. **Security vulnerability issue** - A confidential issue in a GitLab Canonical project (`gitlab-org/some-project`), detailing a security vulnerability and steps to reproduce.

   Usually created via HackerOne.

   Example: [gitlab-org/gitlab#193100](https://gitlab.com/gitlab-org/gitlab/issues/193100).

1. **Security release tracking issue** - A confidential issue in `gitlab-org/gitlab` which is the high-level overview of an entire security release (critical or otherwise).

   Created via [issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Security%20Release.md).

   Example: [gitlab-org/gitlab#209019](https://gitlab.com/gitlab-org/gitlab/-/issues/209019).

1. **Security implementation issue** - An issue in a GitLab Security project (`gitlab-org/security/some-project`), outlining the versions affected, detailing the steps a developer needs to take to remediate the vulnerability, and associating backports.

   Created via [issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Security%20developer%20workflow.md).

   Example: [gitlab-org/security/gitlab#80](https://gitlab.com/gitlab-org/security/gitlab/-/issues/80).

1. **Security merge request (MR)** - A merge request in a GitLab Security project (`gitlab-org/security/some-project`), resolving a vulnerability in a specific branch.

   Created via [merge request template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Security%20Release.md).

   Example: [gitlab-org/security/gitlab!314](https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/314).

1. **Release task issue** - Used by [release managers] as a checklist of steps to complete the release.

   Created via ChatOps (`/chatops run release prepare --security`).

   Example: [gitlab-org/release/tasks#1272](https://gitlab.com/gitlab-org/release/tasks/-/issues/1272).

#### Issue association

```mermaid
graph RL
  classDef canonical fill:#74bd3d
  classDef security fill:#ff7272

  secVuln(Security vulnerability Issue)
  secRelease(Security release Issue)
  secFix(Security implementation issue)
  task{{Release task issue}}

  class secVuln canonical
  class secRelease canonical
  class secFix security

  secFix -->|references| secVuln
  secFix ---|related| secRelease

  task -.->|references| secRelease
```

#### Merge request association

```mermaid
graph RL
  classDef canonical fill:#74bd3d
  classDef security fill:#ff7272

  secFix(Security implementation issue)
88
  secMR1(["Security MR 1 (default-branch)"])
89
90
91
92
93
94
95
  secMR2(["Security MR 2 (backport)"])
  secMR3(["Security MR 3 (backport)"])
  secMR4(["Security MR 4 (backport)"])

  class secFix security
  class secMR1,secMR2,secMR3,secMR4 security

96
  secMR1 & secMR2 & secMR3 & secMR4 -.->|references| secFix
97
98
99
100
101
102
103
104
```
#### Merging

```mermaid
graph TD
  classDef security fill:#ff7272

  coMerge{{"/chatops run release merge --security"}}
105
  coMergeDefaultBranch{{"/chatops run release merge --security --default-branch"}}
106

107
  secMR1(["Security MR 1 (default-branch)"])
108
109
110
111
112
  secMR2(["Security MR 2 (backport)"])
  secMR3(["Security MR 3 (backport)"])
  secMR4(["Security MR 4 (backport)"])

  autodeploy(auto-deploy)
113
  defaultbranch(default-branch)
114
115
116
117
118
119
120
121
122
123
124
125
126

  stable129(12-9-stable-ee)
  stable128(12-8-stable-ee)
  stable127(12-7-stable-ee)

  coMerge --> secMR2
  coMerge --> secMR3
  coMerge --> secMR4

  secMR2 --> |merge| stable129
  secMR3 --> |merge| stable128
  secMR4 --> |merge| stable127

127
128
129
  coMergeDefaultBranch --> secMR1
  secMR1 --> |merge| defaultbranch
  defaultbranch -.-> |cherry-pick| autodeploy
130
131
132

  class secMR1,secMR2,secMR3,secMR4 security
  class stable129,stable128,stable127 security
133
  class defaultbranch,autodeploy security
134
135
136
137
138
```

[release managers]: https://about.gitlab.com/community/release-managers/

## Non-critical Security Releases
139

140
Around the 28th of the month, we release patch versions of GitLab containing
141
non-critical security fixes. This date has been chosen as it gives enough
142
time to release a non-security patch release after standard release on the 22nd of
143
144
145
146
147
148
149
150
151
152
153
154
the month. 

### Security Release deadlines

Just as with patch releases, deadlines for non-critical security releases are at
the discretion of release managers, the 28th of the month is used as default due date.

Final deadline will depend on a variety of factors:
* GitLab.com availability.
* Workload on Release Managers.
* Stability of auto-deployments.

155
In the run-up to the due date, release managers will start processing the
156
157
security implementation issues associated with the Security Release Tracking issue.

158
159
To guarantee the security fix is included on the Security Release, **all merge requests
associated should be ready to be processed 48h before the due date** established in the
160
161
162
163
Security Release Tracking Issue.

Fixes not ready by that date may not be considered for the security release
and they'll have to go into the next one.
164

165
166
167
168
169
170
171
Security issues that are linked to the Security Release Tracking issue less than 24 hours
before the due date will be automatically unlinked by the Release Tools Bot. For
example, if the due date of the Security Release Tracking issue is 28th August, security
issues added after 27th August 00:00 UTC will be automatically unlinked. If you have
an urgent security issue that cannot wait for the next security release, you can ping
the [Release Managers](https://about.gitlab.com/community/release-managers/) for an exception.

172
## Critical Security Releases
173

174
Depending on the severity and impact of the vulnerability, an
175
176
immediate patch release consisting of just the security fix may be warranted.

177
178
179
The critical release process is a superset of a non-critical release, and for
completeness the process is written here:

180
181
182
183
184
185
186
1. The vulnerability has been reported and discussed between a [Security
   Engineer] and a [Developer]. When the timeline of a possible fix is
   established, the Security Engineer informs a [Release Manager] of a need
   for the critical security release.
1. A release manager proposes a timeline for the critical security release based
   on the other release tasks, and informs a security engineer and a [Quality
   Engineer] in the designated release issue.
187
188
189
190
191
1. Depending on the nature of the issue, a security engineer may decide that
   a temporary mitigation strategy is required. If a patch is required,
   the security engineer works with the developer and infrastructure engineer
   following the [post-deployment patch process](../deploy/post-deployment-patches.md).
   In other cases, the security engineer will work with infrastructure
192
193
194
   engineers on remediation.
1. The security engineer creates a [security operations issue](https://gitlab.com/gitlab-com/gl-security/security-operations/sirt/operations/-/issues/new?issuable_template=User+Impact+Investigation)
   using the template `user_impact_investigation` for the
195
196
   security operations team to track further investigations and other post-remediation
   activities, such as determining and notifying affected users.
197
198
1. A security engineer works on a blog post to pre-announce the critical
   security release. They also work with the marketing team to inform the
Jerome Ng's avatar
Jerome Ng committed
199
200
201
   customers of an upcoming release. They also follow the [JiHu Support Security Release Process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/#security-release-process). 
   The security engineer also prepares the second blog post which will be published 
   at release time.
202
1. The developer follows the [Developer process](developer.md) to create the
203
   artifacts necessary for a release.
204
1. A release manager prepares the release with all backports, and deploys to
205
206
207
208
209
   staging (if applicable). If there are any non security releases pending, such
   as a release candidate, release managers may decide to release those first.
1. If any post-deployment patches were applied, an on-call Security engineer
   should verify that the applied patches are still working; unless the newly
   deployed version removes the need for these patches.
210
211
212
1. A quality engineer prepares environments and executes QA tasks together with
   the security engineer to verify that all released versions have successfully
   resolved the vulnerability.
213
214
215
216
1. When all environments contain the fix, any temporary mitigation strategy is
   reverted.
1. Following deployment, a release manager coordinates with a security engineer
   on the exact timing of a release.
217
218
219
220
221
222
223
224
225
226
227
1. A release manager will promote the packages at the designated time, and merge
   the release blog post.
1. A security engineer works with the marketing team to send notification emails
   to any affected users.
1. A release manager closes all release issues.
1. A security engineer keeps the issues where the vulnerabilities were reported
   open for the next 30 days.
1. A security engineer prepares a blog post that explains the vulnerability in
   detail and releases it approximately 30 days after the original release.
1. Once the final blog post is released, a security engineer removes the
   confidentiality from the issues and closes them.
228
229
230

Each involved role should follow their own guide, and create separate issues
linking to the main release issue.
231
232
233
234

---

[Return to Guides](../README.md)
235
236
237
238
239

[Release Manager]: release-manager.md
[Security Engineer]: security-engineer.md
[Developer]: developer.md
[Quality Engineer]: quality.md