pipelines.md 12.6 KB
Newer Older
1
# Creating and using CI/CD pipelines
2

3
> Introduced in GitLab 8.8.
4

5 6
## Introduction

7
Pipelines are the top-level component of continuous integration, delivery, and deployment.
8 9 10 11

Pipelines comprise:

- Jobs that define what to run. For example, code compilation or test runs.
12
- Stages that define when and how to run. For example, that tests run only after code compilation.
13

14
Multiple jobs in the same stage are executed by [Runners](runners/README.md) in parallel, if there are enough concurrent [Runners](runners/README.md).
15

16
If all the jobs in a stage:
17 18

- Succeed, the pipeline moves on to the next stage.
19
- Fail, the next stage is not (usually) executed and the pipeline ends early.
20

21
NOTE: **Note:**
22
If you have a [mirrored repository that GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter),
23 24 25
you may need to enable pipeline triggering in your project's
**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.

26
### Simple pipeline example
27

28
As an example, imagine a pipeline consisting of four stages, executed in the following order:
29

30 31 32 33
- `build`, with a job called `compile`.
- `test`, with two jobs called `test` and `test2`.
- `staging`, with a job called `deploy-to-stage`.
- `production`, with a job called `deploy-to-prod`.
34

35
## Visualizing pipelines
36

37
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5742) in GitLab 8.11.
38

39
Pipelines can be complex structures with many sequential and parallel jobs.
40

41
To make it easier to understand the flow of a pipeline, GitLab has pipeline graphs for viewing pipelines
42
and their statuses.
43

44 45
Pipeline graphs can be displayed in two different ways, depending on the page you
access the graph from.
46

47 48
NOTE: **Note:**
GitLab capitalizes the stages' names when shown in the [pipeline graphs](#pipeline-graphs).
49

50
### Regular pipeline graphs
51

52
Regular pipeline graphs show the names of the jobs of each stage. Regular pipeline graphs can
53
be found when you are on a [single pipeline page](#seeing-pipeline-status). For example:
54

55
![Pipelines example](img/pipelines.png)
56

57
### Pipeline mini graphs
58

59 60
Pipeline mini graphs take less space and can tell you at a
quick glance if all jobs passed or something failed. The pipeline mini graph can
61
be found when you navigate to:
62

63 64 65
- The pipelines index page.
- A single commit page.
- A merge request page.
66

67 68 69
Pipeline mini graphs allow you to see all related jobs for a single commit and the net result
of each stage of your pipeline. This allows you to quickly see what failed and
fix it.
70

71
Stages in pipeline mini graphs are collapsible. Hover your mouse over them and click to expand their jobs.
72

73 74 75
| Mini graph                                                   | Mini graph expanded                                            |
|:-------------------------------------------------------------|:---------------------------------------------------------------|
| ![Pipelines mini graph](img/pipelines_mini_graph_simple.png) | ![Pipelines mini graph extended](img/pipelines_mini_graph.png) |
76

77
### Job ordering in pipeline graphs
78

79
Job ordering depends on the type of pipeline graph. For [regular pipeline graphs](#regular-pipeline-graphs), jobs are sorted by name.
80

81 82
For [pipeline mini graphs](#pipeline-mini-graphs) ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9760)
in GitLab 9.0), jobs are sorted by severity and then by name.
83

84
The order of severity is:
85

86 87 88 89 90 91 92 93 94 95
- failed
- warning
- pending
- running
- manual
- scheduled
- canceled
- success
- skipped
- created
96

97
For example:
98

99
![Pipeline mini graph sorting](img/pipelines_mini_graph_sorting.png)
100

101
### How pipeline duration is calculated
102

103
Total running time for a given pipeline excludes retries and pending
104
(queued) time.
105

106
Each job is represented as a `Period`, which consists of:
107

108 109
- `Period#first` (when the job started).
- `Period#last` (when the job finished).
110

111
A simple example is:
112

113 114 115
- A (1, 3)
- B (2, 4)
- C (6, 7)
116

117
In the example:
118

119 120 121
- A begins at 1 and ends at 3.
- B begins at 2 and ends at 4.
- C begins at 6 and ends at 7.
122

123
Visually, it can be viewed as:
124

125 126 127 128 129 130
```text
0  1  2  3  4  5  6  7
   AAAAAAA
      BBBBBBB
                  CCCC
```
131

132
The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time is:
133

134 135 136
```text
(4 - 1) + (7 - 6) => 4
```
137

138
## Configuring pipelines
139

140
Pipelines, and their component jobs and stages, are defined in the [`.gitlab-ci.yml`](yaml/README.md) file for each project.
141

142
In particular:
143

144 145
- Jobs are the [basic configuration](yaml/README.html#introduction) component.
- Stages are defined using the [`stages`](yaml/README.html#stages) keyword.
146

147
For all available configuration options, see the [GitLab CI/CD Pipeline Configuration Reference](yaml/README.md).
148

149
### Settings and schedules
150

151 152
In addition to configuring jobs through `.gitlab-ci.yml`, additional configuration options are available
through the GitLab UI:
153

154 155
- Pipeline settings for each project. For more information, see [Pipeline settings](../user/project/pipelines/settings.md).
- Schedules for pipelines. For more information, see [Pipeline schedules](../user/project/pipelines/schedules.md).
156

157
### Grouping jobs
158

159
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6242) in GitLab 8.12.
160

161 162
If you have many similar jobs, your [pipeline graph](#visualizing-pipelines) becomes long and hard
to read.
163

164
For that reason, similar jobs can automatically be grouped together.
165 166
If the job names are formatted in certain ways, they will be collapsed into
a single group in regular pipeline graphs (not the mini graphs).
167

168 169 170 171 172 173
You'll know when a pipeline has grouped jobs if you don't see the retry or
cancel button inside them. Hovering over them will show the number of grouped
jobs. Click to expand them.

![Grouped pipelines](img/pipelines_grouped.png)

174 175 176
#### Configuring grouping

In the pipeline [configuration file](yaml/README.md), job names must include two numbers separated with one of
177 178
the following (you can even use them interchangeably):

179 180 181 182 183 184
- A space.
- A slash (`/`).
- A colon (`:`).

NOTE: **Note:**
More specifically, it uses [this](https://gitlab.com/gitlab-org/gitlab-ce/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) regular expression: `\d+[\s:\/\\]+\d+\s*`.
185

186
#### How grouping works
187 188 189 190 191 192

The jobs will be ordered by comparing those two numbers from left to right. You
usually want the first to be the index and the second the total.

For example, the following jobs will be grouped under a job named `test`:

193 194 195
- `test 0 3`
- `test 1 3`
- `test 2 3`
196 197 198

The following jobs will be grouped under a job named `test ruby`:

199 200
- `test 1:2 ruby`
- `test 2:2 ruby`
201 202 203

The following jobs will be grouped under a job named `test ruby` as well:

204 205 206
- `1/3 test ruby`
- `2/3 test ruby`
- `3/3 test ruby`
207

208
### Pipelines for merge requests
209

210 211
GitLab supports configuring pipelines that run only for merge requests. For more information, see
[Pipelines for merge requests](merge_request_pipelines/index.md).
212

213
### Badges
214

215
Pipeline status and test coverage report badges are available and configurable for each project.
216

217
For information on adding pipeline badges to projects, see [Pipeline badges](../user/project/pipelines/settings.md#pipeline-badges).
218

219
## Multi-project pipelines **[PREMIUM]**
220

221
Pipelines for different projects can be combined and visualized together.
222

223
For more information, see [Multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html).
224

225
## Working with pipelines
226

227
In general, pipelines are executed automatically and require no intervention once created.
228

229
However, there are instances where you'll need to interact with pipelines. These are documented below.
230

231
### Manually executing pipelines
232

233
Pipelines can be manually executed, with predefined or manually-specified [variables](variables/README.md).
234

235 236
You might do this if the results of a pipeline (for example, a code build) is required outside the normal
operation of the pipeline.
237

238
To execute a pipeline manually:
239

240 241 242 243 244 245
1. Navigate to your project's **CI/CD > Pipelines**.
1. Click on the **Run Pipeline** button.
1. On the **Run Pipeline** page:
    1. Select the branch to run the pipeline for in the **Create for** field.
    1. Enter any [environment variables](variables/README.md) required for the pipeline run.
    1. Click the **Create pipeline** button.
246

247
The pipeline will execute the jobs as configured.
248

249
### Accessing pipelines
250

251 252 253
You can find the current and historical pipeline runs under your project's
**CI/CD > Pipelines** page. Clicking on a pipeline will show the jobs that were run for
that pipeline.
254

255
![Pipelines index page](img/pipelines_index.png)
256

257
You can also access pipelines for a merge request by navigating to its **Pipelines** tab.
258

259
### Accessing individual jobs
260

261
When you access a pipeline, you can see the related jobs for that pipeline.
262

263
Clicking on an individual job will show you its job trace, and allow you to:
264

265 266 267 268 269 270 271 272 273 274 275 276 277 278 279
- Cancel the job.
- Retry the job.
- Erase the job trace.

### Seeing the failure reason for jobs

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782) in GitLab 10.7.

When a pipeline fails or is allowed to fail, there are several places where you
can quickly check the reason it failed:

- In the pipeline graph, on the pipeline detail view.
- In the pipeline widgets, in the merge requests and commit pages.
- In the job views, in the global and detailed views of a job.

280
In each place, if you hover over the failed job you can see the reason it failed.
281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296

![Pipeline detail](img/job_failure_reason.png)

From [GitLab 10.8](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17814),
you can also see the reason it failed on the Job detail page.

### Manual actions from pipeline graphs

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7931) in GitLab 8.15.

Manual actions, configured using the [`when:manual`](yaml/README.md#whenmanual) parameter,
allow you to require manual interaction before moving forward in the pipeline.

You can do this straight from the pipeline graph. Just click on the play button
to execute that particular job.

297 298
For example, your pipeline start automatically, but require manual action to
[deploy to production](environments.md#manually-deploying-to-environments). In the example below, the `production`
299 300 301 302 303 304 305
stage has a job with a manual action.

![Pipelines example](img/pipelines.png)

### Delay a job in a pipeline graph

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4.
306

fschl's avatar
fschl committed
307
When you do not want to run a job immediately, you can use the [`when:delayed`](yaml/README.md#whendelayed) parameter to
308
delay a job's execution for a certain period.
309

310 311
This is especially useful for timed incremental rollout where new code is rolled out gradually.

312
For example, if you start rolling out new code and:
313

314 315 316
- Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%.
- Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline
  and [rolling](environments.md#rolling-back-changes) back to the last stable version.
317 318 319 320 321 322 323 324 325 326 327 328

![Pipelines example](img/pipeline_incremental_rollout.png)

### Using the API

GitLab provides API endpoints to:

- Perform basic functions. For more information, see [Pipelines API](../api/pipelines.md).
- Maintain pipeline schedules. For more information, see [Pipeline schedules API](../api/pipeline_schedules.md).
- Trigger pipeline runs. For more information, see:
  - [Triggering pipelines through the API](triggers/README.md).
  - [Pipeline triggers API](../api/pipeline_triggers.md).
329

330 331 332 333 334 335 336 337
## Security on protected branches

A strict security model is enforced when pipelines are executed on
[protected branches](../user/project/protected_branches.md).

The following actions are allowed on protected branches only if the user is
[allowed to merge or push](../user/project/protected_branches.md#using-the-allowed-to-merge-and-allowed-to-push-settings)
on that specific branch:
338

339
- Run manual pipelines (using the [Web UI](#manually-executing-pipelines) or pipelines API).
340 341 342
- Run scheduled pipelines.
- Run pipelines using triggers.
- Trigger manual actions on existing pipelines.
343
- Retry or cancel existing jobs (using the Web UI or pipelines API).
344

345
**Variables** marked as **protected** are accessible only to jobs that
346
run on protected branches, preventing untrusted users getting unintended access to
347 348 349 350 351 352 353
sensitive information like deployment credentials and tokens.

**Runners** marked as **protected** can run jobs only on protected
branches, avoiding untrusted code to be executed on the protected runner and
preserving deployment keys and other credentials from being unintentionally
accessed. In order to ensure that jobs intended to be executed on protected
runners will not use regular runners, they must be tagged accordingly.