2016-07-29-the-basics-of-gitlab-ci.html.md.erb 11.7 KB
Newer Older
inem's avatar
inem committed
1
---
2 3 4
title: "GitLab CI: Run jobs sequentially, in parallel or build a custom pipeline"
description: "GitLab CI: Learn how to run jobs sequentially, in parallel, or build a custom pipeline"
categories: GitLab CI
inem's avatar
inem committed
5 6
author: Ivan Nemytchenko
author_twitter: inemation
inem's avatar
inem committed
7
image_title: '/images/blogimages/the-basics-of-gitlab-ci/hello.png'
8
categories: engineering
inem's avatar
inem committed
9 10
---

11
Let's assume that you don't know anything about what Continuous Integration is and why it's needed. Or, you just forgot. Anyway, we're starting from scratch here.
inem's avatar
inem committed
12

Amara Nwaigwe's avatar
Amara Nwaigwe committed
13
Imagine that you work on a project, where all the code consists of two text files. Moreover, it is super-critical that the concatenation of these two files contains the phrase "Hello world."
inem's avatar
inem committed
14

Amara Nwaigwe's avatar
Amara Nwaigwe committed
15
If there's no such phrase, the whole development team stays without a salary for a month. Yeah, it is that serious!
inem's avatar
inem committed
16

17 18
<!-- more -->

inem's avatar
inem committed
19
The most responsible developer wrote a small script to run every time we are about to send our code to customers.
20
The code is pretty sophisticated:
inem's avatar
inem committed
21 22 23 24 25

```bash
cat file1.txt file2.txt | grep -q "Hello world"
```

Mark Pundsack's avatar
Mark Pundsack committed
26
The problem is that there are ten developers in the team, and, you know, human factors can hit hard.
inem's avatar
inem committed
27

28
A week ago, a new guy forgot to run the script and three clients got broken builds. So you decided to solve the problem once and for all. Luckily, your code is already on GitLab, and you remember that there is a [built-in CI system](/features/gitlab-ci-cd/). Moreover, you heard at a conference that people use CI to run tests...
inem's avatar
inem committed
29

inem's avatar
inem committed
30
## Run our first test inside CI
inem's avatar
inem committed
31

32
After a couple minutes to find and read the docs, it seems like all we need is these two lines of code in a file called `.gitlab-ci.yml`:
33

inem's avatar
inem committed
34 35 36 37 38 39

```yaml
test:
  script: cat file1.txt file2.txt | grep -q 'Hello world'
```

inem's avatar
inem committed
40
Committing it, and hooray! Our build is successful:
inem's avatar
inem committed
41
![Build succeeded](/images/blogimages/the-basics-of-gitlab-ci/success.png){: .shadow}
inem's avatar
inem committed
42

Mark Pundsack's avatar
Mark Pundsack committed
43
Let's change "world" to "Africa" in the second file and check what happens:
inem's avatar
inem committed
44
![Build failed](/images/blogimages/the-basics-of-gitlab-ci/failure.png){: .shadow}
inem's avatar
inem committed
45

inem's avatar
inem committed
46
The build fails as expected!
47 48

Okay, we now have automated tests here! GitLab CI will run our test script every time we push new code to the repository.
inem's avatar
inem committed
49

inem's avatar
inem committed
50
## Make results of builds downloadable
inem's avatar
inem committed
51

Amara Nwaigwe's avatar
Amara Nwaigwe committed
52
The next business requirement is to package the code before sending it to our customers. Let's automate that as well!
Mark Pundsack's avatar
Mark Pundsack committed
53

Amara Nwaigwe's avatar
Amara Nwaigwe committed
54
All we need to do is define another job for CI. Let's name the job "package":
55

inem's avatar
inem committed
56 57 58 59
```yaml
test:
  script: cat file1.txt file2.txt | grep -q 'Hello world'

inem's avatar
inem committed
60 61
package:
  script: cat file1.txt file2.txt | gzip > package.gz
inem's avatar
inem committed
62 63
```

inem's avatar
inem committed
64
We have two tabs now:
inem's avatar
inem committed
65
![Two tabs - generated from two jobs](/images/blogimages/the-basics-of-gitlab-ci/twotabs.png){: .shadow}
inem's avatar
inem committed
66

Amara Nwaigwe's avatar
Amara Nwaigwe committed
67
However, we forgot to specify that the new file is a build _artifact_, so that it could be downloaded. We can fix it by adding an `artifacts` section:
68

inem's avatar
inem committed
69 70 71 72
```yaml
test:
  script: cat file1.txt file2.txt | grep -q 'Hello world'

inem's avatar
inem committed
73 74
package:
  script: cat file1.txt file2.txt | gzip > packaged.gz
inem's avatar
inem committed
75 76
  artifacts:
    paths:
inem's avatar
inem committed
77
    - packaged.gz
inem's avatar
inem committed
78 79 80
```

Checking... It is there:
inem's avatar
inem committed
81
![Checking the download buttons](/images/blogimages/the-basics-of-gitlab-ci/artifacts.png){: .shadow}
inem's avatar
inem committed
82 83

Perfect!
inem's avatar
inem committed
84
However, we have a problem to fix: the jobs are running in parallel, but we do not want to package our application if our tests fail.
inem's avatar
inem committed
85

Mark Pundsack's avatar
Mark Pundsack committed
86
## Run jobs sequentially
inem's avatar
inem committed
87

inem's avatar
inem committed
88
We only want to run the 'package' job if the tests are successful. Let's define the order by specifying `stages`:
89

inem's avatar
inem committed
90 91 92
```yaml
stages:
  - test
inem's avatar
inem committed
93
  - package
inem's avatar
inem committed
94 95 96 97 98

test:
  stage: test
  script: cat file1.txt file2.txt | grep -q 'Hello world'

inem's avatar
inem committed
99 100 101
package:
  stage: package
  script: cat file1.txt file2.txt | gzip > packaged.gz
inem's avatar
inem committed
102 103
  artifacts:
    paths:
inem's avatar
inem committed
104
    - packaged.gz
inem's avatar
inem committed
105 106
```

Mark Pundsack's avatar
Mark Pundsack committed
107 108 109
That should be good!

Also, we forgot to mention, that compilation (which is represented by concatenation in our case) takes a while, so we don't want to run it twice. Let's define a separate step for it:
110

inem's avatar
inem committed
111

112 113
```yaml
stages:
inem's avatar
inem committed
114 115
  - compile
  - test
inem's avatar
inem committed
116
  - package
inem's avatar
inem committed
117 118 119 120 121 122 123 124 125 126 127 128

compile:
  stage: compile
  script: cat file1.txt file2.txt > compiled.txt
  artifacts:
    paths:
    - compiled.txt

test:
  stage: test
  script: cat compiled.txt | grep -q 'Hello world'

inem's avatar
inem committed
129 130 131
package:
  stage: package
  script: cat compiled.txt | gzip > packaged.gz
inem's avatar
inem committed
132 133
  artifacts:
    paths:
inem's avatar
inem committed
134
    - packaged.gz
135
```
inem's avatar
inem committed
136 137

Let's take a look at our artifacts:
inem's avatar
inem committed
138

inem's avatar
inem committed
139
![Unnecessary artifact](/images/blogimages/the-basics-of-gitlab-ci/clean-artifacts.png){: .shadow}
inem's avatar
inem committed
140

141
Hmm, we do not need that "compile" file to be downloadable. Let's make our temporary artifacts expire by setting `expire_in` to '20 minutes':
inem's avatar
inem committed
142

143
```yaml
inem's avatar
inem committed
144 145 146 147 148 149
compile:
  stage: compile
  script: cat file1.txt file2.txt > compiled.txt
  artifacts:
    paths:
    - compiled.txt
150
    expire_in: 20 minutes
151
```
inem's avatar
inem committed
152 153

Now our config looks pretty impressive:
inem's avatar
inem committed
154

inem's avatar
inem committed
155
- We have three sequential stages to compile, test, and package our application.
Mark Pundsack's avatar
Mark Pundsack committed
156
- We are passing the compiled app to the next stages so that there's no need to run compilation twice (so it will run faster).
inem's avatar
inem committed
157
- We are storing a packaged version of our app in build artifacts for further usage.
inem's avatar
inem committed
158

159
## Learning which Docker image to use
inem's avatar
inem committed
160

161
So far so good. However, it appears our builds are still slow. Let's take a look at the logs.
inem's avatar
inem committed
162

inem's avatar
inem committed
163
![Ruby 2.1 is the logs](/images/blogimages/the-basics-of-gitlab-ci/logs.png){: .shadow}
inem's avatar
inem committed
164

165
Wait, what is this? Ruby 2.1?
inem's avatar
inem committed
166

167
Why do we need Ruby at all? Oh, GitLab.com uses Docker images to [run our builds](/2016/04/05/shared-runners/), and [by default](/gitlab-com/settings/#shared-runners) it uses the [`ruby:2.1`](https://hub.docker.com/_/ruby/) image. For sure, this image contains many packages we don't need. After a minute of googling, we figure out that there's an image called [`alpine`](https://hub.docker.com/_/alpine/) which is an almost blank Linux image.
inem's avatar
inem committed
168

169
OK, let's explicitly specify that we want to use this image by adding `image: alpine` to `.gitlab-ci.yml`.
inem's avatar
inem committed
170
Now we're talking! We shaved almost 3 minutes off:
inem's avatar
inem committed
171

inem's avatar
inem committed
172
![Build speed improved](/images/blogimages/the-basics-of-gitlab-ci/speed.png){: .shadow}
inem's avatar
inem committed
173

Mark Pundsack's avatar
Mark Pundsack committed
174
It looks like [there's](https://hub.docker.com/_/mysql/) [a lot of](https://hub.docker.com/_/python/) [public images](https://hub.docker.com/_/java/) [around](https://hub.docker.com/_/php/). So we can just grab one for our technology stack. It makes sense to specify an image which contains no extra software because it minimizes download time.
inem's avatar
inem committed
175

inem's avatar
inem committed
176
## Dealing with complex scenarios
inem's avatar
inem committed
177

178
So far so good. However, let's suppose we have a new client who wants us to package our app into `.iso` image instead of `.gz`
179
Since CI does the whole work, we can just add one more job to it.
180
ISO images can be created using the [mkisofs](http://linuxcommand.org/man_pages/mkisofs8.html) command. Here's how our config should look:
181 182

```yaml
Mark Pundsack's avatar
Mark Pundsack committed
183 184
image: alpine

185 186 187
stages:
  - compile
  - test
inem's avatar
inem committed
188
  - package
189

inem's avatar
inem committed
190
# ... "compile" and "test" jobs are skipped here for the sake of compactness
191

192
pack-gz:
inem's avatar
inem committed
193 194
  stage: package
  script: cat compiled.txt | gzip > packaged.gz
195 196
  artifacts:
    paths:
inem's avatar
inem committed
197
    - packaged.gz
198

199
pack-iso:
inem's avatar
inem committed
200
  stage: package
201
  script:
inem's avatar
inem committed
202
  - mkisofs -o ./packaged.iso ./compiled.txt
203 204
  artifacts:
    paths:
inem's avatar
inem committed
205
    - packaged.iso
206 207
```

208
Note that job names shouldn't necessarily be the same. In fact if they were the same, it wouldn't be possible to make the jobs run in parallel inside the same stage. Hence, think of same names of jobs & stages as coincidence.
209 210

Anyhow, the build is failing:
inem's avatar
inem committed
211
![Failed build because of missing mkisofs](/images/blogimages/the-basics-of-gitlab-ci/mkisofs.png){: .shadow}
inem's avatar
inem committed
212

213

214
The problem is that `mkisofs` is not included in the `alpine` image, so we need to install it first.
215

216
## Dealing with missing software/packages
217

218
According to the [Alpine Linux website](https://pkgs.alpinelinux.org/contents?file=mkisofs&path=&name=&branch=&repo=&arch=x86) `mkisofs` is a part of the `xorriso` and `cdrkit` packages. These are the magic commands that we need to run to install a package:
219 220 221

```bash
echo "ipv6" >> /etc/modules  # enable networking
222 223
apk update                   # update packages list
apk add xorriso              # install package
224 225
```

Mark Pundsack's avatar
Mark Pundsack committed
226
For CI, these are just like any other commands. The full list of commands we need to pass to `script` section should look like this:
227 228 229 230 231 232

```yml
script:
- echo "ipv6" >> /etc/modules
- apk update
- apk add xorriso
inem's avatar
inem committed
233
- mkisofs -o ./packaged.iso ./compiled.txt
234 235
```

236
However, to make it semantically correct, let's put commands related to package installation in `before_script`. Note that if you use `before_script` at the top level of a configuration, then the commands will run before all jobs. In our case, we just want it to run before one specific job.
237 238

Our final version of `.gitlab-ci.yml`:
239

inem's avatar
inem committed
240

241
```yaml
inem's avatar
inem committed
242 243 244 245 246
image: alpine

stages:
  - compile
  - test
inem's avatar
inem committed
247
  - package
inem's avatar
inem committed
248 249 250 251 252 253 254

compile:
  stage: compile
  script: cat file1.txt file2.txt > compiled.txt
  artifacts:
    paths:
    - compiled.txt
255
    expire_in: 20 minutes
inem's avatar
inem committed
256

257 258 259 260
test:
  stage: test
  script: cat compiled.txt | grep -q 'Hello world'

261
pack-gz:
inem's avatar
inem committed
262 263
  stage: package
  script: cat compiled.txt | gzip > packaged.gz
inem's avatar
inem committed
264 265
  artifacts:
    paths:
inem's avatar
inem committed
266
    - packaged.gz
inem's avatar
inem committed
267

268
pack-iso:
inem's avatar
inem committed
269
  stage: package
270 271 272 273 274
  before_script:
  - echo "ipv6" >> /etc/modules
  - apk update
  - apk add xorriso
  script:
inem's avatar
inem committed
275
  - mkisofs -o ./packaged.iso ./compiled.txt
inem's avatar
inem committed
276 277
  artifacts:
    paths:
inem's avatar
inem committed
278
    - packaged.iso
inem's avatar
inem committed
279 280
```

281
Wow, it looks like we have just created a pipeline! We have three sequential stages, but jobs `pack-gz` and `pack-iso`, inside the `package` stage, are running in parallel:
inem's avatar
inem committed
282

inem's avatar
inem committed
283
![Pipelines illustration](/images/blogimages/the-basics-of-gitlab-ci/pipeline.png)
inem's avatar
inem committed
284

inem's avatar
inem committed
285
## Summary
inem's avatar
inem committed
286

287
There's much more to cover but let's stop here for now. I hope you liked this short story. All examples were made intentionally trivial so that you could learn the concepts of GitLab CI without being distracted by an unfamiliar technology stack. Let's wrap up what we have learned:
inem's avatar
inem committed
288

289
1. To delegate some work to GitLab CI you should define one or more [jobs](http://docs.gitlab.com/ee/ci/yaml/README.html#jobs) in `.gitlab-ci.yml`.
Amara Nwaigwe's avatar
Amara Nwaigwe committed
290
2. Jobs should have names and it's your responsibility to come up with good ones.
Mark Pundsack's avatar
Mark Pundsack committed
291 292 293
3. Every job contains a set of rules & instructions for GitLab CI, defined by [special keywords](#keywords).
4. Jobs can run sequentially, in parallel, or you can define a custom pipeline.
5. You can pass files between jobs and store them in build artifacts so that they can be downloaded from the interface.
inem's avatar
inem committed
294

inem's avatar
inem committed
295
Below is the last section containing a more formal description of terms and keywords we used, as well as links to the detailed description of GitLab CI functionality.
inem's avatar
inem committed
296 297 298 299 300 301 302


### Keywords description & links to the documentation
{: #keywords}

| Keyword/term       | Description |
|---------------|--------------------|
303 304 305 306 307 308 309
| [.gitlab-ci.yml](http://docs.gitlab.com/ee/ci/yaml/README.html#gitlab-ci-yml) | File containing all definitions of how your project should be built |
| [script](http://docs.gitlab.com/ee/ci/yaml/README.html#script)        | Defines a shell script to be executed |
| [before_script](http://docs.gitlab.com/ee/ci/yaml/README.html#before_script) | Used to define the command that should be run before (all) jobs |
| [image](http://docs.gitlab.com/ee/ci/docker/using_docker_images.html#what-is-image) | Defines what docker image to use |
| [stage](http://docs.gitlab.com/ee/ci/yaml/README.html#stages)         | Defines a pipeline stage (default: `test`) |
| [artifacts](http://docs.gitlab.com/ee/ci/yaml/README.html#artifacts)     | Defines a list of build artifacts |
| [artifacts:expire_in](http://docs.gitlab.com/ee/ci/yaml/README.html#artifactsexpire_in) | Used to delete uploaded artifacts after the specified time |
inem's avatar
inem committed
310 311
| [pipelines](http://docs.gitlab.com/ee/ci/pipelines.html#pipelines) | A pipeline is a group of builds that get executed in stages (batches) |

312 313


314
See also:
Matija Čupić's avatar
Matija Čupić committed
315 316
- [Learn how to set up deployment pipeline to multiple environments](/2016/08/26/ci-deployment-and-environments/)
- [Migrate from Jenkins to GitLab CI](/2016/07/22/building-our-web-app-on-gitlab-ci/)
317
- [Decrease build time with custom docker image](http://beenje.github.io/blog/posts/gitlab-ci-and-conda/)
318 319 320