index.md 19.5 KB
Newer Older
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
1 2
# Auto DevOps

3
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37115) in GitLab 10.0.
4
> - Generally available on GitLab 11.0.
5

6 7 8 9
Auto DevOps provides pre-defined CI/CD configuration allowing you to automatically
detect, build, test, deploy, and monitor your applications. Leveraging CI/CD
best practices and tools, Auto DevOps aims to simplify the setup and execution
of a mature and modern software development lifecycle.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
10 11 12

## Overview

13
You can spend a lot of effort to set up the workflow and processes required to
14
build, deploy, and monitor your project. It gets worse when your company has
15 16 17 18 19 20 21 22 23 24
hundreds, if not thousands, of projects to maintain. With new projects
constantly starting up, the entire software development process becomes
impossibly complex to manage.

Auto DevOps provides you a seamless software development process by
automatically detecting all dependencies and language technologies required to
test, build, package, deploy, and monitor every project with minimal
configuration. Automation enables consistency across your projects, seamless
management of processes, and faster creation of new projects: push your code,
and GitLab does the rest, improving your productivity and efficiency.
25

26
For an introduction to Auto DevOps, watch [AutoDevOps in GitLab 11.0](https://youtu.be/0Tc0YYBxqi4).
27

28 29
For requirements, see [Requirements for Auto DevOps](requirements.md) for more information.

30 31
## Enabled by default

32
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41729) in GitLab 11.3.
33

34 35
Auto DevOps is enabled by default for all projects and attempts to run on all pipelines
in each project. An instance administrator can enable or disable this default in the
36
[Auto DevOps settings](../../user/admin_area/settings/continuous_integration.md#auto-devops).
37
Auto DevOps automatically disables in individual projects on their first pipeline failure,
38 39
if it has not been explicitly enabled for the project.

40
Since [GitLab 12.7](https://gitlab.com/gitlab-org/gitlab/-/issues/26655), Auto DevOps
41
runs on pipelines automatically only if a [`Dockerfile` or matching buildpack](stages.md#auto-build)
42 43 44 45
exists.

If a [CI/CD configuration file](../../ci/yaml/README.md) is present in the project,
it will continue to be used, whether or not Auto DevOps is enabled.
46

47 48
## Quick start

49
If you're using GitLab.com, see the [quick start guide](quick_start_guide.md)
50
for setting up Auto DevOps with GitLab.com and a Kubernetes cluster on Google Kubernetes
51 52
Engine (GKE).

53
If you use a self-managed instance of GitLab, you must configure the
54
[Google OAuth2 OmniAuth Provider](../../integration/google.md) before
55 56
configuring a cluster on GKE. After configuring the provider, you can follow
the steps in the [quick start guide](quick_start_guide.md) to get started.
57

58
In [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/208132) and later, it is
59
possible to leverage Auto DevOps to deploy to [AWS ECS](requirements.md#auto-devops-requirements-for-amazon-ecs).
60

61 62
## Comparison to application platforms and PaaS

63
Auto DevOps provides features often included in an application
64
platform or a Platform as a Service (PaaS). It takes inspiration from the
65
innovative work done by [Heroku](https://www.heroku.com/) and goes beyond it
66
in multiple ways:
67

68 69 70
- Auto DevOps works with any Kubernetes cluster; you're not limited to running
  on GitLab's infrastructure. (Note that many features also work without Kubernetes).
- There is no additional cost (no markup on the infrastructure costs), and you
71
  can use a Kubernetes cluster you host or Containers as a Service on any
72 73 74 75
  public cloud (for example, [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)).
- Auto DevOps has more features including security testing, performance testing,
  and code quality testing.
- Auto DevOps offers an incremental graduation path. If you need advanced customizations,
76
  you can start modifying the templates without starting over on a
77
  completely different platform. Review the [customizing](customize.md) documentation for more information.
78 79

## Features
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
80

Jason Young's avatar
Jason Young committed
81
Comprised of a set of [stages](stages.md), Auto DevOps brings these best practices to your
82
project in a simple and automatic way:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
83

84 85
1. [Auto Build](stages.md#auto-build)
1. [Auto Test](stages.md#auto-test)
86 87 88
1. [Auto Code Quality](stages.md#auto-code-quality)
1. [Auto SAST (Static Application Security Testing)](stages.md#auto-sast)
1. [Auto Secret Detection](stages.md#auto-secret-detection)
89 90 91
1. [Auto Dependency Scanning](stages.md#auto-dependency-scanning) **(ULTIMATE)**
1. [Auto License Compliance](stages.md#auto-license-compliance) **(ULTIMATE)**
1. [Auto Container Scanning](stages.md#auto-container-scanning) **(ULTIMATE)**
92
1. [Auto Review Apps](stages.md#auto-review-apps)
93
1. [Auto DAST (Dynamic Application Security Testing)](stages.md#auto-dast) **(ULTIMATE)**
94
1. [Auto Deploy](stages.md#auto-deploy)
95
1. [Auto Browser Performance Testing](stages.md#auto-browser-performance-testing) **(PREMIUM)**
96
1. [Auto Monitoring](stages.md#auto-monitoring)
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
97

98
As Auto DevOps relies on many different components, you should have a basic
99
knowledge of the following:
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
100

101
- [Kubernetes](https://kubernetes.io/docs/home/)
102
- [Helm](https://helm.sh/docs/)
103 104 105
- [Docker](https://docs.docker.com)
- [GitLab Runner](https://docs.gitlab.com/runner/)
- [Prometheus](https://prometheus.io/docs/introduction/overview/)
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
106

107 108 109 110
Auto DevOps provides great defaults for all the stages and makes use of
[CI templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). You can, however,
[customize](customize.md) almost everything to your needs, and
[manage Auto DevOps with GitLab APIs](customize.md#extend-auto-devops-with-the-api).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
111

112 113
For an overview on the creation of Auto DevOps, read more
[in this blog post](https://about.gitlab.com/blog/2017/06/29/whats-next-for-gitlab-ci/).
Marcia Ramos's avatar
Marcia Ramos committed
114

115
NOTE: **Note:**
Evan Read's avatar
Evan Read committed
116 117 118
Kubernetes clusters can [be used without](../../user/project/clusters/index.md)
Auto DevOps.

119 120 121 122
## Kubernetes requirements

See [Auto DevOps requirements for Kubernetes](requirements.md#auto-devops-requirements-for-kubernetes).

123
## Auto DevOps base domain
124

125
The Auto DevOps base domain is required to use
126
[Auto Review Apps](stages.md#auto-review-apps), [Auto Deploy](stages.md#auto-deploy), and
127 128 129
[Auto Monitoring](stages.md#auto-monitoring). You can define the base domain in
any of the following places:

Jason Young's avatar
Jason Young committed
130
- either under the cluster's settings, whether for an instance,
131 132
  [projects](../../user/project/clusters/index.md#base-domain) or
  [groups](../../user/group/clusters/index.md#base-domain)
133
- or at the project level as a variable: `KUBE_INGRESS_BASE_DOMAIN`
Jason Young's avatar
Jason Young committed
134
- or at the group level as a variable: `KUBE_INGRESS_BASE_DOMAIN`
135
- or as an instance-wide fallback in **Admin Area > Settings** under the
Jason Young's avatar
Jason Young committed
136
  **Continuous Integration and Delivery** section
137

138
The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence
139
as other environment [variables](../../ci/variables/README.md#priority-of-environment-variables).
Jason Young's avatar
Jason Young committed
140 141
If the CI/CD variable is not set and the cluster setting is left blank, the instance-wide **Auto DevOps domain**
setting will be used if set.
142

143
TIP: **Tip:**
144 145 146
If you use the [GitLab managed app for Ingress](../../user/clusters/applications.md#ingress),
the URL endpoint should be automatically configured for you. All you must do
is use its value for the `KUBE_INGRESS_BASE_DOMAIN` variable.
147 148

NOTE: **Note:**
149
`AUTO_DEVOPS_DOMAIN` was [deprecated in GitLab 11.8](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52363)
150
and replaced with `KUBE_INGRESS_BASE_DOMAIN`, and removed in
151
[GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/56959).
152

153 154
Auto DevOps requires a wildcard DNS A record matching the base domain(s). For
a base domain of `example.com`, you'd need a DNS entry like:
155

156
```plaintext
157 158 159
*.example.com   3600     A     1.2.3.4
```

160 161 162 163
In this case, the deployed applications are served from `example.com`, and `1.2.3.4`
is the IP address of your load balancer; generally NGINX ([see requirements](#requirements)).
Setting up the DNS record is beyond the scope of this document; check with your
DNS provider for information.
164

165 166 167
Alternatively, you can use free public services like [nip.io](https://nip.io)
which provide automatic wildcard DNS without any configuration. For [nip.io](https://nip.io),
set the Auto DevOps base domain to `1.2.3.4.nip.io`.
168

169 170
After completing setup, all requests hit the load balancer, which routes requests
to the Kubernetes pods running your application.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
171

172 173 174 175
### AWS ECS

See [Auto DevOps requirements for Amazon ECS](requirements.md#auto-devops-requirements-for-amazon-ecs).

176
## Enabling/Disabling Auto DevOps
177

178 179 180
When first using Auto DevOps, review the [requirements](#requirements) to ensure
all the necessary components to make full use of Auto DevOps are available. First-time
users should follow the [quick start guide](quick_start_guide.md).
181

182 183 184
GitLab.com users can enable or disable Auto DevOps only at the project level.
Self-managed users can enable or disable Auto DevOps at the project, group, or
instance level.
185

186
### At the project level
187

188
If enabling, check that your project does not have a `.gitlab-ci.yml`, or if one exists, remove it.
189

190
1. Go to your project's **Settings > CI/CD > Auto DevOps**.
191 192 193 194
1. Select the **Default to Auto DevOps pipeline** checkbox to enable it.
1. (Optional, but recommended) When enabling, you can add in the
   [base domain](#auto-devops-base-domain) Auto DevOps uses to
   [deploy your application](stages.md#auto-deploy),
195
   and choose the [deployment strategy](#deployment-strategy).
196 197
1. Click **Save changes** for the changes to take effect.

198
After enabling the feature, an Auto DevOps pipeline is triggered on the `master` branch.
199

200
### At the group level
201

202
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52447) in GitLab 11.10.
203

204 205
Only administrators and group owners can enable or disable Auto DevOps at the group level.

206 207 208
When enabling or disabling Auto DevOps at group level, group configuration is
implicitly used for the subgroups and projects inside that group, unless Auto DevOps
is specifically enabled or disabled on the subgroup or project.
209

210
To enable or disable Auto DevOps at the group level:
211

212
1. Go to your group's **Settings > CI/CD > Auto DevOps** page.
213 214
1. Select the **Default to Auto DevOps pipeline** checkbox to enable it.
1. Click **Save changes** for the changes to take effect.
215

216
### At the instance level (Administrators only)
Fabio Busatto's avatar
Fabio Busatto committed
217

218 219
Even when disabled at the instance level, group owners and project maintainers can still enable
Auto DevOps at the group and project level, respectively.
220

221
1. Go to **Admin Area > Settings > Continuous Integration and Deployment**.
222 223 224
1. Select **Default to Auto DevOps pipeline for all projects** to enable it.
1. (Optional) You can set up the Auto DevOps [base domain](#auto-devops-base-domain),
   for Auto Deploy and Auto Review Apps to use.
225
1. Click **Save changes** for the changes to take effect.
226

227
### Deployment strategy
Matija Čupić's avatar
Matija Čupić committed
228

229
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/38542) in GitLab 11.0.
230 231

You can change the deployment strategy used by Auto DevOps by going to your
232
project's **Settings > CI/CD > Auto DevOps**. The following options
233
are available:
Matija Čupić's avatar
Matija Čupić committed
234

235
- **Continuous deployment to production**: Enables [Auto Deploy](stages.md#auto-deploy)
236 237
  with `master` branch directly deployed to production.
- **Continuous deployment to production using timed incremental rollout**: Sets the
238
  [`INCREMENTAL_ROLLOUT_MODE`](customize.md#timed-incremental-rollout-to-production) variable
239
  to `timed`. Production deployments execute with a 5 minute delay between
240 241
  each increment in rollout.
- **Automatic deployment to staging, manual deployment to production**: Sets the
242
  [`STAGING_ENABLED`](customize.md#deploy-policy-for-staging-and-production-environments) and
243
  [`INCREMENTAL_ROLLOUT_MODE`](customize.md#incremental-rollout-to-production) variables
244 245 246 247
  to `1` and `manual`. This means:

  - `master` branch is directly deployed to staging.
  - Manual actions are provided for incremental rollout to production.
Matija Čupić's avatar
Matija Čupić committed
248

249 250 251 252
TIP: **Tip:**
Use the [blue-green deployment](../../ci/environments/incremental_rollouts.md#blue-green-deployment) technique
to minimize downtime and risk.

253
## Using multiple Kubernetes clusters
254

255 256
When using Auto DevOps, you can deploy different environments to
different Kubernetes clusters, due to the 1:1 connection
257
[existing between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters).
258

Jason Young's avatar
Jason Young committed
259
The [Deploy Job template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml)
260
used by Auto DevOps currently defines 3 environment names:
261 262 263 264 265

- `review/` (every environment starting with `review/`)
- `staging`
- `production`

266 267 268
Those environments are tied to jobs using [Auto Deploy](stages.md#auto-deploy), so
except for the environment scope, they must have a different deployment domain.
You must define a separate `KUBE_INGRESS_BASE_DOMAIN` variable for each of the above
Suzanne Selhorn's avatar
Suzanne Selhorn committed
269
[based on the environment](../../ci/variables/README.md#limit-the-environment-scopes-of-environment-variables).
270

271
The following table is an example of how to configure the three different clusters:
272 273 274

| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes |
|--------------|---------------------------|-------------------------------------------|----------------------------|---|
275 276
| review       | `review/*`                | `review.example.com`                      | `review/*`                 | The review cluster which runs all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, used by every environment name starting with `review/`. |
| staging      | `staging`                 | `staging.example.com`                     | `staging`                  | (Optional) The staging cluster which runs the deployments of the staging environments. You must [enable it first](customize.md#deploy-policy-for-staging-and-production-environments). |
277
| production   | `production`              | `example.com`                             | `production`               | The production cluster which runs the production environment deployments. You can use [incremental rollouts](customize.md#incremental-rollout-to-production). |
278 279 280

To add a different cluster for each environment:

Evan Read's avatar
Evan Read committed
281
1. Navigate to your project's **Operations > Kubernetes**.
282 283
1. Create the Kubernetes clusters with their respective environment scope, as
   described from the table above.
284 285
1. After creating the clusters, navigate to each cluster and install
   Ingress. Wait for the Ingress IP address to be assigned.
286
1. Make sure you've [configured your DNS](#auto-devops-base-domain) with the
287
   specified Auto DevOps domains.
Evan Read's avatar
Evan Read committed
288
1. Navigate to each cluster's page, through **Operations > Kubernetes**,
289 290
   and add the domain based on its Ingress IP address.

291
After completing configuration, you can test your setup by creating a merge request
292
and verifying your application is deployed as a Review App in the Kubernetes
293 294 295
cluster with the `review/*` environment scope. Similarly, you can check the
other environments.

296 297 298 299
## Limitations

The following restrictions apply.

300 301
### Private registry support

302 303
No documented way of using private container registry with Auto DevOps exists.
We strongly advise using GitLab Container Registry with Auto DevOps to
304 305
simplify configuration and prevent any unforeseen issues.

306
### Install applications behind a proxy
307

308
GitLab's Helm integration does not support installing applications when
309 310
behind a proxy. Users who want to do so must inject their proxy settings
into the installation pods at runtime, such as by using a
311 312
[`PodPreset`](https://kubernetes.io/docs/concepts/workloads/pods/podpreset/):

313
```yaml
314 315 316 317 318 319 320 321 322 323 324 325 326
apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: gitlab-managed-apps-default-proxy
  namespace: gitlab-managed-apps
spec:
   env:
    - name: http_proxy
      value: "PUT_YOUR_HTTP_PROXY_HERE"
    - name: https_proxy
      value: "PUT_YOUR_HTTPS_PROXY_HERE"
```

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
327 328
## Troubleshooting

329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348
### Unable to select a buildpack

Auto Build and Auto Test may fail to detect your language or framework with the
following error:

```plaintext
Step 5/11 : RUN /bin/herokuish buildpack build
 ---> Running in eb468cd46085
    -----> Unable to select a buildpack
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 1
```

The following are possible reasons:

- Your application may be missing the key files the buildpack is looking for.
  Ruby applications require a `Gemfile` to be properly detected,
  even though it's possible to write a Ruby app without a `Gemfile`.
- No buildpack may exist for your application. Try specifying a
  [custom buildpack](customize.md#custom-buildpacks).

349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364
### Pipeline that extends Auto DevOps with only / except fails

If your pipeline fails with the following message:

```plaintext
Found errors in your .gitlab-ci.yml:

  jobs:test config key may not be used with `rules`: only
```

This error appears when the included job’s rules configuration has been overridden with the `only` or `except` syntax.
To fix this issue, you must either:

- Transition your `only/except` syntax to rules.
- (Temporarily) Pin your templates to the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10).

365 366 367 368 369
### Failure to create a Kubernetes namespace

Auto Deploy will fail if GitLab can't create a Kubernetes namespace and
service account for your project. For help debugging this issue, see
[Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
370

371 372 373 374 375
### Detected an existing PostgreSQL database

After upgrading to GitLab 13.0, you may encounter this message when deploying
with Auto DevOps:

376 377
```plaintext
Detected an existing PostgreSQL database installed on the
378 379
deprecated channel 1, but the current channel is set to 2. The default
channel changed to 2 in of GitLab 13.0.
380 381
[...]
```
382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419

Auto DevOps, by default, installs an in-cluster PostgreSQL database alongside
your application. The default installation method changed in GitLab 13.0, and
upgrading existing databases requires user involvement. The two installation
methods are:

- **channel 1 (deprecated):** Pulls in the database as a dependency of the associated
  Helm chart. Only supports Kubernetes versions up to version 1.15.
- **channel 2 (current):** Installs the database as an independent Helm chart. Required
  for using the in-cluster database feature with Kubernetes versions 1.16 and greater.

If you receive this error, you can do one of the following actions:

- You can *safely* ignore the warning and continue using the channel 1 PostgreSQL
  database by setting `AUTO_DEVOPS_POSTGRES_CHANNEL` to `1` and redeploying.

- You can delete the channel 1 PostgreSQL database and install a fresh channel 2
  database by setting `AUTO_DEVOPS_POSTGRES_DELETE_V1` to a non-empty value and
  redeploying.

  DANGER: **Danger:**
  Deleting the channel 1 PostgreSQL database permanently deletes the existing
  channel 1 database and all its data. See
  [Upgrading PostgreSQL](upgrading_postgresql.md)
  for more information on backing up and upgrading your database.

- If you are not using the in-cluster database, you can set
  `POSTGRES_ENABLED` to `false` and re-deploy. This option is especially relevant to
  users of *custom charts without the in-chart PostgreSQL dependency*.
  Database auto-detection is based on the `postgresql.enabled` Helm value for
  your release. This value is set based on the `POSTGRES_ENABLED` CI variable
  and persisted by Helm, regardless of whether or not your chart uses the
  variable.

DANGER: **Danger:**
Setting `POSTGRES_ENABLED` to `false` permanently deletes any existing
channel 1 database for your environment.

420 421
## Development guides

422
[Development guide for Auto DevOps](../../development/auto_devops.md)