README.md 16.4 KB
Newer Older
Sid Sijbrandij's avatar
Sid Sijbrandij committed
1
# The 5 Minute Production App
Sri's avatar
Sri committed
2

Sid Sijbrandij's avatar
Sid Sijbrandij committed
3
**Quickly** make a **stateful** app that is deployed on a **hypercloud**.
4

Sid Sijbrandij's avatar
Sid Sijbrandij committed
5
### Quickly
6

Sid Sijbrandij's avatar
Sid Sijbrandij committed
7 8
If you have written an application you should be able to deploy it within minutes with the 5-minute production app.
Many other platforms are also quick, for example Heroku. 
9
But configuring Heroku for a production application that needs statefulness (remembering users and data across deploys) is a much longer process.
Sid Sijbrandij's avatar
Sid Sijbrandij committed
10 11 12 13 14
And Heroku doesn't let you deploy on a hypercloud, which has [many advantages](#hypercloud).

### Stateful

The 5 minute production app uses the managed stateful services of a hypercloud so your data is persisted and secure.
15 16 17 18
By leveraging these managed services (databases, caching, objects storage, etc.) you have less to maintain.
Everything is provisioned through Terraform which has the following advantages:

1. Terraform is the most popular IaC tool.
19
1. Terraform works across platforms.
20 21
1. Terraform state can be [stored and viewed in GitLab](https://docs.gitlab.com/ee/user/infrastructure/#gitlab-managed-terraform-state).
1. You avoid the cost and complexity of Kubernetes.
Balasankar "Balu" C's avatar
Balasankar "Balu" C committed
22
1. You have complete control to customize and extend.
Sid Sijbrandij's avatar
Sid Sijbrandij committed
23 24 25 26 27

### Hypercloud

Most people working in organizations use the three western hyper clouds (AWS, Azure, GCP) because: 

28
1. They have a wide selection of managed stateful services (databases, caching, object storage, etc.).
Balasankar "Balu" C's avatar
Balasankar "Balu" C committed
29
1. At least one is probably already approved to be used in your organization.
Sid Sijbrandij's avatar
Sid Sijbrandij committed
30 31 32 33
1. At least one is probably already budgeted for.
1. They have a free tier which the 5 minute production app runs within.
1. They are familiar to other people in your team and organization.
1. They are reliable and secure.
34
1. They are quickly innovating (machines with ARM processors, GPUs, TPUs, etc.).
Sid Sijbrandij's avatar
Sid Sijbrandij committed
35 36 37
1. They have lower prices than services such as Heroku.
1. They allow applications room to grow, no need to move from Heroku to AWS when it becomes popular.

38 39 40 41 42 43 44 45
### Ops2Dev Philosophy

The hardest thing about an application is not the initial deploy, it's maintaining the application over time.
The 5 minute production app brings your Day 2 operations into Day 1 and makes [Day 2 operations simple](https://about.gitlab.com/topics/gitops/gitlab-enables-infrastructure-as-code/).
Use managed services as you develop, and your infrastructure scales with you.

We are working to optimize the 5 minute production app so that you'll be moving just as quickly as you would on a typical Day 1.

46 47
### Relation to Auto DevOps

48
1. [Auto DevOps](https://about.gitlab.com/stages-devops-lifecycle/auto-devops/) provides point-and-click default CI/CD templates that auto-discover your source code to detect, build, test, deploy, and monitor your applications automatically.
49
1. Auto DevOps is already [quick and easy to get started with](https://docs.gitlab.com/ee/topics/autodevops/#quick-start) by default.
50
1. Auto DevOps is stateless, however, the 5 minute production app is stateful.
51 52
1. Auto DevOps uses Kubernetes, which you can use on hyper cloud, but only the 5 minute production app gives you:
   1. The ability to use managed stateful services (databases, caching, object storage, etc.) by default.
53
   1. A free tier which the 5 minute production app runs within. Kubernetes isn't part of a free tier for any hyper cloud.
54 55
1. No Kubernetes overhead reduces cost and makes it simpler to run.
1. The objective is to eventually combine Auto DevOps and the 5 minute production app so that [everyone can contribute](https://about.gitlab.com/company/strategy/#everyone-can-contribute) using Auto DevOps in production more easily.
56 57


Sid Sijbrandij's avatar
Sid Sijbrandij committed
58 59 60 61 62 63 64 65
### Roadmap

You're looking at the primary project for the 5 minute production app.
The roadmap in [its issue board](https://gitlab.com/gitlab-org/5-minute-production-app/deploy-template/-/boards)

### Vision

- [Category direction](https://about.gitlab.com/direction/5-min-production/)
66
- [Blog: A journey from the first code to CI/CD deployments in 5 minutes?](https://about.gitlab.com/blog/2020/12/15/first-code-to-ci-cd-deployments-in-5-minutes/)
67
- [Meeting playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0Krf0LZbfg80yo08DW1c3C36)
68

Alper Akgun's avatar
Alper Akgun committed
69 70 71 72 73
#### Usage Charts

You may use the [5 Minute Production Application Sisense Dashboard](https://app.periscopedata.com/shared/b0987d39-f197-47ca-b830-edd9ea14d2fe) to see the usage for the Sisense.


Sid Sijbrandij's avatar
Sid Sijbrandij committed
74 75
### Table of Contents

Sri's avatar
Sri committed
76
1. [Video Overview](#video-overview)
Sid Sijbrandij's avatar
Sid Sijbrandij committed
77
1. [Prerequisites](#prerequisites)
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
78 79 80 81 82
1. [Infrastructure](#infrastructure)
1. [Usage](#usage)
1. [Environments](#environments)
1. [Using the Postgres Database](#using-the-postgres-database)
1. [Using the S3 Bucket](#using-the-s3-bucket)
Sri's avatar
Sri committed
83 84
1. [Using the SMTP Service](#using-the-smtp-service)
1. [Using the Redis Cluster](#using-the-redis-cluster)
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
85 86 87 88 89
1. [Providing Custom Environment Variables to Webapp](#providing-custom-environment-variables-to-webapp)
1. [Variables Provided to Webapp](#variables-exposed-to-webapp)
1. [Rollback Deployments](#rollback-deployments)
1. [Configure Infra Resources](#configure-infra-resources)
1. [List of All Configuration Variables](#list-of-all-configuration-variables)
90
1. [Custom domain](#custom-domain)
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
91 92 93
1. [Enabling SSL](#enabling-ssl)
1. [Variables](#variables)
1. [Examples](#examples)
94
1. [Feedback](#feedback)
95
1. [Contributing](#contributing)
Sri's avatar
Sri committed
96

Sri's avatar
Sri committed
97 98 99 100
### Video Overview

[![Five Minute Production](http://img.youtube.com/vi/XcDN4bx7sNs/0.jpg)](http://www.youtube.com/watch?v=XcDN4bx7sNs "Five Minute Production")

Sid Sijbrandij's avatar
Sid Sijbrandij committed
101
### Prerequisites
Sri's avatar
Sri committed
102

Sid Sijbrandij's avatar
Sid Sijbrandij committed
103 104
1. You have a Dockerized webapp with a `Dockerfile` or [Auto Build](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-build) works for your application.
1. Your containerized app runs on port 5000 (default setting for Auto Build) or `WEBAPP_PORT` is set in `.gitlab-ci.yml`
Sri's avatar
Sri committed
105

Sri's avatar
Sri committed
106 107
### Infrastructure

Sri's avatar
Sri committed
108
By default, the following AWS free tier infrastructure is provisioned:
Sri's avatar
Sri committed
109

110
- Elastic IP
Sri's avatar
Sri committed
111
- EC2 instance
112 113 114
  - Amazon Linux
  - `t2.micro`
  - Public IP
Sri's avatar
Sri committed
115
- DB instance
116 117 118
  - Postgres
  - `db.t2.micro`
  - 20gb allocated storage
Sri's avatar
Sri committed
119
- S3 Bucket
Sri's avatar
Sri committed
120 121
- (Optional) SMTP service
- (Optional) Redis service
Sri's avatar
Sri committed
122

Sri's avatar
Sri committed
123 124
### Usage

Sid Sijbrandij's avatar
Sid Sijbrandij committed
125
1. Set the following [AWS](https://aws.amazon.com/) credentials as [GitLab CI/CD environment variables](https://docs.gitlab.com/ee/ci/variables/) which you can find under Project => Settings => CI/CD => Variables. If you want to have [review apps](https://docs.gitlab.com/ee/ci/review_apps/) make sure to not [protect the variable](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable).
Sid Sijbrandij's avatar
Sid Sijbrandij committed
126
  - `AWS_ACCESS_KEY_ID` which you can create in [AWS IAM under Access Keys](https://console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials$access_key). You need to have sufficient permissions for the AWS IAM user to successfuly create resources like RDS or EC2. If you are not sure what correct permissions are then use `AdministratorAccess` as a temporay solution.
Sid Sijbrandij's avatar
Sid Sijbrandij committed
127
  - `AWS_SECRET_ACCESS_KEY` which you can create in [AWS IAM under Access Keys](https://console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials$access_key)
Sid Sijbrandij's avatar
Sid Sijbrandij committed
128 129
  - `AWS_DEFAULT_REGION` which is optional and defaults to us-east-1 if not set

130 131
![frame_generic_light_2__compressed](/uploads/ad33728c14f28f44c23362d86cadd816/frame_generic_light_2__compressed.png)

Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
132
2. Create a `.gitlab-ci.yml` file in project root with 5-minute production app [CI template](https://docs.gitlab.com/ee/ci/examples/#cicd-templates) like shown on the picture below:
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
133

134
![frame_generic_light_compressed](/uploads/91fe1bea95dc1f018bf021e81f4c6823/frame_generic_light_compressed.png)
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
135

Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
136
But if you want to use the latest version of template you should create a `.gitlab-ci.yml` file with the following content: 
Sri's avatar
Sri committed
137

Sri's avatar
Sri committed
138 139
```yaml
include:
140
  remote: https://gitlab.com/gitlab-org/5-minute-production-app/deploy-template/-/raw/stable/deploy.yml
Sri's avatar
Sri committed
141
```
Sri's avatar
Sri committed
142

Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
143 144
This is a good choice for development and testing. Or if your GitLab version is not the latest one. 

Sid Sijbrandij's avatar
Sid Sijbrandij committed
145

Sid Sijbrandij's avatar
Sid Sijbrandij committed
146
3. After the `.gitlab-ci.yml` file is added to the repository a new [pipeline](https://docs.gitlab.com/ee/ci/pipelines/) will start which you can see under the menu CI/CD => Pipelines. When the pipeline completed successful a link to your running application will be available from the menu Operations => Environments => master => View Environment.
147

Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
148 149
![frame_generic_light_4__compressed](/uploads/182a5b979f3f35fa164ff5cddf071caf/frame_generic_light_4__compressed.png)

Sri's avatar
Sri committed
150 151 152
### Environments

- Pipeline automatically creates `environments`
153
  - `environment = infrastructure + data + configuration + deployed webapp`
Sri's avatar
Sri committed
154
- Multiple `environments` are supported
155 156
  - `environments` are synced with git branches
  - One environment per branch
Sri's avatar
Sri committed
157 158
- Environment name matches `$CI_COMMIT_REF_SLUG` i.e. branch name or tag name, lowercased and
  slugified, for example:
159 160
  - `production` branch will create and deploy to `production` environment
  - `staging` branch will create and deploy to `staging` environment, etc.
Sri's avatar
Sri committed
161

Sri's avatar
Sri committed
162
### Using the Postgres Database
Sri's avatar
Sri committed
163 164

- `DATABASE_URL` is passed to the webapp container
165
  - Format: `postgres://{db_user}:{db_pass}@{db_host}:{db_port}/{db_name}`
166
- Individual variables are also passed to the webapp container:
167 168 169 170
  - `DATABASE_ENDPOINT`
  - `DATABASE_USERNAME`
  - `DATABASE_PASSWORD`
  - `DATABASE_NAME`
Sri's avatar
Sri committed
171 172
- Environment variables `DB_INITIALIZE` and `DB_MIGRATE`, if set, are executed right
  after `docker run`
173 174 175 176 177 178 179 180
  - These must contain commands that are executed after deployment
  - These commands are executed within the Docker container
  - `DB_INITIALIZE` is executed only once on first deploy, and if successful, will never be executed
    again
    - If the first execution of `DB_INITIALIZE` fails, it will be retried on next deployment
    - `DB_INITIALIZE` can be force-executed if `DB_INITIALIZE_REPEAT` is set to `"True"`
  - `DB_MIGRATE` is executed on every deployment
  - Failing `DB_INITIALIZE` or `DB_MIGRATE` **will not** rollback your deployment
Sri's avatar
Sri committed
181 182 183

### Using the S3 Bucket

Sri's avatar
Readme  
Sri committed
184
An S3 Bucket is generated for your webapp with `public-read` settings. The following env vars are
Sri's avatar
Sri committed
185
made available to your app for use: `S3_BUCKET`, `S3_BUCKET_DOMAIN` and `S3_BUCKET_REGIONAL_DOMAIN`.
Sri's avatar
Sri committed
186

Sri's avatar
Readme  
Sri committed
187 188
You will need to use your AWS credentials in addition to the S3 Bucket name for uploading content.

Sri's avatar
Sri committed
189 190 191 192 193
### Using the SMTP Service

AWS SES provides SMTP service. This service is made available to your webapp if you declare the `SMTP_FROM` variable.

`SMTP_FROM` is an email address that is the sender of emails by your webapp. AWS SES will require this email to be 
194 195
verified. In sandbox mode, the recipient emails also need to be verified. To turn off sandbox mode, please log in to your AWS 
console and configure SES. Read more at [Amazon Simple Email Service documentation](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/request-production-access.html).
Sri's avatar
Sri committed
196 197 198

### Using the Redis Cluster

Sri's avatar
Sri committed
199 200
A Redis Elasticache cluster will be provisioned for your application with REDIS_NODE_TYPE `cache.t2.micro`. Once
provisioned, the environment variables `REDIS_ADDRESS`, `REDIS_PORT`, `REDIS_AVAILABILITY_ZONE` and `REDIS_URL` are made
Sri's avatar
Sri committed
201 202
available to your webapp.

Sri's avatar
Sri committed
203
### Providing Custom Environment Variables to Webapp
Sri's avatar
Readme  
Sri committed
204 205 206 207 208 209 210 211 212 213 214 215 216

Your application might need custom environment variables. These can be passed by declaring them with
the `GL_VAR_` prefix. For example, if you wanted to pass `HELLO=WORLD`, you will need to
declare `GL_VAR_HELLO=WORLD`.

This can be done in two ways:

- Define them in the `.gitlab-ci.yml` file (as shown in
  the [example](#list-of-all-configuration-variables)), or
- Define them in the project or group environment variables set through the GitLab Web UI

**Caution** Make sure your env-var values are properly escaped. The value will be wrapped in a pair
of double-quotes `HELLO="world"` and improper escaping of value can break the deployment.
Sri's avatar
Sri committed
217 218 219

### Variables Exposed to Webapp

Sri's avatar
Readme  
Sri committed
220 221
The following variables are provided to your containerized webapp. Thus are available at runtime.

Sri's avatar
Sri committed
222 223
```yaml
- DATABASE_URL                  # postgres://{db_user}:{db_pass}@{db_host}:{db_port}/{db_name}
224
- DATABASE_ENDPOINT             # {db_host}:{db_port}
225
- DATABASE_ADDRESS              # {db_host}
226 227
- DATABASE_USERNAME             # {db_user}
- DATABASE_PASSWORD             # {db_pass}
Sri's avatar
Sri committed
228
- DATABASE_NAME                 # {db_name}
Sri's avatar
Sri committed
229

230 231 232
- AWS_ACCESS_KEY_ID             # Your AWS access key
- AWS_SECRET_ACCESS_KEY         # Your AWS access key
- AWS_DEFAULT_REGION            # Your AWS region
233

Sri's avatar
Sri committed
234 235 236 237
- S3_BUCKET                     # Environment specific S3 bucket name
- S3_BUCKET_DOMAIN              # Publicly accessible domain
- S3_BUCKET_REGIONAL_DOMAIN     # Publicly accessible regional domain

238 239 240 241 242
- SMTP_HOST                     # AWS SES SMTP server, region specific
- SMTP_FROM                     # AWS SES validated from email address
- SMTP_USER                     # SMTP user
- SMTP_PASSWORD                 # SMTP password

243 244
- REDIS_ADDRESS                 # Address of your Redis cluster
- REDIS_PORT                    # Port of your Redis cluster
Sri's avatar
Sri committed
245 246
- REDIS_AVAILABILITY_ZONE       # Availability zone in case location of data storage matters
- REDIS_URL                     # Redis hostname and port separated with a `:`
247

Sri's avatar
Sri committed
248 249 250
- GL_VAR_*                      # All variables prefixed with `GL_VAR_`
```

Sri's avatar
Sri committed
251
### Rollback Deployments
Sri's avatar
Readme  
Sri committed
252 253 254

- Clean way to rollback is to push a revert commit

Sri's avatar
Sri committed
255
### Configure Infra Resources
256

Sri's avatar
Sri committed
257 258
You can set the environment variables `TF_VAR_EC2_INSTANCE_TYPE`, `TF_VAR_PG_INSTANCE_CLASS`
and `TF_VAR_PG_ALLOCATED_STORAGE` to explicitly define the specs of infra that is provisioned.
259

Sri's avatar
Sri committed
260 261
Default values are shown in the [configuration example](#list-of-all-configuration-variables).

Sri's avatar
Sri committed
262
### List of All Configuration Variables
Sri's avatar
Sri committed
263

Sri's avatar
Readme  
Sri committed
264 265 266 267
The following variables can be defined in your `.gitlab-ci.yml` file or be made available to the
pipeline through any other mechanism. These variables are meant to configure the infrastructure or
deployment process.

Sri's avatar
Sri committed
268 269
```yaml
variables:
Sri's avatar
Sri committed
270

271 272 273 274 275
  # executed successfully once after deployment
  DB_INITIALIZE: "bundle exec rake db:setup RAILS_ENV=production"

  # force DB_INITIALIZE execution
  DB_INITIALIZE_REPEAT: "True"
Sri's avatar
Sri committed
276

277 278
  # executed after every deployment
  DB_MIGRATE: "bundle exec rake db:migrate RAILS_ENV=production"
Sri's avatar
Sri committed
279

280 281
  # configure container port bindings
  WEBAPP_PORT: 3000
Sri's avatar
Sri committed
282

283 284 285 286
  # configure infra specifications
  TF_VAR_EC2_INSTANCE_TYPE: "t2.micro"           # free tier
  TF_VAR_PG_INSTANCE_CLASS: "db.t2.micro"  # free tier
  TF_VAR_PG_ALLOCATED_STORAGE: 20          # 20gb
287
  TF_VAR_DISABLE_POSTGRES: "false"
Sri's avatar
Sri committed
288

289 290 291
  # ssl certificates
  CERT_DOMAIN: 'my-domain.com'
  CERT_EMAIL: 'admin@my-domain.com'
292

293 294
  # smtp
  SMTP_FROM: 'notifications@my-company.com'
Sri's avatar
Readme  
Sri committed
295

Sri's avatar
Sri committed
296
  # redis
297
  TF_VAR_REDIS_NODE_TYPE: 'cache.t2.micro'
298
  TF_VAR_DISABLE_REDIS: "false"
Sri's avatar
Sri committed
299

300 301 302
  # pass custom variables to webapp
  GL_VAR_HELLO: World
  GL_VAR_FOO: Bar
Sri's avatar
Sri committed
303 304
```

Sri's avatar
Sri committed
305
### Cleanup
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
306

307 308 309 310 311
The pipeline includes `destroy` job that will remove all infrastructure created by `terraform_apply`
. To prevent accidental removal of production data, we disabled `destroy` job on protected branches.
However you can always start pipeline with `CI_COMMIT_REF_PROTECTED` variable set to `false`. This
will add `destroy` job to pipeline (you still need to manually trigger it) so you can remove
infrastructure even on protected branch.
Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
312

313 314 315 316 317
### Custom domain

By default the deployment uses EC2 ip address and generates a domain name for it. 
For custom domain see [Enabling SSL](#enabling-ssl). 

Sri's avatar
Sri committed
318 319
### Enabling SSL

320
- SSL is enabled for all environments
321
- By default, the URL structure is `https://{branch-or-tag}.{public-ip}.resolve.toip.host`
322 323
- For custom domain, define `CERT_DOMAIN` variable for your pipeline
  - This can be defined in `.gitlab-ci.yml` alongwith `CERT_EMAIL`
324 325 326

### Variables

327
You can find a list of all variables this project uses [here](VARIABLES.md)
328

Sri's avatar
Sri committed
329 330
### Examples

Sri's avatar
Sri committed
331
Examples across multiple programming languages and web frameworks are listed in
Sri's avatar
Sri committed
332 333 334 335
the [examples subgroup](https://gitlab.com/gitlab-org/5-minute-production-app/examples).

Additional experiments and test cases are located in
the [sandbox subgroup](https://gitlab.com/gitlab-org/5-minute-production-app/sandbox).
336

337
We also collected helpful information on specific languages and frameworks:
338

339
* [Ruby on Rails](RAILS.md)
340

Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
341 342
If you have experience with different language/framework, please contribute to the list above, 
we especially welcome Node, Go, Spring, Django, and Phoenix examples.
343

Dmytro Zaporozhets (DZ)'s avatar
Dmytro Zaporozhets (DZ) committed
344
Simply create a file `NAME_OF_LANGUAGE_OR_FRAMEWORK.md` and put a link in the list.
345

346 347 348
### Feedback

This project is in early stage of development. And we are looking for your feedback. 
349
If everything worked well for you, feel free to mention @gitlab, @srirangan or @dzaporozhets in social networks. 
350
But if you experience any problems or have suggestions, please open an issue in this project.  
351 352 353 354

### Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).