README.md 18.8 KB
Newer Older
Guillaume's avatar
Guillaume committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367
Deploying Symfony test applications on Kubernetes with GitLab Auto DevOps, k3s and Let's Encrypt  - A 30m guide
======

Intro
===
Deployment of tests applications from specific development branches is a daily need for us at [Singulart](https://www.singulart.com/). Iterations on our products happen fast and so is the need of quickly deploying applications to staging for QA or others testing purposes.

Such applications only need a subset of the requirements needed for live applications (no backup or monitoring required for instance), and as such represent a good introduction in learning more about Kubernetes and its integration with GitLab and its *AutoDevOps* pipeline.

In today's primer, I'd like to introduce how RancherLabs's k3s and GitLab make the deployment of staging PHP applications easy and, thanks to k3s, at a fraction of the cost that could be a cloud-hosted Kubernetes cluster for testing purposes.

For this demo, the deployment of a `PHP7-Symfony 4-MySQL` application will be detailed, with the setup of a k3s cluster to use with GitLab.

Let's get started!

Server
===
First thing first, you need a GNU/Linux server where to run your [k3s flavored Kubernetes cluster](https://k3s.io/), so up to you, but you'll need 512MB of ram to get started with k3s.

For this demo I'm using a **Debian 9** based [OVH VPS SSD3](https://www.ovh.com/fr/vps/vps-ssd.xml) with 8GB Ram, 80GB SSD at a modest cost of $USD15/m. Plenty of room for running our testing apps! The server comes with nothing installed, but that's ok, we don't even need  to install Docker as k3s ships with [containerd](https://containerd.io/) instead.

*  The server has a public IP of `50.42.42.42` (demo)
*  A wildcard DNS has been set up on `*.k3s.yourdomain.com` to `50.42.42.42`

> A wildcard DNS is required because automatic domain names such as `myapp-<commitID>-<env>.k3s.yourdomain.com` will be generated when deploying applications from GitLab on your cluster.

Installing your Lightweight Kubernetes cluster with k3s
===

> Before following the install guide for k3s, you must note that GitLab will have to install important Kubernetes applications to deploy web applications with AutoDevOps. Because we will use the **GitLab-managed nginx ingress controller** to route requests to our k8s hosted applications, you must install k3s without the default [traefik](https://traefik.io/) ingress service. 

> More on the GitLab managed apps later!

On your server, as root, [install a server+agent k3s](https://github.com/rancher/k3s/blob/master/README.md) without traefik as follow:

```
# curl -sfL https://get.k3s.io | sh -s - server --no-deploy traefik

# Check for Ready node, takes maybe 30 seconds
```

Check for the installation status of k3s:

```
# k3s kubectl get node
NAME        STATUS   ROLES    AGE   VERSION
vpsXXXXXX   Ready    <none>   65m   v1.13.5-k3s.1

# k3s kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-7748f7f6df-nrvv6   1/1     Running   0          19s
```

Good, k3s is installed, traefik is not, and as you can see, a PhD in k8s clusterology was not required to have maybe yout first in-house Kubernetes cluster running!

Now is also a good time to check what is publicly or not accessible on the box:

```
~# netstat -tanpu|grep -i listen
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:10250         0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:6445          0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      17726/k3s
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      974/sshd
tcp        0      0 127.0.1.1:10010         0.0.0.0:*               LISTEN      17744/containerd
tcp6       0      0 :::6443                 :::*                    LISTEN      17726/k3s
tcp6       0      0 :::22                   :::*                    LISTEN      974/sshd
```

The only public endpoint (except SSH) is our k3s API on `*:6443`, which translate here at  the address `https://50.42.42.42:6443`.

GitLab and the Kubernetes integration
===

> Here a self-hosted, GitLab Community Edition v11.6 is used, but the instructions are the same if you use gitlab.com.

As discussed before, for AutoDevOps to work, and because we are demoing on a brand new k8s cluster, GitLab will have to install some utility apps on your cluster. Thoses apps will be installed in a dedicated namespace on your cluster named `gitlab-managed-apps`. 

As for your own GitLab projects, the AutoDevOps pipeline will deploy them in their **own** dedicated namespaces (one per project).

As such, GitLab requires some accesses and permissions on your cluster that we are about to set up.

Kubernetes permissions
====

Today, modern Kubernetes clusters, such as the one you just deployed with [k3s](https://k3s.io/), use [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to manage access permissions of users and services to resources in the cluster. 

As discussed previously, GitLab [needs an access](https://gitlab.com/help/user/project/clusters/index.md#role-based-access-control-rbac) to your cluster for the following tasks:
 
1.  Manage Gitlab applications in the dedicated `gitlab-managed-apps` namespace.
2.  Ability to create namespaces and projects resources (secrets, deployments...) for each of the GitLab projects you choose to integrate with your Kubernetes cluster.

To simplify, I have put all the necessary permissions to grant on the cluster for the demo in [this YAML file](https://gist.githubusercontent.com/ipernet/ffc9bcf4db3facc752340e224f592fff/raw/b4e4e1feaa65451cc0b64fa5de0a3add654cf841/rancher-gitlab-managed-apps.yml).

> Beware that these permissions basically gives an `admin access` to your entire cluster to the *gitlab-sa* service account!


```
$ k3s kubectl apply -f https://gitlab.com/ipernet/gitlab-symfony-autodevops/raw/master/res/k8s-rbac-gitlab.yaml

namespace/gitlab-managed-apps created
serviceaccount/gitlab-sa created
role.rbac.authorization.k8s.io/gitlab-role created
rolebinding.rbac.authorization.k8s.io/gitlab-rb created
clusterrolebinding.rbac.authorization.k8s.io/gitlab-cluster created
secret/gitlab-secret created
```

GitLab project settings
====

Now, let's to your Gitlab and create a new project.

Once the project is created, go to: *Operations > Kubernetes* and choose *Add existing cluster*:

**Kubernetes cluster name**

This is the name of your k3s cluster.

Nothing fancy here, it's "*default*" by default :)

```
# k3s kubectl config get-clusters
NAME
default
```

**API URL**

`https://<server IP accessible from your GitLab instance>:6443`

For the demo, `https://50.42.42.42:6443`.


**CA Certificate**

Your cluster CA. It has been created when installing k3s, and the certificate is available on the server in the kubeconfig file created when installing k3s at `/etc/rancher/k3s/k3s.yaml`

This one-liner should be able to extract it and output it in plain text:

```
$ cat /etc/rancher/k3s/k3s.yaml | grep certificate-authority-data|awk -F": " '{print $2}'|base64 -d
-----BEGIN CERTIFICATE-----
MIIC5DCCAcygAwIBAgIBADANBgkqhkiG9w0BAQsFADAjMRAwDgYDVQQKEwdrM3Mt
b3JnMQ8wDQYDVQQDEwZrM3MtY2EwHhcNMTkwMzI0MTYyMDA4zIxMTYy...
-----END CERTIFICATE-----
```

> This is the content you must copy on GitLab

**Token**

The access token that was created for the `gitlab-sa` access account when importing/creating all the necessary permissions for GitLab, token stored in the `gitlab-secret` Kubernetes secret.

You should be able to get the access token as follow:

```
$ k3s kubectl describe secrets/gitlab-secret -n gitlab-managed-apps | grep token:|awk -F"token:" '{print $NF}'|awk '{$1=$1};1'

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pbynaXRsYWIty5pby9zZ...
```

**Project namespace (optional, unique)**

If you leave it empty, GitLab will use a random name, better put something familliar here, such as the name of your project or:

`gitlab-symfony-autodevops`


**RBAC-enabled cluster**

k3s has deployed a RBAC cluster, so this box must be checked!

> You can now submit all the settings.


![](docs/img/gitlab-k8s-integration.png)


GitLab managed apps
====

Once you validate the cluster settings, it's time to install all the "utility" apps you'll need to deploy our demo Symfony application.

Here's a quick description of the [apps GitLab offers]((https://gitlab.com/help/user/project/clusters/index.md#installing-applications)) to install and manage for you on your cluster:

- [Helm/Tiller](https://helm.sh/docs/) deploys your app from a descriptive document named an helm chart.
- [nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress), a nginx-based service that will act as a router on your cluster and redirects requests to the applications deployed on your cluster.
- [cert-manager](https://github.com/helm/charts/tree/master/stable/cert-manager), take care of managing TLS certificates for your application using Let's Encrypt CA.

So, to enabled a fully automated deployment of our app:

- Install `Helm Tiller`
- Then  `Ingress` 
- Then `Cert-manager` (please provides a real email for the `issuer`)
- Then `Gitab Runner` (if you have no runner yet setup with GitLab)

You should at the end have the following apps installed:

![](docs/img/gitlab-managed-apps.png)

Now let's work on our test project!

The test project
===

I have chosen to deploy a Symfony based application along with its MySQL database. The code is the default [`symfony/website-skeleton`](https://github.com/symfony/website-skeleton) with a Doctrine entity added.

The deployment of the app will be done with the official [Auto-DevOps.gitlab-ci.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml), that has been extended with a custom [*deploy()*](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/.gitlab-ci.yml#L938) function to support the deployment of a `mysql` database, instead of the default `postgres`.

The application will also have its own [`Dockerfile`](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/Dockerfile), built from the official `php:7.2-fpm` image. Because a `Dockerfile` is available at the root of the app's repository, [no Heroku builback will be used](https://docs.gitlab.com/ee/topics/AutoDevOps/#auto-build-using-heroku-buildpacks) to package the application during the *Auto-DevOps.gitlab-ci.yml* `build` stage, and such you have the flexibility to adapt this starter image to your needs.

Finally, the application will have its own Helm Chart, extended from the [official GitLab auto-deploy app](https://gitlab.com/charts/auto-deploy-app) to support the 3 following changes :

1. `mysql` replaces `postgres` as the database requirement.
2. An intermediate nginx container implements the [Symfony rewriting rules](https://symfony.com/doc/current/setup/web_server_configuration.html#nginx) and redirect PHP requests to the application's `php:7.2-fpm` container (the one described with our `Dockerfile`).
3. Symfony environnement variables, sourced from the GitLab project's secrets, are set to be attached to the application running container.

Project environment variables configuration
===

The key of deploying your application with the GitLab AutoDevOps feature lies in the configuration of various `CI/CD Environment variables`.

These variables (and app secrets) can be managed for your GitLab's project in `Settings > CI/CD Environment variables`.

Staging only deployment
===

Today's focus is to deploy non-production applications to a `staging` environment, so let's disable most of the built-in stages we don't need  in our CI/CD pipeline.

Set the following environement variables in your project's settings:

| Variable name | Value    | 
| ------------- | -------- | 
| CODE_QUALITY_DISABLED       | true   |
| CONTAINER_SCANNING_DISABLED        | true      |
| DEPENDENCY_SCANNING_DISABLED|true      |
| DAST_DISABLED| true     |
| LICENSE_MANAGEMENT_DISABLED| true      |
| PERFORMANCE_DISABLED| true      |
| REVIEW_DISABLED| true     |
| SAST_DISABLED| true      |
| TEST_DISABLED| true      |
| STAGING_ENABLED|true      |

Applications base domain variable
===
The official [Auto-DevOps.gitlab-ci.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) script expects to know the base domain where to deploy your app's releases.

> It's the base domain you have set up a wildcard DNS for at the beginning of the article.

**Add the domain as another project's variable:**

| Variable name | Value    | 
| ------------- | -------- | 
| KUBE_INGRESS_BASE_DOMAIN| k3s.yourdomain.com |

The CI file will [by default](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml#L322) deploy staging apps from the `master` branch to `https://<gitlab user>-<project name>-staging.$KUBE_INGRESS_BASE_DOMAIN`.

> By leveraging the power of the [gitlab-ci.yml](https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic) file, you can deploy whatever branch you want.

MySQL app database and accesses
===

A MySQL container will be deployed along with our demo Symfony app using the [MySQL Helm Chart](https://github.com/helm/charts/tree/master/stable/mysql) as a requirements of our project's own chart.

The rewritten [*deploy()*](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/.gitlab-ci.yml#L938) function added to the project's own [`gitlab-ci.yml`](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/.gitlab-ci.yml) file sets what must be the default `database`, `user` and `password` to be created when the MySQL Helm chart will be deployed on the cluster. These settings must also be shared to your application, so let's set them as expected by both `gitlab-ci.yml` and the app chart:

| Variable name | Value    | Description | 
| ------------- | -------- |  -------- | 
| MYSQL_VERSION| 5.7 | The version (tag) of the official MySQL docker image to use for the MySQL server.
| K8S_SECRET_DB_NAME| mydb | The name of the database you want to have for your project.
| K8S_SECRET_DB_USER| myuser | The user who will have full access to the project database.
| K8S_SECRET_DB_PASSWORD| password | A MySQL compatible password.

Notes that there are 2 things not defined here:

**1.  MySQL `root` password**
> As we don't really need root access, a random password [will be created](https://github.com/helm/charts/blob/master/stable/mysql/values.yaml#L15) by default.
> 
**2. The MYSQL `Host` variable**
> The  variable is dynamically generated at the deployment, based on the app release name (defaults to [`$CI_ENVIRONMENT_SLUG`](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/.gitlab-ci.yml#L976)) and the name of the MySQL Helm chart ([`mysql`](https://github.com/helm/charts/blob/master/stable/mysql/templates/_helpers.tpl#L6)). The resulting variable (*`staging-mysql`*) is set by the chart as an [environment variable](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/chart/templates/deployment.yaml#L72) to the app's PHP container.

**Finally**, The `initial database schema` will be [created by the symphony app](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/res/post-start.sh#L5) when its container [gets started](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/chart/templates/deployment.yaml#L81). Some [sample data](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/res/data.sql) will also be inserted at that moment, before the app is ready to serve requests.


Symfony environments variables
===

Our demo Symfony app requires 2 standard env variables to be defined, and because they are sensitive, let's add them also in the app's k8s secret that will be created by GitLab during the deployment:

| Variable name | Value    | Description | 
| ------------- | -------- |  -------- | 
| K8S_SECRET_APP_ENV| dev | Runs the app in debug mode for demo purpose.
| K8S_SECRET_APP_SECRET| 5e9f0b558b7da... | The [Symfony secret](https://symfony.com/doc/current/reference/configuration/framework.html#secret) to use.

> The `SYMFONY_DATABASE_URL` env variable is [dynamically built](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/chart/templates/deployment.yaml#L71) by the app helm chart from the MySQL values defined above.

**You should have something looking like:**

![](docs/img/gitlab-env.png)

Deployment
===

It is now time to deploy our demo project.

Your own project's repository is actually empty, so add all the files available in this project to it, `commit` and push to `master`.

The AutoDevOps magic begins now, so just watch the CI/CD pipeline gets executed and deploy your app, with its database and its let's encrypt certificate.

![](docs/img/pipeline.png)

![](docs/img/staging-job.png)

![](docs/img/deployed.png)


About the magic bits
===

*What just happened?*

Let's encrypt
====
Your app is automagically **https-enabled**. 

By installing `cert-manager` as a GitLab-managed app, any new deployment using an [`ingress`](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/chart/templates/ingress.yaml#L18) with the k8s annotation `kubernetes.io/tls-acme: "true"` and a [`tls.hosts`](https://gitlab.com/ipernet/gitlab-symfony-autodevops/blob/master/chart/values.yaml#L39) section [gets a new certificate](https://docs.cert-manager.io/en/latest/tasks/issuing-certificates/ingress-shim.html).

**To know more:**

* http://docs.cert-manager.io/en/latest/reference/issuers.html?highlight=email
* https://docs.cert-manager.io/en/latest/tasks/issuing-certificates/ingress-shim.html

And to see what GitLab has setup for you on your cluster:

```
$ k3s kubectl describe pod certmanager -n gitlab-managed-apps | grep 'default-issuer'
      --default-issuer-name=letsencrypt-prod
      --default-issuer-kind=ClusterIssuer

$ k3s kubectl describe ClusterIssuer letsencrypt-prod -n gitlab-managed-apps
Name:         letsencrypt-prod
Spec:
  Acme:
    Email:  <GitLab Issuer Email>
```

Something went wrong?
====

Use k3s to check the status of your app deployment, but make sure to allocate a few minutes for the initial pipeline to execute:

```
$ k3s kubectl get pod -n gitlab-symfony-autodevops

NAME                             READY   STATUS              RESTARTS   AGE
cm-acme-http-solver-rv6pv        0/1     Pending             0          0s
staging-566f47dc8b-ntc2d         0/2     ContainerCreating   0          9s
staging-mysql-78dd8dc474-ftllm   0/1     PodInitializing     0          9s

$ k3s kubectl describe pod <name> -n gitlab-symfony-autodevops
```