GitLab AutoDevOps Ingress on bare metal (with Rancher)

Summary

Hi there,

I use (bare metal) Kubernetes with Rancher 2.1.7 and try to implement the AutoDevOps feature. (I must not have to use Auto-DevOps if any other solution would work like custom .gitlab-ci.yml)

Steps to reproduce

Create a Rancher server on a bare metal machine

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher -d --acme-domain=mysub.domain.com
Spin up 3 nodes (on a bare metal provider like Hetzner, Linode,...)
a.) (see "Configuration used"-chapter)

Connect Gitlab to you K8s cluster

For an easy and fast connection check out my gist (https://gist.github.com/exocode/e5058c06faf02ba94f57b067b9b7f5a0)
After creating your cluster in Gitlab , install Tiller, Ingress, Cert-Manager and GitLab Runner by clicking the "install" buttons.

Push your code (or use a Gitlab template), wait for deployment finished.

Configuration used

a.) I am not sure if I have to do the following step when I deploy nginx-ingress through Gitlabs "one-click"-installation.

Normally, (before I used Gitlab at all) I created a deployment with the nginx-Ingress helm chart by using the following command.

helm install –name ingress –namespace ingress –set...

Ingress via hostNetwork

rbac.create=true
controller.kind=DaemonSet #important
controller.service.type=ClusterIP
controller.hostNetwork=true #important
controller.nodeSelector.role=edge

Or Ingress via externalIPs

rbac.create=true
controller.kind=DaemonSet #important
controller.service.type=ClusterIP
controller.service.externalIPs[0]=MY_IP_ADDRESS #important
controller.nodeSelector.role=edge

I use a custom Helm chart (https://github.com/exocode/helm-charts/tree/master/hetzner-failover-ip) which points my dynamic (Hetzner) IP to the node with the label "role=edge" and assigns the network settings accordingly on that node. So I am able to call that node through my domain.

Current behavior

Everything looks fine (no errors so far), except the ingress-nginx-ingress-controller-Deployment and -Service keeps in "Pending" state. Also, the LetsEncrypt cert is valid and I can request my url which ends in a 503 error. (503 Service Temporarily Unavailable nginx/1.13.8)

Expected behavior

ingress-nginx-ingress-controller-Deployment and -Service should be "Active" and app workload should be available

Versions

Chart: (tagged version | branch | hash git rev-parse HEAD)
Platform:
- Cloud: (Hetzner cloud) https://www.hetzner.com/cloud
- Self-hosted: (Rancher 2.1.7)
Kubernetes: (kubectl version)
- Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:36:43Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
- Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Helm: (helm version)
- Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
- Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}

Relevant logs

kubectl get all -n ingress-nginx    1 ↵
NAME                                        READY   STATUS    RESTARTS   AGE
pod/default-http-backend-7f8fbb85db-lwkw2   1/1     Running   0          13h
pod/nginx-ingress-controller-6kc86          1/1     Running   0          13h
pod/nginx-ingress-controller-lhrrp          1/1     Running   0          13h
pod/nginx-ingress-controller-sjt4g          1/1     Running   0          7h42m

NAME                           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/default-http-backend   ClusterIP   10.43.92.161   <none>        80/TCP    13h

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress-controller   3         3         3       3            3           <none>          13h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/default-http-backend   1/1     1            1           13h

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/default-http-backend-7f8fbb85db   1         1         1       13h

 janjezek   …/github/nodejs-debugging     3824b219  master   1  kubectl get all -n gitlab-managed-apps                                                                                                                        ✔
NAME                                                         READY   STATUS    RESTARTS   AGE
pod/certmanager-cert-manager-6df979599b-97bmc                1/1     Running   0          7h27m
pod/ingress-nginx-ingress-controller-7cf6944677-fxtgn        1/1     Running   0          7h41m
pod/ingress-nginx-ingress-default-backend-7f7bf55777-fqjwj   1/1     Running   0          7h41m
pod/runner-gitlab-runner-669ffbb846-mv9bv                    1/1     Running   0          7h
pod/tiller-deploy-7cbfdc5df7-7hxfq                           1/1     Running   0          7h43m

NAME                                             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-ingress-controller         LoadBalancer   10.43.39.220   <pending>     80:32180/TCP,443:30277/TCP   7h41m
service/ingress-nginx-ingress-controller-stats   ClusterIP      10.43.45.143   <none>        18080/TCP                    7h41m
service/ingress-nginx-ingress-default-backend    ClusterIP      10.43.37.28    <none>        80/TCP                       7h41m
service/tiller-deploy                            ClusterIP      10.43.179.38   <none>        44134/TCP                    7h43m

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/certmanager-cert-manager                1/1     1            1           7h27m
deployment.apps/ingress-nginx-ingress-controller        1/1     1            1           7h41m
deployment.apps/ingress-nginx-ingress-default-backend   1/1     1            1           7h41m
deployment.apps/runner-gitlab-runner                    1/1     1            1           7h
deployment.apps/tiller-deploy                           1/1     1            1           7h43m

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/certmanager-cert-manager-6df979599b                1         1         1       7h27m
replicaset.apps/ingress-nginx-ingress-controller-7cf6944677        1         1         1       7h41m
replicaset.apps/ingress-nginx-ingress-default-backend-7f7bf55777   1         1         1       7h41m
replicaset.apps/runner-gitlab-runner-669ffbb846                    1         1         1       7h
replicaset.apps/tiller-deploy-7cbfdc5df7                           1         1         1       7h43m

Is the ingress-controller installed by Gitlab a replacement to my "old" approach or are they running side-by-side?
This "one-click"-installation looks a little bit like magic to me. I don't know how to modify the Gitlab installed nginx-ingress to controller.kind=DaemonSet or controller.hostNetwork=true (like I did regularly with helm install).
- Are these settings made somewhere in the configMaps?
- Or do I have to do this by helm?
Or do I have to customize my own .gitlab-ci.yml file (maybe from the Auto-DevOps example)?

Sorry for that maybe awkwardness questions.

Thank you very much in advance

Edited Sep 02, 2020 by 🤖 GitLab Bot 🤖