Total CPU needed for each review app deployment is more than a single node
Total cpu requests in scripts/review_apps/base-config.yaml is 2690m, more than the current cluster node 2vCPU.
This means every review app deployment requires a new instance to spin up, which takes time and increases the likelihood of helm failure.
This could also explain now we have helm failed deployments, but when inspecting the pods of the failed deployment, all are running.
$ helm list --tiller-namespace=review-apps-ee --failed
NAME REVISION UPDATED STATUS CHART NAMESPACE
review-introduce-epuki8 1 Fri Oct 25 16:01:14 2019 FAILED gitlab-2.3.7 review-apps-ee
$ kubectl get pods -l release=review-introduce-epuki8
NAME READY STATUS RESTARTS AGE
review-introduce-epuki8-gitaly-0 1/1 Running 0 80m
review-introduce-epuki8-gitlab-shell-76bfb49444-4hbpm 1/1 Running 0 80m
review-introduce-epuki8-gitlab-shell-76bfb49444-c6ld9 1/1 Running 0 78m
review-introduce-epuki8-gitlab-shell-76bfb49444-tc2wh 1/1 Running 0 80m
review-introduce-epuki8-migrations.1-gm49p 0/1 Completed 0 80m
review-introduce-epuki8-minio-5dcc4c68c4-n7v5w 1/1 Running 0 80m
review-introduce-epuki8-minio-create-buckets.1-5bxlh 0/1 Completed 0 80m
review-introduce-epuki8-nginx-ingress-controller-676b564b42j4wf 1/1 Running 1 80m
review-introduce-epuki8-nginx-ingress-controller-676b564b4pbhvx 1/1 Running 2 80m
review-introduce-epuki8-nginx-ingress-default-backend-6f579gl55 1/1 Running 0 80m
review-introduce-epuki8-postgresql-78b88f6b4f-7mgz5 1/1 Running 0 80m
review-introduce-epuki8-redis-7cd55456db-gvmkz 2/2 Running 0 80m
review-introduce-epuki8-registry-8665b86757-nv494 1/1 Running 0 80m
review-introduce-epuki8-registry-8665b86757-q8wfk 1/1 Running 0 80m
review-introduce-epuki8-sidekiq-all-in-1-7c99cf74b5-sgsbh 1/1 Running 0 80m
review-introduce-epuki8-task-runner-678746949d-xzv7t 1/1 Running 0 80m
review-introduce-epuki8-unicorn-598f88d99-7jgls 2/2 Running 0 80m
review-introduce-epuki8-unicorn-598f88d99-fm656 2/2 Running 0 80m