GitLab Kubernetes deployment does not work on isolated network
Summary
Had a customer try to set-up a project to use Kubernetes on an isolated network (not connected to internet in any way). They ran into several issues getting this to work and were ultimately able to in a round-about way.
Steps to reproduce
They were able to get GitLab deployed in an isolated environment to prove it's achievable in short order with the official GitLab Helm chart, but required modifications to many values.yml files, as there's no central value in the global namespace to control the registries that images get pulled from.
This isn't a unique issue to GitLab's chart, as most of the official Helm charts don't make it painless either to change the registry locations with one central knob (as images get pulled in from multiple registries usually). They use a large number of Helm charts to alleviate the pain of running with alternate registry locations. The key thing they do is control it through the global.imageRegistry key, which makes it less painful when deploying to an isolated environment, as there's only one value to set. They then modify the image: keys in the chart templates to use this value, so that we don't have to redefine the repository location and repository tag for all the various images used in the charts. They haven't really tried to upstream the changes, as it's different maintainers for all the charts and not all are responsive to pull requests. Current plan is to continue to vendor the repos with minimal modifications, as pulling in upstream modifications for new feature we want isn't too bad. Subject to change in the future with additional engineering resources, but not all upstream maintainers are amicable to pull requests.
For the GitLab chart, they'll likely stick to the same approach for now due to their engineering availability and their deliverable schedule for our customers versus trying to upstream a bunch of changes. But long term, it'd be nice to have it both well documented and possibly with less values to set to run isolated.
From customer directly: I pushed a repo that contains the contents of the chart I pulled and the commits showing the steps to make it work in an isolated environment: https://gitlab.com/knack-aron/gitlab. I shared this repo with this GitLab group so everyone should be able to access it. Note this is not a fork of the official GitLab Helm chart, but just there to show the diff for a rough idea of the changes. It also includes a crude shell script I used to mirror the images into a local registry.
Some screenshots and output below to show things running isolated. The environment depicted below has no outbound Internet access.
[aparsons@lap-aparsons gitlab (master *%)]$ helm status gitlab
LAST DEPLOYED: Mon Jun 3 16:54:51 2019
NAMESPACE: gitlab
STATUS: DEPLOYED
RESOURCES:
==> v1beta2/StatefulSet
NAME DESIRED CURRENT AGE
gitlab-gitaly 1 1 6m22s
==> v1/Job
NAME COMPLETIONS DURATION AGE
gitlab-issuer.1 1/1 17s 6m22s
gitlab-migrations.1 1/1 3m7s 6m22s
gitlab-minio-create-buckets.1 1/1 25s 6m22s
==> v1/ConfigMap
NAME DATA AGE
gitlab-certmanager-issuer-certmanager 2 6m23s
gitlab-gitaly 3 6m23s
gitlab-gitlab-shell 2 6m23s
gitlab-nginx-ingress-tcp 1 6m23s
gitlab-migrations 4 6m23s
gitlab-sidekiq-all-in-1 1 6m23s
gitlab-sidekiq 6 6m23s
gitlab-task-runner 4 6m23s
gitlab-workhorse-config 3 6m23s
gitlab-unicorn 7 6m23s
gitlab-unicorn-tests 1 6m23s
gitlab-minio-config-cm 3 6m23s
gitlab-nginx-ingress-controller 8 6m23s
gitlab-nginx-ingress-custom-headers 1 6m23s
gitlab-postgresql 0 6m23s
gitlab-redis 2 6m23s
gitlab-registry 2 6m23s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 6m23s
gitlab-gitlab-shell ClusterIP 10.97.240.248 <none> 22/TCP 6m22s
gitlab-unicorn ClusterIP 10.101.252.133 <none> 8080/TCP,8181/TCP 6m22s
gitlab-minio-svc ClusterIP 10.105.12.125 <none> 9000/TCP 6m22s
gitlab-nginx-ingress-controller-metrics ClusterIP 10.105.188.17 <none> 9913/TCP 6m22s
gitlab-nginx-ingress-controller LoadBalancer 10.105.46.249 192.168.133.6 80:32358/TCP,443:31974/TCP,22:32474/TCP 6m22s
gitlab-nginx-ingress-controller-stats ClusterIP 10.104.124.149 <none> 18080/TCP 6m22s
gitlab-nginx-ingress-default-backend ClusterIP 10.99.220.143 <none> 80/TCP 6m22s
gitlab-postgresql ClusterIP 10.96.250.77 <none> 5432/TCP 6m22s
gitlab-redis ClusterIP 10.103.157.189 <none> 6379/TCP,9121/TCP 6m22s
gitlab-registry ClusterIP 10.108.133.156 <none> 5000/TCP 6m22s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
gitlab-postgresql 1 1 1 1 6m22s
==> v2beta1/HorizontalPodAutoscaler
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gitlab-gitlab-shell Deployment/gitlab-gitlab-shell <unknown>/100m 2 10 2 6m22s
gitlab-sidekiq-all-in-1 Deployment/gitlab-sidekiq-all-in-1 <unknown>/350m 1 10 1 6m22s
gitlab-unicorn Deployment/gitlab-unicorn <unknown>/1 2 10 2 6m22s
gitlab-registry Deployment/gitlab-registry <unknown>/75% 2 10 2 6m22s
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
gitlab-gitaly N/A 1 1 6m23s
gitlab-gitlab-shell N/A 1 1 6m23s
gitlab-sidekiq N/A 1 1 6m23s
gitlab-unicorn N/A 1 1 6m23s
gitlab-minio-v1 N/A 1 1 6m23s
gitlab-nginx-ingress-controller 2 N/A 1 6m23s
gitlab-nginx-ingress-default-backend 1 N/A 1 6m23s
gitlab-redis-v1 N/A 1 1 6m23s
gitlab-registry-v1 N/A 1 1 6m23s
==> v1/ServiceAccount
NAME SECRETS AGE
gitlab-certmanager-issuer 1 6m23s
gitlab-nginx-ingress 1 6m23s
==> v1/Role
NAME AGE
gitlab-certmanager-issuer 6m23s
gitlab-nginx-ingress 6m23s
==> v1/RoleBinding
NAME AGE
gitlab-certmanager-issuer 6m23s
gitlab-nginx-ingress 6m23s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
gitlab-gitlab-shell-658d69fcd8-ftxhm 1/1 Running 0 6m7s
gitlab-gitlab-shell-658d69fcd8-xtz95 1/1 Running 0 6m22s
gitlab-sidekiq-all-in-1-77fd457679-g9j4n 1/1 Running 0 6m22s
gitlab-task-runner-58dcf69f84-2xkvm 1/1 Running 0 6m22s
gitlab-unicorn-6b7489f89c-st86r 2/2 Running 0 6m22s
gitlab-unicorn-6b7489f89c-wd4dq 2/2 Running 0 6m7s
gitlab-minio-7d995c78fc-mlvlt 1/1 Running 0 6m22s
gitlab-nginx-ingress-controller-55fd7df8d-7phwm 1/1 Running 0 6m22s
gitlab-nginx-ingress-controller-55fd7df8d-g2vtp 1/1 Running 0 6m22s
gitlab-nginx-ingress-controller-55fd7df8d-mcww2 1/1 Running 0 6m22s
gitlab-nginx-ingress-default-backend-69645b45c8-jtqf4 1/1 Running 0 6m22s
gitlab-nginx-ingress-default-backend-69645b45c8-t9fdx 1/1 Running 0 6m22s
gitlab-postgresql-79ccf979c5-h294d 2/2 Running 0 6m22s
gitlab-redis-5764df7966-hqh99 2/2 Running 0 6m22s
gitlab-registry-5f48bb6647-tdr9f 1/1 Running 0 6m6s
gitlab-registry-5f48bb6647-tfmtw 1/1 Running 0 6m21s
gitlab-gitaly-0 1/1 Running 0 6m22s
gitlab-issuer.1-prf7p 0/1 Completed 0 6m22s
gitlab-migrations.1-qk46q 0/1 Completed 0 6m22s
gitlab-minio-create-buckets.1-rb78c 0/1 Completed 0 6m22s
==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
gitlab-nginx-ingress false NET_BIND_SERVICE RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,secret
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gitlab-minio Bound pvc-e5267bd0-8641-11e9-be25-566fac170041 10Gi RWO nfs-client 6m23s
gitlab-postgresql Bound pvc-e527c4ec-8641-11e9-be25-566fac170041 8Gi RWO nfs-client 6m23s
gitlab-redis Bound pvc-e528e7ee-8641-11e9-be25-566fac170041 5Gi RWO nfs-client 6m23s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
gitlab-gitlab-shell 2 2 2 2 6m22s
gitlab-sidekiq-all-in-1 1 1 1 1 6m22s
gitlab-task-runner 1 1 1 1 6m22s
gitlab-unicorn 2 2 2 2 6m22s
gitlab-minio 1 1 1 1 6m22s
gitlab-nginx-ingress-controller 3 3 3 3 6m22s
gitlab-nginx-ingress-default-backend 2 2 2 2 6m22s
gitlab-redis 1 1 1 1 6m22s
gitlab-registry 2 2 2 2 6m22s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
gitlab-unicorn gitlab.jt.dev.knack.works 192.168.133.6 80, 443 6m22s
gitlab-minio minio.jt.dev.knack.works 192.168.133.6 80, 443 6m22s
gitlab-registry registry.jt.dev.knack.works 192.168.133.6 80, 443 6m22s
NOTES:[aparsons@lap-aparsons gitlab (master *%)]$ kubectl -n gitlab describe pod | grep Image:
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitaly:v1.42.2
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-shell:v9.1.0
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-shell:v9.1.0
Image: docker.knack.works/gitlab/kubectl:1f8690f03f7aeef27e727396927ab3cc96ac89e7
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-rails-ee:v11.11.1
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/minio:RELEASE.2017-12-28T01-21-00Z
Image: docker.knack.works/gitlab/mc:RELEASE.2018-07-13T00-53-22Z
Image: docker.knack.works/gitlab/nginx-ingress-controller:0.20.0
Image: docker.knack.works/gitlab/nginx-ingress-controller:0.20.0
Image: docker.knack.works/gitlab/nginx-ingress-controller:0.20.0
Image: docker.knack.works/gitlab/defaultbackend:1.4
Image: docker.knack.works/gitlab/defaultbackend:1.4
Image: docker.knack.works/gitlab/postgres:9.6.8
Image: docker.knack.works/gitlab/postgres_exporter:v0.1.1
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/redis:3.2.12
Image: docker.knack.works/gitlab/redis_exporter:latest
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/registry:2.7.1
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/registry:2.7.1
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-sidekiq-ee:v11.11.1
Image: docker.knack.works/gitlab/gitlab-sidekiq-ee:v11.11.1
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-task-runner-ee:v11.11.1
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-unicorn-ee:v11.11.1
Image: docker.knack.works/gitlab/gitlab-unicorn-ee:v11.11.1
Image: docker.knack.works/gitlab/gitlab-workhorse-ee:v11.11.1
Image: docker.knack.works/gitlab/alpine-certificates:20171114-r3
Image: docker.knack.works/gitlab/busybox:latest
Image: docker.knack.works/gitlab/gitlab-unicorn-ee:v11.11.1
Image: docker.knack.works/gitlab/gitlab-unicorn-ee:v11.11.1
Image: docker.knack.works/gitlab/gitlab-workhorse-ee:v11.11.1Example Project
not possible as it only happens on an isolated network
What is the current bug behavior?
(What actually happens)
What is the expected correct behavior?
You should be able to point GitLab to an existing Kubernetes cluster on an isolated network and it should work.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Unfortunately, we cannot copy anything off of the isolated network to put here
Results of GitLab application Check
Unfortunately, we cannot copy anything off of the isolated network to put here
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)