Pods fail because of secrets even with shared-secrets
Summary
When deploying Gitlab as described in the deployment guide, several pods fail because of missing secrets
Steps to reproduce
- Setup a Kubernetes cluster
- Mark one storageclass as default
- Deploy Gitlab using the command given in the deployment guide.
helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--namespace gitlab \
--set global.hosts.domain=gitlab.nicklehmann.me \
--set global.hosts.externalIP=sever-ip \
--set certmanager-issuer.email=nicklehmann@protonmail.comConfiguration used
See above.
Current behavior
The following pods get stuck in the initialization phase:
- Migrations
- Sidekiq
- Task runner
- Unicorn (both)
The descriptions of the pods returned by kubectl describe are listed at the end.
Expected behavior
See all pods up and running and be able to access Gitlab via the web interface.
Versions
- Chart: gitlab-2.0.2 (from
helm list) - Platform: Three VPS hosted at Contabo
- Master: 8 vCPU Xeon CPU E5-2640 @ 2.2Ghz, 30Gb RAM
- 2xWorker: 4 vCPU Xeon CPU E5-2640 @ 2.2Ghz, 8Gb RAM
- Kubernetes: (
kubectl version)
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"archive", BuildDate:"2019-06-08T16:22:01Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}- Helm: (
helm version)
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}Relevant logs
Migrations
> kubectl -n gitlab describe pod gitlab-migrations.1-scz55 | tail
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned gitlab/gitlab-migrations.1-scz55 to server2
Warning FailedMount 69s (x13 over 11m) kubelet, server2 MountVolume.SetUp failed for volume "init-migrations-secrets" : secret "gitlab-rails-secret" not found
Warning FailedMount 22s (x5 over 9m24s) kubelet, server2 Unable to mount volumes for pod "gitlab-migrations.1-scz55_gitlab(fc25f11f-947e-489d-bf70-b938d5f8cc05)": timeout expired waiting for volumes to attach or mount for pod "gitlab"/"gitlab-migrations.1-scz55". list of unmounted volumes=[init-migrations-secrets]. list of unattached volumes=[migrations-config init-migrations-secrets migrations-secrets etc-ssl-certs default-token-92jwz]Sidekiq
> kubectl -n gitlab describe pod gitlab-sidekiq-all-in-1-59f9bb658f-8rv6n | tail
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned gitlab/gitlab-sidekiq-all-in-1-59f9bb658f-8rv6n to server1
Warning FailedMount 2m12s (x5 over 11m) kubelet, server1 Unable to mount volumes for pod "gitlab-sidekiq-all-in-1-59f9bb658f-8rv6n_gitlab(30603e5c-2a18-42a1-8a29-10244ae73b3a)": timeout expired waiting for volumes to attach or mount for pod "gitlab"/"gitlab-sidekiq-all-in-1-59f9bb658f-8rv6n". list of unmounted volumes=[init-sidekiq-secrets]. list of unattached volumes=[sidekiq-metrics sidekiq-config init-sidekiq-secrets sidekiq-secrets etc-ssl-certs default-token-92jwz]
Warning FailedMount 62s (x14 over 13m) kubelet, server1 MountVolume.SetUp failed for volume "init-sidekiq-secrets" : secret "gitlab-rails-secret" not foundTask runner
> kubectl -n gitlab describe pod gitlab-task-runner-778f487fcb-ffccg | tail
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned gitlab/gitlab-task-runner-778f487fcb-ffccg to server1
Warning FailedMount 82s (x6 over 12m) kubelet, server1 Unable to mount volumes for pod "gitlab-task-runner-778f487fcb-ffccg_gitlab(8e7511fa-ab44-46dc-9c19-91efd7eccd1c)": timeout expired waiting for volumes to attach or mount for pod "gitlab"/"gitlab-task-runner-778f487fcb-ffccg". list of unmounted volumes=[init-task-runner-secrets]. list of unattached volumes=[task-runner-config task-runner-tmp init-task-runner-secrets task-runner-secrets etc-ssl-certs default-token-92jwz]
Warning FailedMount 25s (x15 over 14m) kubelet, server1 MountVolume.SetUp failed for volume "init-task-runner-secrets" : secret "gitlab-rails-secret" not foundUnicorn
kubectl -n gitlab describe pod gitlab-unicorn-69f98988cb-pz84f | tailQoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300sEvents:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned gitlab/gitlab-unicorn-69f98988cb-pz84f to server3
Warning FailedMount 111s (x6 over 13m) kubelet, server3 Unable to mount volumes for pod "gitlab-unicorn-69f98988cb-pz84f_gitlab(7ba2b581-7c0f-4259-bf61-1e97b9b61bb4)": timeout expired waiting for volumes to attach or mount for pod "gitlab"/"gitlab-unicorn-69f98988cb-pz84f". list of unmounted volumes=[init-unicorn-secrets]. list of unattached volumes=[unicorn-metrics unicorn-config workhorse-config init-unicorn-secrets unicorn-secrets workhorse-secrets shared-upload-directory shared-artifact-directory etc-ssl-certs default-token-92jwz]
Warning FailedMount 56s (x15 over 15m) kubelet, server3 MountVolume.SetUp failed for volume "init-unicorn-secrets" : secret "gitlab-rails-secret" not foundPod overview
> kubectl -n gitlab get pods
NAME READY STATUS RESTARTS AGE
gitlab-certmanager-57bc6fb4fd-bm89v 1/1 Running 0 17m
gitlab-gitaly-0 1/1 Running 0 17m
gitlab-gitlab-monitor-7dccc8485f-wjj2n 1/1 Running 0 17m
gitlab-gitlab-runner-d5b699c97-9v2xl 0/1 Running 5 17m
gitlab-gitlab-shell-688994bdfb-75q9z 1/1 Running 0 17m
gitlab-gitlab-shell-688994bdfb-nbztd 1/1 Running 0 16m
gitlab-migrations.1-scz55 0/1 Init:0/2 0 17m
gitlab-minio-75567fcbb6-kwnmw 1/1 Running 0 17m
gitlab-minio-create-buckets.1-k6cjk 0/1 Completed 0 17mgitlab-nginx-ingress-controller-698fbc4c64-7grz4 1/1 Running 0 17m
gitlab-nginx-ingress-controller-698fbc4c64-g6rld 1/1 Running 0 17mgitlab-nginx-ingress-controller-698fbc4c64-q8hvr 1/1 Running 0 17mgitlab-nginx-ingress-default-backend-6cd54c5f86-dwqsv 1/1 Running 0 17mgitlab-nginx-ingress-default-backend-6cd54c5f86-ggnj9 1/1 Running 0 17m
gitlab-postgresql-66d8d9574b-fj927 2/2 Running 0 17mgitlab-prometheus-server-95c656c6d-h948t 2/2 Running 0 17m
gitlab-redis-566f8849df-lzl2z 2/2 Running 0 17m
gitlab-registry-bcd7f49b4-8lqm4 1/1 Running 0 17m
gitlab-registry-bcd7f49b4-gh65k 1/1 Running 0 17m
gitlab-sidekiq-all-in-1-59f9bb658f-8rv6n 0/1 Init:0/3 0 17m
gitlab-task-runner-778f487fcb-ffccg 0/1 Init:0/2 0 17m
gitlab-unicorn-69f98988cb-kpvmn 0/2 Init:0/3 0 17m
gitlab-unicorn-69f98988cb-pz84f 0/2 Init:0/3 0 16mSecrets
> kubectl -n gitlab get secrets
NAME TYPE DATA AGE
default-token-92jwz kubernetes.io/service-account-token 3 16m
gitlab-certmanager-issuer-token-htfsp kubernetes.io/service-account-token 3 16m
gitlab-certmanager-token-rfgzh kubernetes.io/service-account-token 3 16mgitlab-gitaly-secret Opaque 1 16m
gitlab-gitlab-initial-root-password Opaque 1 16mgitlab-gitlab-runner-secret Opaque 2 16mgitlab-gitlab-runner-token-7h8zr kubernetes.io/service-account-token 3 16mgitlab-gitlab-shell-host-keys Opaque 8 16m
gitlab-gitlab-shell-secret Opaque 1 16mgitlab-gitlab-workhorse-secret Opaque 1 16mgitlab-minio-secret Opaque 2 16mgitlab-nginx-ingress-token-wbqml kubernetes.io/service-account-token 3 16mgitlab-postgresql-password Opaque 1 16m
gitlab-prometheus-alertmanager-token-5zxl5 kubernetes.io/service-account-token 3 16mgitlab-prometheus-kube-state-metrics-token-z7j5l kubernetes.io/service-account-token 3 16m
gitlab-prometheus-node-exporter-token-hcbj5 kubernetes.io/service-account-token 3 16m
gitlab-prometheus-server-token-j42cl kubernetes.io/service-account-token 3 16m
gitlab-redis-secret Opaque 1 16m
gitlab-registry-httpsecret Opaque 1 16m
gitlab-registry-secret Opaque 2 16mEdited by Nick Lehmann