review apps failed deployment - missing secret
Some review apps deployment failed due to missing secrets:
- review-33494-dele-jqfp4w
Traces
$ kubectl get pods,secret -n review-apps-ee | grep review-33494-dele-jqfp4w
pod/review-33494-dele-jqfp4w-gitaly-0 0/1 Init:0/2 0 45m
pod/review-33494-dele-jqfp4w-gitlab-runner-857fc4c957-rbt48 0/1 Init:0/1 0 45m
pod/review-33494-dele-jqfp4w-gitlab-shell-66b45d54cb-25wn6 0/1 Init:0/2 0 45m
pod/review-33494-dele-jqfp4w-gitlab-shell-66b45d54cb-5cvbl 0/1 Init:0/2 0 45m
pod/review-33494-dele-jqfp4w-migrations.1-vcxdw 0/1 Init:0/2 0 45m
pod/review-33494-dele-jqfp4w-minio-b96b88b87-tz8q8 0/1 Init:0/1 0 45m
pod/review-33494-dele-jqfp4w-nginx-ingress-controller-658f48cbg9k26 1/1 Running 0 45m
pod/review-33494-dele-jqfp4w-nginx-ingress-controller-658f48cbzwvsz 1/1 Running 0 45m
pod/review-33494-dele-jqfp4w-nginx-ingress-default-backend-b65rwjll 1/1 Running 0 45m
pod/review-33494-dele-jqfp4w-postgresql-8559dcffc4-rgvjn 0/1 ContainerCreating 0 45m
pod/review-33494-dele-jqfp4w-redis-fdd687878-69c47 0/2 Init:0/1 0 45m
pod/review-33494-dele-jqfp4w-registry-6c87bfd5d-fmfjx 0/1 Init:0/2 0 45m
pod/review-33494-dele-jqfp4w-sidekiq-all-in-1-7c557f69f9-kh85w 0/1 Init:0/3 0 45m
pod/review-33494-dele-jqfp4w-task-runner-6676c6d978-jkntj 0/1 Init:0/2 0 45m
pod/review-33494-dele-jqfp4w-unicorn-556478f856-vdn99 0/2 Init:0/3 0 45m
pod/review-33494-dele-jqfp4w-unicorn-556478f856-xc8d7 0/2 Init:0/3 0 45m
secret/review-33494-dele-jqfp4w-gitlab-initial-root-password Opaque 1 46m
secret/review-33494-dele-jqfp4w-gitlab-license Opaque 1 46m
secret/review-33494-dele-jqfp4w-gitlab-runner-token-l6td5 kubernetes.io/service-account-token 3 45m
secret/review-33494-dele-jqfp4w-nginx-ingress-token-b82gh kubernetes.io/service-account-token 3 45m
Inspecting one of the pods reveals missing secrets:
$ kubectl describe pod review-33494-dele-jqfp4w-gitaly-0
...
Warning FailedMount 6m44s (x18 over 45m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-kgqk Unable to mount volumes for pod "review-33494-dele-jqfp4w-gitaly-0_review-apps-ee(814a66f3-0f3a-11ea-9f4c-42010af0010d)": timeout expired waiting for volumes to attach or mount for pod "review-apps-ee"/"review-33494-dele-jqfp4w-gitaly-0". list of unmounted volumes=[init-gitaly-secrets]. list of unattached volumes=[repo-data gitaly-config gitaly-secrets init-gitaly-secrets etc-ssl-certs default-token-k9j8n]
Warning FailedMount 10s (x31 over 47m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-kgqk MountVolume.SetUp failed for volume "init-gitaly-secrets" : [secrets "review-33494-dele-jqfp4w-gitaly-secret" not found, secrets "review-33494-dele-jqfp4w-gitlab-shell-secret" not found, secrets "review-33494-dele-jqfp4w-redis-secret" not found]
Some time starting about This may not be true, 2019-11-21T09:27:28Z
, secrets were not created during deployment.initial-root-password
and license
secrets are not cleaned up because they are not labeled with the release. https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/review_apps/review-apps.sh#L200
Edited by Albert Salim