review apps failed deployment - missing secret

Some review apps deployment failed due to missing secrets:

• review-33494-dele-jqfp4w

---

Traces

$ kubectl get pods,secret -n review-apps-ee | grep review-33494-dele-jqfp4w                                                                            
pod/review-33494-dele-jqfp4w-gitaly-0                                 0/1     Init:0/2            0          45m
pod/review-33494-dele-jqfp4w-gitlab-runner-857fc4c957-rbt48           0/1     Init:0/1            0          45m
pod/review-33494-dele-jqfp4w-gitlab-shell-66b45d54cb-25wn6            0/1     Init:0/2            0          45m
pod/review-33494-dele-jqfp4w-gitlab-shell-66b45d54cb-5cvbl            0/1     Init:0/2            0          45m
pod/review-33494-dele-jqfp4w-migrations.1-vcxdw                       0/1     Init:0/2            0          45m
pod/review-33494-dele-jqfp4w-minio-b96b88b87-tz8q8                    0/1     Init:0/1            0          45m
pod/review-33494-dele-jqfp4w-nginx-ingress-controller-658f48cbg9k26   1/1     Running             0          45m
pod/review-33494-dele-jqfp4w-nginx-ingress-controller-658f48cbzwvsz   1/1     Running             0          45m
pod/review-33494-dele-jqfp4w-nginx-ingress-default-backend-b65rwjll   1/1     Running             0          45m
pod/review-33494-dele-jqfp4w-postgresql-8559dcffc4-rgvjn              0/1     ContainerCreating   0          45m
pod/review-33494-dele-jqfp4w-redis-fdd687878-69c47                    0/2     Init:0/1            0          45m
pod/review-33494-dele-jqfp4w-registry-6c87bfd5d-fmfjx                 0/1     Init:0/2            0          45m
pod/review-33494-dele-jqfp4w-sidekiq-all-in-1-7c557f69f9-kh85w        0/1     Init:0/3            0          45m
pod/review-33494-dele-jqfp4w-task-runner-6676c6d978-jkntj             0/1     Init:0/2            0          45m
pod/review-33494-dele-jqfp4w-unicorn-556478f856-vdn99                 0/2     Init:0/3            0          45m
pod/review-33494-dele-jqfp4w-unicorn-556478f856-xc8d7                 0/2     Init:0/3            0          45m
secret/review-33494-dele-jqfp4w-gitlab-initial-root-password   Opaque                                1      46m
secret/review-33494-dele-jqfp4w-gitlab-license                 Opaque                                1      46m
secret/review-33494-dele-jqfp4w-gitlab-runner-token-l6td5      kubernetes.io/service-account-token   3      45m
secret/review-33494-dele-jqfp4w-nginx-ingress-token-b82gh      kubernetes.io/service-account-token   3      45m

Inspecting one of the pods reveals missing secrets:

$ kubectl describe pod review-33494-dele-jqfp4w-gitaly-0
...
 Warning  FailedMount             6m44s (x18 over 45m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-kgqk  Unable to mount volumes for pod "review-33494-dele-jqfp4w-gitaly-0_review-apps-ee(814a66f3-0f3a-11ea-9f4c-42010af0010d)": timeout expired waiting for volumes to attach or mount for pod "review-apps-ee"/"review-33494-dele-jqfp4w-gitaly-0". list of unmounted volumes=[init-gitaly-secrets]. list of unattached volumes=[repo-data gitaly-config gitaly-secrets init-gitaly-secrets etc-ssl-certs default-token-k9j8n]
  Warning  FailedMount             10s (x31 over 47m)    kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-kgqk  MountVolume.SetUp failed for volume "init-gitaly-secrets" : [secrets "review-33494-dele-jqfp4w-gitaly-secret" not found, secrets "review-33494-dele-jqfp4w-gitlab-shell-secret" not found, secrets "review-33494-dele-jqfp4w-redis-secret" not found]

Some time starting about 2019-11-21T09:27:28Z, secrets were not created during deployment. This may not be true, initial-root-password and license secrets are not cleaned up because they are not labeled with the release. https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/review_apps/review-apps.sh#L200

/cc @gl-quality/eng-prod