Review Apps aren't accessible and healthy because the `dns-gitlab-review-app-external-dns` Deployment is pending
After some digging, I found out that many deployments/pods were failing to start because Review Apps weren't exposed with a DNS record (e.g. https://console.cloud.google.com/kubernetes/deployment/us-central1-a/review-apps-ce/review-apps-ce/review-ce-9578-ad-b8pic4-gitlab-runner?project=gitlab-review-apps).
In turn, I went to investigate the dns-gitlab-review-app-external-dns Deployment at https://console.cloud.google.com/kubernetes/deployment/us-central1-a/review-apps-ce/review-apps-ce/dns-gitlab-review-app-external-dns?project=gitlab-review-apps&tab=overview&deployment_overview_active_revisions_tablesize=50&duration=PT1H&pod_summary_list_tablesize=20&service_list_datatablesize=20&deployment_revision_history_tablesize=50&dns-gitlab-review-app-external-dns_events_tablesize=50 and found out that its status is "Pods are pending"...
After digging into the Deployment's pod, I found the following interesting error messages in the "Events" tab:
Unable to mount volumes for pod "dns-gitlab-review-app-external-dns-5d997ff5c6-qb7zb_review-apps-ce(06add1c3-87b4-11e9-80a9-42010a800107)": timeout expired waiting for volumes to attach or mount for pod "review-apps-ce"/"dns-gitlab-review-app-external-dns-5d997ff5c6-qb7zb". list of unmounted volumes=[aws-credentials dns-gitlab-review-app-external-dns-token-sj5jm]. list of unattached volumes=[aws-credentials dns-gitlab-review-app-external-dns-token-sj5jm] FailedMount Jun 5, 2019, 7:06:51 PM Jun 6, 2019, 10:51:48 AM 418
MountVolume.SetUp failed for volume "dns-gitlab-review-app-external-dns-token-sj5jm" : mount failed: exit status 1 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/06add1c3-87b4-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/dns-gitlab-review-app-external-dns-token-sj5jm --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/06add1c3-87b4-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/dns-gitlab-review-app-external-dns-token-sj5jm Output: Failed to start transient scope unit: Connection timed out FailedMount Jun 5, 2019, 7:05:52 PM Jun 6, 2019, 10:37:59 AM 382I've now deleted the Deployment and retried a deploy, which recreated the dns-gitlab-review-app-external-dns Deployment automatically but we still have the same mounting problem.