Review Apps aren't accessible and healthy because the `dns-gitlab-review-app-external-dns` Deployment is pending

added Quality ci-build priority1 severity1 typebug labels

changed the description

@WarheadsSE or @ibaum maybe you can help with that?

mentioned in merge request !28906 (merged)

I'm looking into this @rymai

Thanks! The problematic Deployment is https://console.cloud.google.com/kubernetes/deployment/us-central1-a/review-apps-ce/review-apps-ce/dns-gitlab-review-app-external-dns?project=gitlab-review-apps&tab=overview&deployment_overview_active_revisions_tablesize=50&duration=PT1H&pod_summary_list_tablesize=20&service_list_datatablesize=20&selectedType=v&selectedName=aws-credentials.

It looks like the secret it is attempting to mount (aws-credentials) have been deleted from the namespace!

I was wrong. Just misread the volume definition

This error is seeming to be a node-level issue. I'm attempting to drain the node, but seeing Cannot evict pod as it would violate the pod's disruption budget. from many of the pods.

Looking into one of these pods, I am seeing unt -t tmpfs tmpfs /var/lib/kubelet/pods/6e682ca5-87db-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/default-token-5vgvd Output: Failed to start transient scope unit: Connection timed out from a registry pod I'm attempting to move. This is occuring on another node (db983b3e-pq36) apart from the original (db983b3e-5sj9).

This appears to be an underlying problem with the systemd units on the node, and systemctl becoming unresponsive.

It appears from this issue that a strong suggestion is for v237 or a patch. The GCP node shows v232. I can't be sure it if has this patch.

This appears to be an underlying problem with the systemd units on the node, and systemctl becoming unresponsive.

It appears from this issue that a strong suggestion is for v237 or a patch. The GCP node shows v232. I can't be sure it if has this patch.

Good find! I found this issue but wasn't sure how to act according to it.

Could you please share your debugging steps (when you have the time) and also how would we update systemd on the GCP nodes?

Debugging steps:

switch kubectl context to review-apps-ce (I use kubectx)
kubectl get pods | grep dns
kubectl describe pod <pod name> & confirm exact error message
web search for exact error message, following rabbit hole.
Access node over SSH via GCP console
systemctl --version => systemd 232
Gather some information:
- mount | grep kube | wc -l => 290
- systemctl list-units --all | grep -i var-lib-kube | wc -l => 142

Node db983b3e-bkkk seems to be able to take the pod I've been attempting to move. I can't ssh to db983b3e-pq36 as it seems GCP can't deploy the SSH keys to it successfully.

I am attempting to drain node db983b3e-pq36 as well.

You can fetch all pods on a given node via kubectl get pods --field-selector=spec.nodeName=NODE_NAME.

db983b3e-pq36 has 94 pods, 71 of which are stuck in ContainerCreating or Init with failing mounts.

Node db983b3e-5sj9 had 25 remaining, with 6 stuck.

I forcibly drained (e.g. kubectl delete pods --field-selector=spec.nodeName=NODE_NAME) & restarted the node over SSH. I've kubectl uncordon'd it as well.

I've managed to get ssh into db983b3e-pq36 (shortened), and performing the same steps:

db983b3e-pq36 $ systemctl list-units --all | grep kube | wc -l
115
db983b3e-pq36 $ mount | grep kube | wc -l
210

laptop $ kubectl drain db983b3e-pq36
laptop $ kubectl get pods --field-selector=spec.nodeName=db983b3e-pq36
laptop $ kubectl delete pods --field-selector=spec.nodeName=db983b3e-pq36

db983b3e-pq36 $ sudo systemctl reboot

laptop $ kubectl uncordon db983b3e-pq36

Success

$ kubectl get pods | grep dns
dns-gitlab-review-app-external-dns-675f4f56b7-dbzb2               1/1     Running                      0          50m

@rymai

how would we update systemd on the GCP nodes?

Short answer: we don't. We don't have that control, and we really don't want to get into the business of machine image maintenance.

@WarheadsSE Wow, your messages are so useful, I'm learning so much with each of your messages! I cannot thank you enough, I'm going to document all that! Thanks again!

Short answer: we don't. We don't have that control, and we really don't want to get into the business of machine image maintenance.

Aha, that's what I thought!

assigned to @WarheadsSE

Summary of what happened:

new mounts were failing, when being performed with transient scopes (e.g. pods) of systemd-mount
Found others had seen this before, in varying degrees, on multiple platforms
Attempted to drain the node affected, which failed because pods were being rescheduled onto another node hitting this issue
kubectl draind affected nodes simultaneously. This results in cordoned state (new pods can't be scheduled).
After as many pods as possible moved off, forcibly removed any remaining pods & restarted nodes.
uncordon'd affected nodes, now that they have been rebooted.

Thanks for the summary. Do you think we could avoid this in the future by using machines with less resources so that a single node won't receive too many mounts?

I'm thinking it may be needed to reduce the load on an individual system, especially if we're using many mounts. It's becoming apparent we're not hitting system resource limitations in terms of CPU/memory, but rather subsystem limitations exposed by the automation Kubernetes provides.

We're seeing ~70 pods per node. At least with the GitLab application suite deployment .. this is probably just too many.

I can't fully say this is solved, because there are pods still not in running state, due to volumes.

One such node: db983b3e-w3t4 has 70 pods in non-Running states.

This node is also seeing

Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/9069fa14-8856-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/default-token-5vgvd --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/9069fa14-8856-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/default-token-5vgvd
Output: Failed to start transient scope unit: Connection timed out

I am going to attempt one of the suggestions, systemctl daemon-reload to see if this removes the need to restart the node. I do not expect it will.

Certainly seeing this.

node # systemctl list-units --all | wc -l
Failed to list units: Connection timed out
0

node # systemctl daemon-reload
Failed to reload daemon: Connection timed out

... Okay ...

node # mount | grep kube | wc -l
582

laptop $ kubectl drain db983b3e-w3t4
...
node # systemctl list-units --all | wc -l
737
node # systemctl daemon-reload
node # echo $?
0

Now watching to see if any of the "troublesome" pods start working.

It has taken some time, but it appears that the number of non-Running state containers has dropped from 60 to 50 while Running has climbed to 21, and mounts have climbed to 1026

I've uncordond the db983b3e-w3t4 node

Climbing! We're at 31 over 38 now, from 60 over 8! (non-running over running)

This node has not full recovered yet, but I think a number of these not running are due to dependencies.

I have found db983b3e-fj6n is also in this state.

added 1 deleted label

added Unscheduled label

@WarheadsSE FYI in parallel I've been deleting old Review Apps using helm delete because they were probably stuck in a state where they had no DNS because of the dns-gitlab-review-app-external-dns Deployment issue originally reported here.

I've also created a new node pools of smaller machines and will delete the previous pool.

It looks like everything is rolling over.

kubectl get pods | grep -v 'Running' | grep -v 'Completed' shows that there are no pods in a bad stated

@WarheadsSE Yeah, everything's good now! Thanks again, I'll try to document this as best as I could.

If you need help or explanation with them, let me know

@rymai For the sake of completeness, I am looking at the review-apps-ee cluster as well. I will report back, but we may have similar issues there.

sigh Yup.

Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/7859e8ea-886f-11e9-83b7-42010af0001f/volumes/kubernetes.io~secret/default-token-k9j8n --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/7859e8ea-886f-11e9-83b7-42010af0001f/volumes/kubernetes.io~secret/default-token-k9j8n
Output: Failed to start transient scope unit: Connection timed out

Let me clear the Helm releases first.

It looks like even new nodes (from the new smaller-machines pool) are immediately hitting...

Unable to mount volumes for pod "review-5456-add-c-38s8rz-registry-7cb4797cd5-2mj8j_review-apps-ee(e1357a37-8888-11e9-83b7-42010af0001f)": timeout expired waiting for volumes to attach or mount for pod "review-apps-ee"/"review-5456-add-c-38s8rz-registry-7cb4797cd5-2mj8j". list of unmounted volumes=[registry-secrets default-token-k9j8n]. list of unattached volumes=[registry-server-config registry-secrets etc-ssl-certs default-token-k9j8n]

Ok, I think we're back at a good state, at least the tiller and external-dns Deployments are running properly.

With services appearing to work again, and documentation opened elsewhere, I'm going to close this issue.

If further problems related to this surface, we can re-open.

closed

@WarheadsSE @rymai

We're seeing this error now:

Error: a release named dns-gitlab-review-app already exists.
Run: helm ls --all dns-gitlab-review-app; to check the status of the release
Or run: helm del --purge dns-gitlab-review-app; to delete it

Could it be related?

Example jobs from different pipelines:

Strange that it sees the previous deployment as FAILED:

** Checking for previous deployment of dns-gitlab-review-app **
Previous deployment found, checking status...
Previous deployment state: FAILED

But it looks fine to me: https://console.cloud.google.com/kubernetes/deployment/us-central1-a/review-apps-ce/review-apps-ce/dns-gitlab-review-app-external-dns?project=gitlab-review-apps&tab=overview&deployment_overview_active_revisions_tablesize=50&duration=PT1H&pod_summary_list_tablesize=20&service_list_datatablesize=20

No, wait, that's the CE one

I don't see one in the EE cluster: https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps&workload_list_tablesize=50&workload_list_tablequery=%255B%257B_22k_22_3A_22is_system_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22false_5C_22_22_2C_22s_22_3Atrue%257D_2C%257B_22k_22_3A_22metadata%252Fname_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22dns-gitlab-review-app-external-dns_5C_22_22_2C_22s_22_3Atrue%257D_2C%257B_22k_22_3A_22metadata%252FclusterReference%252Fname_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22review-apps-ee_5C_22_22_2C_22s_22_3Atrue%257D%255D

Yeah, I think that's the issue. I'll investigate, thanks!

It seems the namespace is now of the form gitlab-ee-278964 instead of review-apps-ee, e.g.

https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/228719825 shows

** Checking if dns-gitlab-review-app exists in the gitlab-ee-278964 namespace... **

compared to an old review-deploy job which was showing:

** Checking if dns-gitlab-review-app exists in the review-apps-ce namespace... **

Note the wrong namespace in the first case. I think something changed in our AutoDevOps feature recently...

This can be seen clearly here:

** Ensuring the gitlab-ee-278964 namespace exists... **
Name:         gitlab-ee-278964
Labels:       <none>
Annotations:  <none>
Status:       Active

No resource quota.

No resource limits.

vs before

** Ensuring the review-apps-ce namespace exists... **
Name:         review-apps-ce
Labels:       <none>
Annotations:  <none>
Status:       Active

No resource quota.

No resource limits.

Ok, I found that in our documentation:

If you choose to manage your own cluster, project-specific resources will not be created automatically. If you are using Auto DevOps, you will need to explicitly provide the KUBE_NAMESPACE deployment variable that will be used by your deployment jobs, otherwise a namespace will be created for you.

– https://docs.gitlab.com/ee/user/project/clusters/#gitlab-managed-clusters

I've added KUBE_NAMESPACE as a CI/CD variable for now but I think the existing namespace specified in the UI (e.g. ) should be respected so I'll open a bug.

The external-dns chart was properly installed now:

** Installing external DNS for domain gitlab-review.app... **

** Checking if dns-gitlab-review-app exists in the review-apps-ee namespace... **
Deployment status for dns-gitlab-review-app is 1
Installing external-dns Helm chart
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
NAME:   dns-gitlab-review-app
LAST DEPLOYED: Tue Jun 11 09:26:35 2019
NAMESPACE: review-apps-ee
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Deployment
NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
dns-gitlab-review-app-external-dns  1        1        1           0          0s

==> v1/Pod(related)
NAME                                                 READY  STATUS             RESTARTS  AGE
dns-gitlab-review-app-external-dns-5ccf5d76c7-f4swg  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME                                TYPE    DATA  AGE
dns-gitlab-review-app-external-dns  Opaque  2     0s

==> v1/ServiceAccount
NAME                                SECRETS  AGE
dns-gitlab-review-app-external-dns  1        0s

==> v1beta1/ClusterRole
NAME                                AGE
dns-gitlab-review-app-external-dns  0s

==> v1beta1/ClusterRoleBinding
NAME                                AGE
dns-gitlab-review-app-external-dns  0s

==> v1/Service
NAME                                TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
dns-gitlab-review-app-external-dns  ClusterIP  10.63.248.135  <none>       7979/TCP  0s


NOTES:
To verify that external-dns has started, run:

  kubectl --namespace=review-apps-ee get pods -l "app=external-dns,release=dns-gitlab-review-app"

Created https://gitlab.com/gitlab-org/gitlab-ce/issues/63079 to track that.

reopened

closed

@WarheadsSE I think I've stabilized the Review Apps deployments now. I've created a new node pool of n1-standard-1 machines so that we avoid having too many pods trying to mount too many points. I don't think that will address 100% of the problem but that should at least avoid it in most cases...

All Review Apps are currently properly deployed, and no pods are in a bad state.

mentioned in issue #64356 (closed)

added groupdistribution label

Review Apps aren't accessible and healthy because the `dns-gitlab-review-app-external-dns` Deployment is pending

Designs

Child items ...

Activity