Environment incorrectly reports Pod state

Summary

When a Pod is Evicted, the UI for the running Pods indicates the Pod is failed. This is not precisely correct which leads to other inaccurate information displayed.

image

As noted by Kubernetes, the Pod was evicted.

kubectl get pods -n gitlab
NAME                               READY   STATUS    RESTARTS   AGE
gitlab-registry-59cfd4f645-xcpnz   0/1     Evicted   0          8d

This also impacts the % Complete of the deployment. While we indicate in the above screenshot "87%" complete, this is false. The deployment is completed, and the desired number of replicas are indeed running and Kubernetes is no longer performing any work on this deployment.

kubectl get replicaset -n gitlab gitlab-registry-59cfd4f645
NAME                         DESIRED   CURRENT   READY   AGE
gitlab-registry-59cfd4f645   35        35        35      8d

(Please ignore the fact that we show 48 instances above and only 35 desired pods. Screenshot and text capture were done at differing times.)

What is the expected correct behavior?

Pods that are Evicted are not relevent to the count of Pods and measure of completeness in this example. They should also not count towards failed Pods. Kubernetes has it's own metric and devices for handling Pods that fail. In this example it would appear something is horribly wrong with our deployment, when that is not the case. In this instance the cluster was at some point starved for resources, but Kubernetes handled well and we continue to have the appropriate number of running and healthy Pods.

I'm not sure how GitLab should handle Evicted pods in this case, however, the information displayed is misleading and will lead to questions that may not fall into the realm that GitLab is able to handle.

Output of checks

GitLab.com

Possible fixes

  • I would suggest relying solely on information from the ReplicaSet for the % Complete metric
  • Evicted Pods should be highlighted in a different manner