Environment incorrectly reports Pod state
When a Pod is Evicted, the UI for the running Pods indicates the Pod is failed. This is not precisely correct which leads to other inaccurate information displayed.
As noted by Kubernetes, the Pod was evicted.
kubectl get pods -n gitlab NAME READY STATUS RESTARTS AGE gitlab-registry-59cfd4f645-xcpnz 0/1 Evicted 0 8d
This also impacts the % Complete of the deployment. While we indicate in the above screenshot "87%" complete, this is false. The deployment is completed, and the desired number of replicas are indeed running and Kubernetes is no longer performing any work on this deployment.
kubectl get replicaset -n gitlab gitlab-registry-59cfd4f645 NAME DESIRED CURRENT READY AGE gitlab-registry-59cfd4f645 35 35 35 8d
(Please ignore the fact that we show 48 instances above and only 35 desired pods. Screenshot and text capture were done at differing times.)
What is the expected correct behavior?
Pods that are Evicted are not relevent to the count of Pods and measure of completeness in this example. They should also not count towards failed Pods. Kubernetes has it's own metric and devices for handling Pods that fail. In this example it would appear something is horribly wrong with our deployment, when that is not the case. In this instance the cluster was at some point starved for resources, but Kubernetes handled well and we continue to have the appropriate number of running and healthy Pods.
I'm not sure how GitLab should handle Evicted pods in this case, however, the information displayed is misleading and will lead to questions that may not fall into the realm that GitLab is able to handle.
Output of checks
- I would suggest relying solely on information from the ReplicaSet for the % Complete metric
- Evicted Pods should be highlighted in a different manner