Failed to upgrade Gitlab from version 14.0.3 to 14.10.1
Ansible was failed to upgrade Gitlab from version 14.0.3 to 14.10.1. Sidekiq and Webservice Pods are stuck in Init:rashLoopBackOff
. We did notice the pod for data-migration
job is not showing on the list anymore. Right now we can not bring back the service, keep getting 502 Error. Just wondering how to troubleshoot this issue, and how to fix it and restore the service? Please advise. Thanks.
NAMESPACE NAME READY STATUS RESTARTS AGE
default gitlab-gitlab-exporter-679895ccf6-chsfq 1/1 Running 0 49m
default gitlab-gitlab-shell-66b5cf74f8-bgpl9 1/1 Running 0 48m
default gitlab-gitlab-shell-66b5cf74f8-d2z99 1/1 Running 0 49m
default gitlab-grafana-app-75fb676cb7-rtgt5 2/2 Running 0 161m
default gitlab-nginx-ingress-controller-75fb464dfc-gkvjb 1/1 Running 0 49m
default gitlab-nginx-ingress-controller-75fb464dfc-pqlrc 1/1 Running 0 48m
default gitlab-nginx-ingress-defaultbackend-5cb4779c97-m7z8p 1/1 Running 0 49m
default gitlab-prometheus-server-85749767f-7fb6p 2/2 Running 0 49m
default gitlab-sidekiq-all-in-1-v2-7b88f9c8b9-vlds9 0/1 Init:CrashLoopBackOff 12 49m
default gitlab-toolbox-865c4f8cbb-m565s 1/1 Running 0 49m
default gitlab-webservice-default-69c8b7bd98-ntrpp 2/2 Running 0 161m
default gitlab-webservice-default-8cc6f57f-l4lt4 0/2 Init:CrashLoopBackOff 12 48m
default gitlab-webservice-default-8cc6f57f-vssl9 0/2 Init:CrashLoopBackOff 12 49m
kube-system aws-node-grt4f 1/1 Running 0 8h
kube-system aws-node-h2pt7 1/1 Running 0 8h
kube-system aws-node-hv22w 1/1 Running 0 8h
kube-system aws-node-hwdlf 1/1 Running 0 8h
kube-system aws-node-kgg72 1/1 Running 0 8h
kube-system aws-node-spvld 1/1 Running 0 8h
kube-system coredns-6bfbc5f9f8-95jck 1/1 Running 0 8h
kube-system coredns-6bfbc5f9f8-d5pn6 1/1 Running 0 8h
kube-system kube-proxy-29hjs 1/1 Running 0 8h
kube-system kube-proxy-blvg4 1/1 Running 0 8h
kube-system kube-proxy-g4jll 1/1 Running 0 8h
kube-system kube-proxy-pchc2 1/1 Running 0 8h
kube-system kube-proxy-t47nw 1/1 Running 0 8h
kube-system kube-proxy-vqv5k 1/1 Running 0 8h
kube-system metrics-server-6594d67d48-qt65k 1/1 Running 0 4h32m
This is what we got after executing the command - kubectl logs gitlab-webservice-default-8cc6f57f-l4lt4 -c dependencies
Database Schema - main (gitlabhq_production) - current: 20220322035654, codebase: 20220413075921
Checking: main
Database Schema - main (gitlabhq_production) - current: 20220322035654, codebase: 20220413075921
WARNING: Not all services were operational, with data migrations completed.
If this container continues to fail, please see: https://docs.gitlab.com/charts/troubleshooting/index.html#application-containers-constantly-initializing
Then we have followed the link above and performed some troubleshooting. This is what we got back.
kubectl get job -lapp=migrations
NAME COMPLETIONS DURATION AGE
gitlab-migrations-3 0/1 52m 52m
kubectl get pod -ljob-name=gitlab-migrations-3
No resources found in default namespace.