review-deploy jobs failing with migration errors and Gitaly failures
Status
Resolved, trigger a new pipeline in failing merge requests to redeploy the review app.
- As of 2022-05-10 2053 UTC both MRs have been fixed. Current advise is to rebase failed MRs to resolve the problem. Testing is underway to determine if Running a new pipeline will in addition to rebase.
- As of 2022-05-10 1757 UTC, MRs which require Review Deploy are blocked from merge due to 2 failures with fixes in progress:
- Gitaly failed to start due to a mismatch of chart version and image from gitlab-org/charts/gitlab!2535 (merged) and gitlab-org/build/CNG!998 (merged). Images used in the deploy build from
CNG:master
but they are deployed with a pinned version of the chart at5.10.0
=> Will be fixed by moving the chart version forward with !87122 (merged) - Migrations failed to execute due to
wait_for_deps
returning nil instead of 0 when the schema hasn't been initialized due to gitlab-org/build/CNG!953 (merged) => fixed by gitlab-org/build/CNG!1016 (merged)
https://gitlab.com/gitlab-org/gitlab/-/jobs/2433807995
W0510 01:41:30.092790 123 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Error: timed out waiting for the condition
Pods for release review-ml-run-qa-3jjvqh
NAME READY STATUS RESTARTS AGE
review-ml-run-qa-3jjvqh-gitaly-0 0/1 CrashLoopBackOff 8 20m
review-ml-run-qa-3jjvqh-gitlab-runner-54bf89dcc7-mlhm4 1/1 Running 3 20m
review-ml-run-qa-3jjvqh-gitlab-shell-6486d4c79f-g8xc7 1/1 Running 0 18m
review-ml-run-qa-3jjvqh-gitlab-shell-6486d4c79f-tcq9n 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-minio-5966d4d794-648hn 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-minio-create-buckets-1-qbshk 0/1 Completed 0 20m
review-ml-run-qa-3jjvqh-nginx-ingress-controller-57f557b848sqkw 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-nginx-ingress-controller-57f557b84hz94f 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-nginx-ingress-defaultbackend-5c5797b84v 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-postgresql-0 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-redis-master-0 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-registry-68f77657c6-6nxpn 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-sidekiq-all-in-1-v2-7c5db6dc4-zqk57 0/1 Init:CrashLoopBackOff 7 20m
review-ml-run-qa-3jjvqh-toolbox-59b5cfc7f7-mgg9r 1/1 Running 0 20m
review-ml-run-qa-3jjvqh-webservice-default-f55fc6cb8-bbnfh 0/2 Init:CrashLoopBackOff 7 20m
review-ml-run-qa-3jjvqh-webservice-default-f55fc6cb8-m5c55 0/2 Init:CrashLoopBackOff 7 18m
Unsuccessful Jobs for release review-ml-run-qa-3jjvqh
NAME COMPLETIONS DURATION AGE
review-ml-run-qa-3jjvqh-migrations-1 0/1 20m 20m
Edited by Kyle Wiebers