"Error: UPGRADE FAILED: timed out waiting for the condition" when trying to update
Summary
I'm trying to update my gitlab from chart version 1.3.3 to 1.5.1.
Steps to reproduce
Run 'helm upgrade gitlab gitlab/gitlab --version 1.5.1 -f gitlab.yaml'
Configuration used
certmanager-issuer:
email: XXX
global:
email:
display_name: XXXX
from: XXX
reply_to: XXX
subject_suffix: XXX
hosts:
domain: XXX
externalIP: XXX
psql:
host: XXX
password:
key: password
secret: postgres-password
username: XXX
redis:
host: XXX
password:
key: key
secret: redis-access-key
smtp:
address: XXX
authentication: login
enabled: true
openssl_verify_mode: none
password:
key: password
secret: email-password
port: 587
starttls_auto: true
user_name: XXX
postgresql:
install: false
redis:
enabled: false
Current behavior
Update fails with 'Error: UPGRADE FAILED: timed out waiting for the condition'. Also after trying several times I see a lot of pending 'shared secrets' pods (I tried updating 4 days ago and today...). I also specified a longer timeout with '--timeout 1200', but that didn't help.
NAME READY STATUS RESTARTS AGE
cm-acme-http-solver-cv89h 1/1 Running 0 3h13m
cm-acme-http-solver-fpn9z 1/1 Running 0 3h13m
gitlab-certmanager-664f847794-lhztl 1/1 Running 6 62d
gitlab-gitaly-0 1/1 Running 1 54d
gitlab-gitlab-runner-5d4bf868d-lfw75 1/1 Running 7 62d
gitlab-gitlab-shell-5f66f76d6f-5cn4d 1/1 Running 1 54d
gitlab-gitlab-shell-5f66f76d6f-bn6wk 1/1 Running 1 54d
gitlab-migrations.3-r29f9 0/1 Completed 0 54d
gitlab-minio-8dc7f5964-949cf 1/1 Running 1 62d
gitlab-minio-create-buckets.3-658d8 0/1 Completed 0 54d
gitlab-nginx-ingress-controller-688b48c456-g9hp9 1/1 Running 2 62d
gitlab-nginx-ingress-controller-688b48c456-gdzln 1/1 Running 2 62d
gitlab-nginx-ingress-controller-688b48c456-zd8nt 1/1 Running 2 62d
gitlab-nginx-ingress-default-backend-cb9857f68-4mw6n 1/1 Running 1 62d
gitlab-nginx-ingress-default-backend-cb9857f68-wr47c 1/1 Running 1 62d
gitlab-prometheus-server-8cf4fdd8-78s98 2/2 Running 2 62d
gitlab-registry-5779d776d6-28zd5 1/1 Running 1 62d
gitlab-registry-5779d776d6-lccgw 1/1 Running 1 62d
gitlab-shared-secrets.10-xlp-rt9jc 0/1 Pending 0 4d2h
gitlab-shared-secrets.11-yeh-drmzf 0/1 Pending 0 4h9m
gitlab-shared-secrets.4-p37-854zc 0/1 Pending 0 4d4h
gitlab-shared-secrets.5-trl-9n8xd 0/1 Pending 0 4d4h
gitlab-shared-secrets.6-3ld-mkgjv 0/1 Pending 0 4d4h
gitlab-shared-secrets.7-uep-ft84v 0/1 Pending 0 4d3h
gitlab-shared-secrets.8-v6a-wp8ts 0/1 Pending 0 4d3h
gitlab-shared-secrets.9-qk7-cdh9t 0/1 Pending 0 4d3h
gitlab-sidekiq-all-in-1-6898654bff-5mdp2 0/1 Pending 0 54d
gitlab-sidekiq-all-in-1-7895845f74-6j47n 1/1 Running 1 62d
gitlab-task-runner-cfff49db8-g5z26 1/1 Running 1 54d
gitlab-unicorn-58999cbf64-f99st 0/2 Pending 0 54d
gitlab-unicorn-fd847cc9-kdhqq 2/2 Running 2 62d
gitlab-unicorn-fd847cc9-mmq6w 0/2 Pending 0 62d
Expected behavior
Update works.
Versions
-
Chart: 1.3.3
-
Platform:
- Cloud: AKS
-
Kubernetes: (
kubectl version
)- Client: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
- Server:Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.11", GitCommit:"1bfeeb6f212135a22dc787b73e1980e5bccef13d", GitTreeState:"clean", BuildDate:"2018-09-28T21:35:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
-
Helm: (
helm version
)- Client:Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
- Server:Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Relevant logs
Which logs would help debugging?
Thank you!