Newest release is very slow on gitlab-shared-secrets
Summary
When trying to update I had this problem across 10 different clusters. So this is almost certainly related to something new that changed in here.
Steps to reproduce
Perform helm upgrade
to >= 5.9.0
Current behavior
Container: gitlab
Disconnected
/tmp/tmp.DptlNxwh3V /
secret "gitlab-gitlab-initial-root-password" already exists.
I0324 20:11:40.626028 64 request.go:621] Throttling request took 1.190136088s, request: GET:https://10.43.0.1:443/apis/acme.cert-manager.io/v1alpha2?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-initial-root-password not labeled
I0324 20:11:47.495929 95 request.go:621] Throttling request took 1.189648919s, request: GET:https://10.43.0.1:443/apis/metacontroller.k8s.io/v1alpha1?timeout=32s
secret/gitlab-gitlab-initial-root-password not labeled
secret "gitlab-redis-secret" already exists.
I0324 20:12:08.134779 170 request.go:621] Throttling request took 1.186344232s, request: GET:https://10.43.0.1:443/apis/acme.cert-manager.io/v1beta1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-redis-secret not labeled
I0324 20:12:15.029269 189 request.go:621] Throttling request took 1.188103383s, request: GET:https://10.43.0.1:443/apis/events.k8s.io/v1beta1?timeout=32s
secret/gitlab-redis-secret not labeled
secret "gitlab-postgresql-password" already exists.
I0324 20:12:42.567116 298 request.go:621] Throttling request took 1.175817785s, request: GET:https://10.43.0.1:443/apis/ui.cattle.io/v1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-postgresql-password not labeled
I0324 20:12:49.463813 317 request.go:621] Throttling request took 1.188485426s, request: GET:https://10.43.0.1:443/apis/volumesnapshot.external-storage.k8s.io/v1?timeout=32s
secret/gitlab-postgresql-password not labeled
secret "gitlab-gitlab-shell-secret" already exists.
I0324 20:13:10.090286 394 request.go:621] Throttling request took 1.188010157s, request: GET:https://10.43.0.1:443/apis/coordination.k8s.io/v1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-shell-secret not labeled
I0324 20:13:16.974211 413 request.go:621] Throttling request took 1.18908914s, request: GET:https://10.43.0.1:443/apis/autoscaling/v2beta1?timeout=32s
secret/gitlab-gitlab-shell-secret not labeled
secret "gitlab-gitaly-secret" already exists.
I0324 20:13:37.597482 488 request.go:621] Throttling request took 1.18379451s, request: GET:https://10.43.0.1:443/apis/acme.cert-manager.io/v1beta1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-gitaly-secret not labeled
I0324 20:13:44.464645 507 request.go:621] Throttling request took 1.189994141s, request: GET:https://10.43.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s
secret/gitlab-gitaly-secret not labeled
secret "gitlab-gitlab-runner-secret" already exists.
I0324 20:14:12.053609 613 request.go:621] Throttling request took 1.185463084s, request: GET:https://10.43.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-runner-secret not labeled
I0324 20:14:18.935503 632 request.go:621] Throttling request took 1.18116581s, request: GET:https://10.43.0.1:443/apis/cert-manager.io/v1beta1?timeout=32s
secret/gitlab-gitlab-runner-secret not labeled
secret "gitlab-incoming-email-auth-token" already exists.
I0324 20:14:39.547671 709 request.go:621] Throttling request took 1.188565434s, request: GET:https://10.43.0.1:443/apis/apps/v1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-incoming-email-auth-token not labeled
I0324 20:14:46.418187 728 request.go:621] Throttling request took 1.180197107s, request: GET:https://10.43.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s
secret/gitlab-incoming-email-auth-token not labeled
Generating a RSA private key
....++++
..........................................................................................++++
writing new private key to 'certs/registry-example-com.key'
-----
secret "gitlab-registry-secret" already exists.
I0324 20:15:00.517819 778 request.go:621] Throttling request took 1.19007846s, request: GET:https://10.43.0.1:443/apis/coordination.k8s.io/v1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-secret not labeled
I0324 20:15:07.389924 797 request.go:621] Throttling request took 1.186092279s, request: GET:https://10.43.0.1:443/apis/cert-manager.io/v1alpha2?timeout=32s
secret/gitlab-registry-secret not labeled
I0324 20:15:21.186782 836 request.go:621] Throttling request took 1.187134377s, request: GET:https://10.43.0.1:443/apis/machinelearning.seldon.io/v1alpha2?timeout=32s
secret/gitlab-rails-secret configured
I0324 20:15:31.458310 921 request.go:621] Throttling request took 1.178646781s, request: GET:https://10.43.0.1:443/apis/storage.k8s.io/v1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-rails-secret not labeled
I0324 20:15:38.355945 941 request.go:621] Throttling request took 1.176902004s, request: GET:https://10.43.0.1:443/apis/metacontroller.k8s.io/v1alpha1?timeout=32s
secret/gitlab-rails-secret not labeled
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
secret "gitlab-gitlab-shell-host-keys" already exists.
I0324 20:15:52.307905 993 request.go:621] Throttling request took 1.188963817s, request: GET:https://10.43.0.1:443/apis/batch/v1beta1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-shell-host-keys not labeled
I0324 20:15:59.187825 1012 request.go:621] Throttling request took 1.177780729s, request: GET:https://10.43.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
secret/gitlab-gitlab-shell-host-keys not labeled
secret "gitlab-gitlab-workhorse-secret" already exists.
I0324 20:16:19.880111 1089 request.go:621] Throttling request took 1.186716345s, request: GET:https://10.43.0.1:443/apis/hobbyfarm.io/v1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-workhorse-secret not labeled
I0324 20:16:26.774411 1108 request.go:621] Throttling request took 1.186664974s, request: GET:https://10.43.0.1:443/apis/batch/v1?timeout=32s
secret/gitlab-gitlab-workhorse-secret not labeled
secret "gitlab-registry-httpsecret" already exists.
I0324 20:16:47.410820 1184 request.go:621] Throttling request took 1.187143175s, request: GET:https://10.43.0.1:443/apis/metacontroller.k8s.io/v1alpha1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-httpsecret not labeled
I0324 20:16:54.275517 1204 request.go:621] Throttling request took 1.188060148s, request: GET:https://10.43.0.1:443/apis/argoproj.io/v1alpha1?timeout=32s
secret/gitlab-registry-httpsecret not labeled
secret "gitlab-registry-notification" already exists.
I0324 20:17:14.935078 1279 request.go:621] Throttling request took 1.186876406s, request: GET:https://10.43.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-notification not labeled
I0324 20:17:21.796385 1298 request.go:621] Throttling request took 1.186781571s, request: GET:https://10.43.0.1:443/apis/policy/v1beta1?timeout=32s
secret/gitlab-registry-notification not labeled
Expected behavior
Shouldn't take that long, but like before. It is so slow the upgrade fails because of a timeout for the precondition... .
Versions
- Chart: 5.9.1
- Platform:
- Self-hosted: Rancher RKE)
- Kubernetes: (
kubectl version
)- Client: v1.23.5
- Server: v1.23.5
- Helm: (
helm version
)- Client: rancher
- Server: ranhcer
Edited by Jason Plum