gitlab installation failed by helm
Summary
Maybe this is not a issure, but i really need help. I have there hosts(host-01,host-02,host-03) in aliyun ecs configured as a k8s cluster , when I install gitlab from command line, some pods does not have right status. I need to to install gitlab by helm and use data directory(pv), the pv will be reused when i reinstall the gitlab, but I didn't find any effective instructions about this, could somebody help me, thanks.
Steps to reproduce
1.uninstall current gitlab
# helm uninstall gitlab -n gitlab
2.delete the data directory(/var/data/gitlab) in host-02 and delete pvc and pv
kubectl delete pvc --all -n gitlab;
kubectl delete pv data-gitlab-postgresql-0-pv;
kubectl delete pv gitlab-minio-pv;
kubectl delete pv gitlab-prometheus-server-pv;
kubectl delete pv redis-data-gitlab-redis-master-0-pv;
kubectl delete pv repo-data-gitlab-gitaly-0-pv;
3.apply the pv and pvc definition
kubectl apply -f gitlab-pv-pvc.yaml -n gitlab;
kubectl delete pvc gitlab-minio -n gitlab;
4.execute the installation command, the domain, externalIP and email will be replaced with my real information
helm install gitlab gitlab/gitlab -n gitlab \
--set global.hosts.domain=test.tech \
--set global.hosts.externalIP=47.93.1.1 \
--set redis.volumePermissions.enabled=true \
--set postgresql.volumePermissions.enabled=true \
--set prometheus.volumePermissions.enabled=true \
--set gitlab.gitaly.volumePermissions.enabled=true \
--set postgresql.primary.persistence.volumeName=data-gitlab-postgresql-0 \
--set minio.persistence.existingClaim=gitlab-minio \
--set prometheus.server.persistentVolume.existingClaim=gitlab-prometheus-server \
--set redis.master.persistence.existingClaim=redis-data-gitlab-redis-master-0 \
--set gitlab.gitaly.persistence.existingClaim=repo-data-gitlab-gitaly-0 \
--set certmanager-issuer.email=test@163.com
Configuration used
gitlab-pv-pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: data-gitlab-postgresql-0-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/data/gitlab/data-gitlab-postgresql-0-pv"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- host-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: gitlab-minio-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/data/gitlab/gitlab-minio-pv"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- host-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: gitlab-prometheus-server-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/data/gitlab/gitlab-prometheus-server-pv"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- host-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-data-gitlab-redis-master-0-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/data/gitlab/redis-data-gitlab-redis-master-0-pv"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- host-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: repo-data-gitlab-gitaly-0-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/data/gitlab/repo-data-gitlab-gitaly-0-pv"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- host-02
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-gitlab-postgresql-0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
volumeName: data-gitlab-postgresql-0-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitlab-minio
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
volumeName: gitlab-minio-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitlab-prometheus-server
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
volumeName: gitlab-prometheus-server-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-gitlab-redis-master-0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
volumeName: redis-data-gitlab-redis-master-0-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: repo-data-gitlab-gitaly-0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
volumeName: repo-data-gitlab-gitaly-0-pv
Current behavior
1.there are four pods which are CrashLoopBackOff status or Pending status
# kubectl get pod -n gitlab
NAME READY STATUS RESTARTS AGE
gitlab-certmanager-6dbbfb6f55-q579w 1/1 Running 0 129m
gitlab-certmanager-cainjector-76cff7ff49-l2zxm 1/1 Running 0 129m
gitlab-certmanager-webhook-f95976bc9-tfvbp 1/1 Running 0 129m
gitlab-gitaly-0 0/1 CrashLoopBackOff 29 (5m8s ago) 129m
gitlab-gitlab-exporter-9596548fc-dh9zt 1/1 Running 0 129m
gitlab-gitlab-runner-b59fcc6d8-hrnbl 0/1 Running 36 (5m58s ago) 129m
gitlab-gitlab-shell-9cf98f4f9-bcvwc 1/1 Running 0 128m
gitlab-gitlab-shell-9cf98f4f9-frm2p 1/1 Running 0 129m
gitlab-issuer-1-5rrwf 0/1 Completed 0 129m
gitlab-kas-6767b4594b-5ffpk 1/1 Running 2 (128m ago) 128m
gitlab-kas-6767b4594b-hxzrv 1/1 Running 3 (128m ago) 129m
gitlab-migrations-1-csjg9 0/1 Completed 2 129m
gitlab-minio-5b5bc9fbdb-dzcjs 0/1 Pending 0 129m
gitlab-nginx-ingress-controller-56cfd4bf78-gk4kj 1/1 Running 0 129m
gitlab-nginx-ingress-controller-56cfd4bf78-rlq6s 1/1 Running 0 129m
gitlab-postgresql-0 2/2 Running 0 129m
gitlab-prometheus-server-c4478546-v2qn8 1/2 CrashLoopBackOff 30 (30s ago) 129m
gitlab-redis-master-0 2/2 Running 0 129m
gitlab-registry-7b679ff55b-bps68 1/1 Running 0 128m
gitlab-registry-7b679ff55b-wff7n 1/1 Running 0 129m
gitlab-sidekiq-all-in-1-v2-b4cfdd6fb-7nmmt 1/1 Running 0 129m
gitlab-toolbox-69667dbc7f-hllrz 1/1 Running 0 129m
gitlab-webservice-default-7569b9fdfd-bbmnx 2/2 Running 0 129m
gitlab-webservice-default-7569b9fdfd-tw6qd 2/2 Running 0 128m
2.The pod description shows that it has back-off error
# kubectl describe pod gitlab-gitaly-0 -n gitlab
Name: gitlab-gitaly-0
Namespace: gitlab
Priority: 0
Service Account: default
Node: host-02/172.17.179.16
Start Time: Sun, 28 Apr 2024 21:31:06 +0800
Labels: app=gitaly
chart=gitaly-7.10.1
controller-revision-hash=gitlab-gitaly-789cb799cc
heritage=Helm
release=gitlab
statefulset.kubernetes.io/pod-name=gitlab-gitaly-0
Annotations: checksum/config: 1c24c866398af06a1fa0dba4ea8d15cebdc57bb14fdd86c6d41ea5f247ee1cf7
gitlab.com/prometheus_path: /metrics
gitlab.com/prometheus_port: 9236
gitlab.com/prometheus_scrape: true
prometheus.io/path: /metrics
prometheus.io/port: 9236
prometheus.io/scrape: true
Status: Running
IP: 10.244.1.30
IPs:
IP: 10.244.1.30
Controlled By: StatefulSet/gitlab-gitaly
Init Containers:
certificates:
Container ID: containerd://52649f7149a9cd170499d12bd6baeb6cf1d343046005f6ca595263d215c23508
Image: registry.gitlab.com/gitlab-org/build/cng/certificates:v16.10.1
Image ID: registry.gitlab.com/gitlab-org/build/cng/certificates@sha256:1a2be152600bd4be2f681ec0cbfe9b4d89e97be9b5a77f0109ac45d9e1700377
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Apr 2024 21:31:09 +0800
Finished: Sun, 28 Apr 2024 21:31:11 +0800
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment: <none>
Mounts:
/etc/pki/ca-trust/extracted/pem from etc-pki-ca-trust-extracted-pem (rw)
/etc/ssl/certs from etc-ssl-certs (rw)
configure:
Container ID: containerd://c8e1ff7f090033602301e8422f43482c04a9281856758866970ef12af7d2fde1
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v16.10.1
Image ID: registry.gitlab.com/gitlab-org/build/cng/gitlab-base@sha256:1a57bd6cd15b07fed52ede68bd48d33037935ad54f5376068cf5778738795ea6
Port: <none>
Host Port: <none>
Command:
sh
/config/configure
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Apr 2024 21:31:13 +0800
Finished: Sun, 28 Apr 2024 21:31:13 +0800
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment: <none>
Mounts:
/config from gitaly-config (ro)
/init-config from init-gitaly-secrets (ro)
/init-secrets from gitaly-secrets (rw)
Containers:
gitaly:
Container ID: containerd://e7ae65714c68f02b7ae2791c534c02dd77a5de33fb0a50df11e1c11df0a7d790
Image: registry.gitlab.com/gitlab-org/build/cng/gitaly:v16.10.1
Image ID: registry.gitlab.com/gitlab-org/build/cng/gitaly@sha256:5fc833702a79af5a2cc4a617e8056c9b9619c7d9b979616cc83589636c412ef8
Ports: 8075/TCP, 9236/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 28 Apr 2024 23:40:20 +0800
Finished: Sun, 28 Apr 2024 23:40:20 +0800
Ready: False
Restart Count: 30
Requests:
cpu: 100m
memory: 200Mi
Liveness: exec [/scripts/healthcheck] delay=30s timeout=3s period=10s #success=1 #failure=3
Readiness: exec [/scripts/healthcheck] delay=10s timeout=3s period=10s #success=1 #failure=3
Environment:
CONFIG_TEMPLATE_DIRECTORY: /etc/gitaly/templates
CONFIG_DIRECTORY: /etc/gitaly
GITALY_CONFIG_FILE: /etc/gitaly/config.toml
SSL_CERT_DIR: /etc/ssl/certs
Mounts:
/etc/gitaly/templates from gitaly-config (rw)
/etc/gitlab-secrets from gitaly-secrets (ro)
/etc/pki/ca-trust/extracted/pem from etc-pki-ca-trust-extracted-pem (ro)
/etc/ssl/certs/ from etc-ssl-certs (ro)
/home/git/repositories from repo-data (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
repo-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: repo-data-gitlab-gitaly-0
ReadOnly: false
gitaly-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-gitaly
Optional: false
gitaly-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
init-gitaly-secrets:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: gitlab-gitaly-secret
SecretOptionalName: <nil>
SecretName: gitlab-gitlab-shell-secret
SecretOptionalName: <nil>
etc-ssl-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
etc-pki-ca-trust-extracted-pem:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 69s (x634 over 131m) kubelet Back-off restarting failed container gitaly in pod gitlab-gitaly-0_gitlab(71edcb03-93b6-4eb3-af25-9bea1377950b)
Expected behavior
All the pods status are Running.
Versions
# helm version
version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.21.7"}
# helm list -n gitlab
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
gitlab gitlab 1 2024-04-28 21:30:44.780709041 +0800 CST deployed gitlab-7.10.1 v16.10.1
# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:49Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
host-01 Ready control-plane 278d v1.27.4 172.17.179.14 <none> Ubuntu 22.04.2 LTS 5.15.0-73-generic containerd://1.7.2
host-02 Ready <none> 278d v1.27.4 172.17.179.16 <none> Ubuntu 22.04.2 LTS 5.15.0-73-generic containerd://1.7.2
host-03 Ready <none> 278d v1.27.4 172.17.179.17 <none> Ubuntu 22.04.2 LTS 5.15.0-73-generic containerd://1.7.2
Relevant logs
# kubectl get events -n gitlab
LAST SEEN TYPE REASON OBJECT MESSAGE
4m Warning BackOff pod/gitlab-gitaly-0 Back-off restarting failed container gitaly in pod gitlab-gitaly-0_gitlab(71edcb03-93b6-4eb3-af25-9bea1377950b)
8m53s Warning Unhealthy pod/gitlab-gitlab-runner-b59fcc6d8-hrnbl Readiness probe failed:
3m56s Warning BackOff pod/gitlab-gitlab-runner-b59fcc6d8-hrnbl Back-off restarting failed container gitlab-gitlab-runner in pod gitlab-gitlab-runner-b59fcc6d8-hrnbl_gitlab(7a8376da-fc8a-4e31-8493-f0767c3a5175)
3m44s Warning FailedGetResourceMetric horizontalpodautoscaler/gitlab-gitlab-shell failed to get cpu usage: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
3m29s Normal Sync ingress/gitlab-kas Scheduled for sync
3m29s Normal Sync ingress/gitlab-kas Scheduled for sync
3m29s Normal Sync ingress/gitlab-kas Scheduled for sync
3m44s Warning FailedGetResourceMetric horizontalpodautoscaler/gitlab-kas failed to get cpu usage: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
3m41s Warning FailedScheduling pod/gitlab-minio-5b5bc9fbdb-dzcjs 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
3m53s Normal FailedBinding persistentvolumeclaim/gitlab-minio no persistent volumes available for this claim and no storage class is set
3m29s Normal Sync ingress/gitlab-minio Scheduled for sync
3m29s Normal Sync ingress/gitlab-minio Scheduled for sync
3m29s Normal Sync ingress/gitlab-minio Scheduled for sync
3m53s Warning BackOff pod/gitlab-prometheus-server-c4478546-v2qn8 Back-off restarting failed container prometheus-server in pod gitlab-prometheus-server-c4478546-v2qn8_gitlab(b138d94d-96ae-4084-a31b-ffbab1df070f)
3m29s Normal Sync ingress/gitlab-registry Scheduled for sync
3m29s Normal Sync ingress/gitlab-registry Scheduled for sync
3m29s Normal Sync ingress/gitlab-registry Scheduled for sync
3m44s Warning FailedGetResourceMetric horizontalpodautoscaler/gitlab-registry failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
3m44s Warning FailedGetResourceMetric horizontalpodautoscaler/gitlab-sidekiq-all-in-1-v2 failed to get cpu usage: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
3m29s Normal Sync ingress/gitlab-webservice-default Scheduled for sync
3m29s Normal Sync ingress/gitlab-webservice-default Scheduled for sync
3m29s Normal Sync ingress/gitlab-webservice-default Scheduled for sync
3m44s Warning FailedGetResourceMetric horizontalpodautoscaler/gitlab-webservice-default failed to get cpu usage: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)