Unable To Install Gitlab Community Edition On K3S
Summary
I've tried to install Gitlab Community Edition in K3s cluster which is installed in a Docker Container in privileged mode. The K3s engine is running as a single server, without any other additional agents / nodes.
Steps to reproduce
I created a yaml file which contains the Helm chart and I copy that file to
K3s engine in /var/lib/rancher/k3s/server/manifests/
and then it's picked up
by K3s and installed automatically with Helm Chart.
Configuration used
The yaml file that I've created looks as follow:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: gitlab
namespace: kube-system
spec:
chart: gitlab
repo: https://charts.gitlab.io/
targetNamespace: kube-system
version: v6.2.2
set:
valuesContent: |-
global:
edition: ce
hosts:
domain: domain.io
https: true
externalIP: MY_EXTERNAL_IP_ADDRESS
ssh: ~
gitlab:
name: git.domain.io
https: true
registry:
name: registry.domain.io
https: true
minio:
name: storage.domain.io
https: true
minio:
enabled: true
ingress:
configureCertmanager: false
class: "nginx"
enabled: true
tls:
enabled: true
certmanager:
install: false
nginx-ingress:
enabled: true
prometheus:
install: false
redis:
install: true
postgresql:
install: true
gitlab-runner:
install: false
registry:
enable: true
Current behavior
The error that I've encountered is for gitlab-sidekiq-all-in-1-v2
which throws every time:
"Event occurred" object="kube-system/gitlab-sidekiq-all-in-1-v2" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"
The metrics server respond:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
returns:
{
"apiVersion": "metrics.k8s.io/v1beta1",
"items": [
{
"metadata": {
"creationTimestamp": "2022-08-19T11:26:13Z",
"labels": {
"beta.kubernetes.io/arch": "arm64",
"beta.kubernetes.io/instance-type": "k3s",
"beta.kubernetes.io/os": "linux",
"egress.k3s.io/cluster": "true",
"kubernetes.io/arch": "arm64",
"kubernetes.io/hostname": "bd3c30b7cae0",
"kubernetes.io/os": "linux",
"node-role.kubernetes.io/control-plane": "true",
"node-role.kubernetes.io/master": "true",
"node.kubernetes.io/instance-type": "k3s"
},
"name": "bd3c30b7cae0"
},
"timestamp": "2022-08-19T11:26:02Z",
"usage": {
"cpu": "896405200n",
"memory": "1888960Ki"
},
"window": "10s"
}
],
"kind": "NodeMetricsList",
"metadata": {}
}
The Pods statuses are:
kube-system local-path-provisioner-7b7dc8d6f5-vcvtj 1/1 Running 0 3h27m
kube-system coredns-b96499967-69ws2 1/1 Running 0 3h27m
kube-system metrics-server-668d979685-gmfgc 1/1 Running 0 3h27m
kube-system helm-install-gitlab-qqhkp 0/1 Completed 0 3h27m
kube-system gitlab-nginx-ingress-defaultbackend-576f497646-6jlz5 1/1 Running 0 3h25m
kube-system svclb-gitlab-nginx-ingress-controller-81016b60-t2mrf 3/3 Running 0 3h25m
kube-system gitlab-minio-74dfc6b6c7-fxdjf 1/1 Running 0 3h25m
kube-system gitlab-minio-create-buckets-1-gnc4s 0/1 Completed 0 3h25m
kube-system gitlab-kas-7c86f6fc4b-wsskd 1/1 Running 0 3h25m
kube-system gitlab-kas-7c86f6fc4b-29gxx 1/1 Running 0 3h25m
kube-system gitlab-registry-b4c6f7dff-64mmh 1/1 Running 0 3h25m
kube-system gitlab-registry-b4c6f7dff-g4sdd 1/1 Running 0 3h25m
kube-system gitlab-gitlab-exporter-688b496459-kt9m6 1/1 Running 0 3h25m
kube-system gitlab-migrations-1-vsz6n 0/1 Error 0 3h25m
kube-system gitlab-nginx-ingress-controller-859645d798-lt2vf 0/1 Completed 0 3h25m
kube-system gitlab-nginx-ingress-controller-859645d798-tswmk 0/1 Completed 0 3h25m
kube-system gitlab-webservice-default-c7f7f848-tpmxl 0/2 Init:Error 1 (3h20m ago) 3h25m
kube-system gitlab-webservice-default-c7f7f848-j9p54 0/2 Init:Error 1 (3h20m ago) 3h25m
kube-system gitlab-gitlab-shell-87ccff5bd-qm9wf 0/1 Completed 0 3h25m
kube-system gitlab-gitlab-shell-87ccff5bd-lgq9r 0/1 Completed 0 3h25m
kube-system gitlab-toolbox-c6b8555d4-6c7j8 0/1 Error 0 3h25m
kube-system gitlab-sidekiq-all-in-1-v2-84c98d697b-smhtc 0/1 Init:Error 1 (3h20m ago) 3h25m
kube-system gitlab-redis-master-0 2/2 Running 0 3h20m
kube-system gitlab-postgresql-0 2/2 Running 0 3h20m
kube-system gitlab-nginx-ingress-controller-859645d798-vkvw2 1/1 Running 0 3h19m
kube-system gitlab-nginx-ingress-controller-859645d798-t6xgh 1/1 Running 0 3h19m
kube-system gitlab-gitlab-shell-87ccff5bd-6t9f6 1/1 Running 0 3h19m
kube-system gitlab-gitlab-shell-87ccff5bd-8pz86 1/1 Running 0 3h19m
kube-system gitlab-toolbox-c6b8555d4-8ss7l 1/1 Running 0 3h18m
kube-system gitlab-webservice-default-c7f7f848-dprjx 0/2 Init:CrashLoopBackOff 28 (95s ago) 3h19m
kube-system gitlab-gitaly-0 0/1 CrashLoopBackOff 41 (80s ago) 3h20m
kube-system gitlab-sidekiq-all-in-1-v2-84c98d697b-mvpp2 0/1 Init:CrashLoopBackOff 28 (82s ago) 3h18m
kube-system gitlab-webservice-default-c7f7f848-mv8vn 0/2 Init:CrashLoopBackOff 28 (67s ago) 3h19m
Expected behavior
I am expected from Helm to install all the Pods / Services and to work without any other problems.
Versions
- Chart: v6.2.2
- Platform:
- Docker(M1 Max): K3s - Edited (Here is the solution)
- Kubernetes:
- Client: 1.24
- Server: 1.24
- Helm: v3
Relevant logs
K3s Logs:
I0819 14:53:39.741899 50 scope.go:110] "RemoveContainer" containerID="081e25b23883aa2ca867d1cb81e0ad53e02ac472e3dd773162c9d8b15e8c264c"
I0819 14:53:39.742199 50 scope.go:110] "RemoveContainer" containerID="081e25b23883aa2ca867d1cb81e0ad53e02ac472e3dd773162c9d8b15e8c264c"
E0819 14:53:39.746897 50 remote_runtime.go:510] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to set removing state for container \"081e25b23883aa2ca867d1cb81e0ad53e02ac472e3dd773162c9d8b15e8c264c\": container is already in removing state" containerID="081e25b23883aa2ca867d1cb81e0ad53e02ac472e3dd773162c9d8b15e8c264c"
E0819 14:53:39.746922 50 kuberuntime_container.go:797] failed to remove pod init container "dependencies": rpc error: code = Unknown desc = failed to set removing state for container "081e25b23883aa2ca867d1cb81e0ad53e02ac472e3dd773162c9d8b15e8c264c": container is already in removing state; Skipping pod "gitlab-webservice-default-c7f7f848-mv8vn_kube-system(a79b256e-22f8-43ad-9d58-a47aa6f9cd7c)"
E0819 14:53:39.747436 50 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dependencies\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=dependencies pod=gitlab-webservice-default-c7f7f848-mv8vn_kube-system(a79b256e-22f8-43ad-9d58-a47aa6f9cd7c)\"" pod="kube-system/gitlab-webservice-default-c7f7f848-mv8vn" podUID=a79b256e-22f8-43ad-9d58-a47aa6f9cd7c
I0819 14:53:40.749551 50 scope.go:110] "RemoveContainer" containerID="6fcf188fc9c7ae6e590bacb68e6ae79cc703d6ad6187e218d3a426e0e1dfd6cf"
E0819 14:53:45.376287 50 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dependencies\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=dependencies pod=gitlab-webservice-default-c7f7f848-dprjx_kube-system(e67ab014-5acf-4b74-b7dd-e2e24f1cfcb8)\"" pod="kube-system/gitlab-webservice-default-c7f7f848-dprjx" podUID=e67ab014-5acf-4b74-b7dd-e2e24f1cfcb8
E0819 14:53:48.048852 50 horizontal.go:226] failed to compute desired number of replicas based on listed metrics for Deployment/kube-system/gitlab-webservice-default: invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
I0819 14:53:48.048938 50 event.go:294] "Event occurred" object="kube-system/gitlab-webservice-default" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"
I0819 14:53:48.048956 50 event.go:294] "Event occurred" object="kube-system/gitlab-webservice-default" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"
E0819 14:53:48.055434 50 horizontal.go:226] failed to compute desired number of replicas based on listed metrics for Deployment/kube-system/gitlab-sidekiq-all-in-1-v2: invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
I0819 14:53:48.055485 50 event.go:294] "Event occurred" object="kube-system/gitlab-sidekiq-all-in-1-v2" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"
I0819 14:53:48.055498 50 event.go:294] "Event occurred" object="kube-system/gitlab-sidekiq-all-in-1-v2" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"
E0819 14:53:49.376171 50 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dependencies\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=dependencies pod=gitlab-sidekiq-all-in-1-v2-84c98d697b-mvpp2_kube-system(abb1e34d-407d-4297-ae37-db25274b35af)\"" pod="kube-system/gitlab-sidekiq-all-in-1-v2-84c98d697b-mvpp2" podUID=abb1e34d-407d-4297-ae37-db25274b35af
Edited by Mihai