gitlab-postgresql-0 fails with a CrashLoopBackOff error when using efs dynamic provisioning
Summary
gitlab-postgresql-0 fails with a CrashLoopBackOff error.
NOTE: I have plans to use AWS RDS Postgres and AWS Elasticache (for redis) in future but just want to get a POC running for now.
Steps to reproduce
- Create an EKS cluster (I am using Bottlerocket AMI)
- Enable EFS for persistence
- Run the below commands
kubectl create ns gitlab
helm install gitlab gitlab/gitlab --version 5.1.2 \
--set nginx-ingress.enabled=False \
--set global.ingress.configureCertmanager=False \
--set certmanager.install=False \
--set global.hosts.domain=my.domain.comConfiguration used
As above, I am using these values in a vanilla Kubernetes (EKS) cluster
helm install gitlab gitlab/gitlab --version 5.1.2 \
--set nginx-ingress.enabled=False \
--set global.ingress.configureCertmanager=False \
--set certmanager.install=False \
--set global.hosts.domain=my.domain.comCurrent behavior
gitlab-postgresql-0 gets stuck in a CrashLoopBackOff error. There are not much logs available for me to figure out why it is failing.
The official bitnami postegresql helm chart works fine on my cluster, I am assuming there is something missing in the postgres in this chart.
Expected behavior
All pods should be in Running state.
Versions
- Chart: 5.1.2
- Platform:
- Cloud: EKS
- Kubernetes: (
kubectl version)- Client: v1.21.2
- Server: v1.21.2
❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:52:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}- Helm:
❯ helm version
version.BuildInfo{Version:"v3.6.2", GitCommit:"ee407bdf364942bcb8e8c665f82e15aa28009b71", GitTreeState:"dirty", GoVersion:"go1.16.5"}Relevant logs
❯ k logs -f gitlab-postgresql-0 gitlab-postgresql
postgresql 23:16:46.29
postgresql 23:16:46.30 Welcome to the Bitnami postgresql container
postgresql 23:16:46.30 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 23:16:46.30 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 23:16:46.30
postgresql 23:16:46.31 INFO ==> ** Starting PostgreSQL setup **
postgresql 23:16:46.33 INFO ==> Validating settings in POSTGRESQL_* env vars..
postgresql 23:16:46.34 INFO ==> Loading custom pre-init scripts...
postgresql 23:16:46.35 INFO ==> Loading user's custom files from /docker-entrypoint-preinitdb.d ...
postgresql 23:16:46.35 INFO ==> Initializing PostgreSQL database...
postgresql 23:16:46.39 INFO ==> pg_hba.conf file not detected. Generating it...
postgresql 23:16:46.40 INFO ==> Generating local authentication configuration
❯ k describe po gitlab-postgresql-0
Name: gitlab-postgresql-0
Namespace: gitlab
Priority: 0
Node: ip-10-20-24-19.ec2.internal/10.20.24.19
Start Time: Thu, 12 Aug 2021 09:10:42 +1000
Labels: app=postgresql
chart=postgresql-8.9.4
controller-revision-hash=gitlab-postgresql-78bbc78f9c
heritage=Helm
release=gitlab
role=master
statefulset.kubernetes.io/pod-name=gitlab-postgresql-0
Annotations: kubernetes.io/psp: eks.privileged
postgresql.gitlab/init-revision: 1
Status: Running
IP: 10.20.23.233
IPs:
IP: 10.20.23.233
Controlled By: StatefulSet/gitlab-postgresql
Containers:
gitlab-postgresql:
Container ID: containerd://363c814faa23647b01448390d32b20594189ddfdb3304417d6a99160a8dd9b5a
Image: docker.io/bitnami/postgresql:12.7.0
Image ID: docker.io/bitnami/postgresql@sha256:2fa950ba3a549a0c9090fc6c88cc7cc793fe89cf3c467668dc82966d2b271816
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 12 Aug 2021 09:13:52 +1000
Finished: Thu, 12 Aug 2021 09:13:53 +1000
Ready: False
Restart Count: 5
Requests:
cpu: 250m
memory: 256Mi
Liveness: exec [/bin/sh -c exec pg_isready -U "gitlab" -d "gitlabhq_production" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [/bin/sh -c -e exec pg_isready -U "gitlab" -d "gitlabhq_production" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
Environment:
BITNAMI_DEBUG: false
POSTGRESQL_PORT_NUMBER: 5432
POSTGRESQL_VOLUME_DIR: /bitnami/postgresql
PGDATA: /bitnami/postgresql/data
POSTGRES_POSTGRES_PASSWORD_FILE: /opt/bitnami/postgresql/secrets/postgresql-postgres-password
POSTGRES_USER: gitlab
POSTGRES_PASSWORD_FILE: /opt/bitnami/postgresql/secrets/postgresql-password
POSTGRES_DB: gitlabhq_production
POSTGRESQL_ENABLE_LDAP: no
Mounts:
/bitnami/postgresql from data (rw)
/dev/shm from dshm (rw)
/docker-entrypoint-initdb.d/ from custom-init-scripts (rw)
/docker-entrypoint-preinitdb.d/init_revision.sh from custom-init-scripts (rw,path="init_revision.sh")
/opt/bitnami/postgresql/secrets/ from postgresql-password (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95sl2 (ro)
metrics:
Container ID: containerd://b62924f93576751773f5bfe08b8f14a35d00b881a982d2165d2b822cdbcb4293
Image: docker.io/bitnami/postgres-exporter:0.8.0-debian-10-r99
Image ID: docker.io/bitnami/postgres-exporter@sha256:d5a45468d8f586c8cd6e052c5d7840f53e61b528a64484b9f6b6a37ec590c156
Port: 9187/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 12 Aug 2021 09:10:51 +1000
Ready: True
Restart Count: 0
Liveness: http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6
Readiness: http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6
Environment:
DATA_SOURCE_URI: 127.0.0.1:5432/gitlabhq_production?sslmode=disable
DATA_SOURCE_PASS_FILE: /opt/bitnami/postgresql/secrets/postgresql-password
DATA_SOURCE_USER: gitlab
Mounts:
/opt/bitnami/postgresql/secrets/ from postgresql-password (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95sl2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-gitlab-postgresql-0
ReadOnly: false
postgresql-password:
Type: Secret (a volume populated by a Secret)
SecretName: gitlab-postgresql-password
Optional: false
custom-init-scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-postgresql-init-db
Optional: false
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 1Gi
kube-api-access-95sl2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m25s default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 5m24s default-scheduler Successfully assigned gitlab/gitlab-postgresql-0 to ip-10-20-24-19.ec2.internal
Normal Pulled 5m15s kubelet Container image "docker.io/bitnami/postgres-exporter:0.8.0-debian-10-r99" already present on machine
Normal Created 5m15s kubelet Created container metrics
Normal Started 5m15s kubelet Started container metrics
Normal Pulled 4m21s (x4 over 5m15s) kubelet Container image "docker.io/bitnami/postgresql:12.7.0" already present on machine
Normal Created 4m21s (x4 over 5m15s) kubelet Created container gitlab-postgresql
Normal Started 4m21s (x4 over 5m15s) kubelet Started container gitlab-postgresql
Warning BackOff 11s (x33 over 5m11s) kubelet Back-off restarting failed containerEdited by DJ Mountney