Praefect pod fails when using Custom CA certificates
Summary
Pods with Praefect won't start after turning on global.praefect.enabled=true
.
Statefulsets created.
k -n gitlab get statefulsets
NAME READY AGE
gitlab-gitaly 3/3 211d
gitlab-praefect 0/2 10m
k -n gitlab describe statefulsets gitlab-praefect | tail -1
Warning FailedCreate 5s (x18 over 11m) statefulset-controller create Pod gitlab-praefect-0 in StatefulSet gitlab-praefect failed error: Pod "gitlab-praefect-0" is invalid: spec.initContainers[0].volumeMounts[1].name: Not found: "custom-ca-certificates"
Steps to reproduce
Enable Praefect global.praefect.enabled=true
Versions
- Chart: 4.4.3
- Platform:
- Cloud: bare metal
- Self-hosted: Rancher RKE
- Kubernetes:
- Client:v1.18.3
- Server:v1.18.3
- Helm: (
helm version
)- Client: v3.3.4
- Server: v3.3.4
Relevant logs
Name: gitlab-praefect
Namespace: gitlab
CreationTimestamp: Wed, 07 Oct 2020 21:36:44 +0300
Selector: app=praefect,release=gitlab
Labels: app=praefect
app.kubernetes.io/managed-by=Helm
chart=praefect-4.4.3
heritage=Helm
release=gitlab
Annotations: meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: gitlab
Replicas: 2 desired | 0 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=praefect
release=gitlab
Annotations: checksum/config: b268a2b3f87712185db370b39d21ae5542c2c1d81bab47844d496a96750f719d
Init Containers:
certificates:
Image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/etc/ssl/certs from etc-ssl-certs (rw)
/usr/local/share/ca-certificates from custom-ca-certificates (ro)
configure:
Image: busybox:latest
Port: <none>
Host Port: <none>
Command:
sh
/config/configure
Environment: <none>
Mounts:
/config from praefect-config (ro)
/init-config from init-praefect-secrets (ro)
/init-secrets from praefect-secrets (rw)
Containers:
praefect:
Image: registry.gitlab.com/gitlab-org/build/cng/gitaly:v13.4.3
Ports: 8075/TCP, 9236/TCP
Host Ports: 0/TCP, 0/TCP
Args:
sh
/scripts/praefect/praefect-start
Requests:
cpu: 100m
memory: 200Mi
Environment:
CONFIG_TEMPLATE_DIRECTORY: /etc/gitaly/templates
CONFIG_DIRECTORY: /etc/gitaly
PRAEFECT_CONFIG_FILE: /etc/gitaly/config.toml
SSL_CERT_DIR: /etc/ssl/certs
PRAEFECT_PROMETHEUS_LISTEN_ADDR: :9236
Mounts:
/etc/gitaly/templates from praefect-config (rw)
/etc/gitlab-secrets from praefect-secrets (ro)
/etc/ssl/certs/ from etc-ssl-certs (ro)
/scripts/praefect from praefect-scripts (rw)
Volumes:
praefect-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-praefect
Optional: false
praefect-scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-praefect-scripts
Optional: false
praefect-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
init-praefect-secrets:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: gitlab-gitaly-secret
SecretOptionalName: <nil>
SecretName: gitlab-praefect-secret
SecretOptionalName: <nil>
SecretName: gitlab-praefect-dbsecret
SecretOptionalName: <nil>
etc-ssl-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
Volume Claims: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 6m13s (x18 over 17m) statefulset-controller create Pod gitlab-praefect-0 in StatefulSet gitlab-praefect failed error: Pod "gitlab-praefect-0" is invalid: spec.initContainers[0].volumeMounts[1].name: Not found: "custom-ca-certificates"
Edited by Jason Plum