Praefect pod fails when using Custom CA certificates

Summary

Pods with Praefect won't start after turning on global.praefect.enabled=true. Statefulsets created.

k -n gitlab get statefulsets
NAME                           READY   AGE
gitlab-gitaly                  3/3     211d
gitlab-praefect                0/2     10m

k -n gitlab describe statefulsets gitlab-praefect | tail -1
Warning  FailedCreate  5s (x18 over 11m)  statefulset-controller  create Pod gitlab-praefect-0 in StatefulSet gitlab-praefect failed error: Pod "gitlab-praefect-0" is invalid: spec.initContainers[0].volumeMounts[1].name: Not found: "custom-ca-certificates"

Steps to reproduce

Enable Praefect global.praefect.enabled=true

Versions

  • Chart: 4.4.3
  • Platform:
    • Cloud: bare metal
    • Self-hosted: Rancher RKE
  • Kubernetes:
    • Client:v1.18.3
    • Server:v1.18.3
  • Helm: (helm version)
    • Client: v3.3.4
    • Server: v3.3.4

Relevant logs

Name:               gitlab-praefect
Namespace:          gitlab
CreationTimestamp:  Wed, 07 Oct 2020 21:36:44 +0300
Selector:           app=praefect,release=gitlab
Labels:             app=praefect
                    app.kubernetes.io/managed-by=Helm
                    chart=praefect-4.4.3
                    heritage=Helm
                    release=gitlab
Annotations:        meta.helm.sh/release-name: gitlab
                    meta.helm.sh/release-namespace: gitlab
Replicas:           2 desired | 0 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=praefect
                release=gitlab
  Annotations:  checksum/config: b268a2b3f87712185db370b39d21ae5542c2c1d81bab47844d496a96750f719d
  Init Containers:
   certificates:
    Image:        registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /etc/ssl/certs from etc-ssl-certs (rw)
      /usr/local/share/ca-certificates from custom-ca-certificates (ro)
   configure:
    Image:      busybox:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      /config/configure
    Environment:  <none>
    Mounts:
      /config from praefect-config (ro)
      /init-config from init-praefect-secrets (ro)

      /init-secrets from praefect-secrets (rw)
  Containers:
   praefect:
    Image:       registry.gitlab.com/gitlab-org/build/cng/gitaly:v13.4.3
    Ports:       8075/TCP, 9236/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      sh
      /scripts/praefect/praefect-start
    Requests:
      cpu:     100m
      memory:  200Mi
    Environment:
      CONFIG_TEMPLATE_DIRECTORY:        /etc/gitaly/templates
      CONFIG_DIRECTORY:                 /etc/gitaly
      PRAEFECT_CONFIG_FILE:             /etc/gitaly/config.toml
      SSL_CERT_DIR:                     /etc/ssl/certs
      PRAEFECT_PROMETHEUS_LISTEN_ADDR:  :9236
    Mounts:
      /etc/gitaly/templates from praefect-config (rw)
      /etc/gitlab-secrets from praefect-secrets (ro)
      /etc/ssl/certs/ from etc-ssl-certs (ro)
      /scripts/praefect from praefect-scripts (rw)
  Volumes:
   praefect-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gitlab-praefect
    Optional:  false
   praefect-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gitlab-praefect-scripts
    Optional:  false
   praefect-secrets:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
   init-praefect-secrets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          gitlab-gitaly-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-praefect-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-praefect-dbsecret
    SecretOptionalName:  <nil>
   etc-ssl-certs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
Volume Claims:  <none>
Events:
  Type     Reason        Age                   From                    Message
  ----     ------        ----                  ----                    -------
  Warning  FailedCreate  6m13s (x18 over 17m)  statefulset-controller  create Pod gitlab-praefect-0 in StatefulSet gitlab-praefect failed error: Pod "gitlab-praefect-0" is invalid: spec.initContainers[0].volumeMounts[1].name: Not found: "custom-ca-certificates"
Edited by Jason Plum