Adding LDAP specification to GitLab CRD results in CrashLoopBackupOff state for gitlab-toolbox

Summary

GitLab instance was successfully deployed through the operator v0.7.0 on Openshift 4.9.15 and the landing page accessed. Gitlab CRD was amended to add global.appConfig.ldap tag. The operator processes the change but gitlab-toolbox-* ends in CrashLoopBackOff state.

Pod state:

$ oc get pods
NAME                                          READY   STATUS                  RESTARTS         AGE
# snip
gitlab-sidekiq-all-in-1-v2-648659d555-4fwdf   1/1     Running                 0                9h
gitlab-sidekiq-all-in-1-v2-7ff8df956c-jmvtm   0/1     Init:CrashLoopBackOff   14 (5m9s ago)    52m
gitlab-toolbox-58c845696f-ftzln               0/1     CrashLoopBackOff        13 (4m39s ago)   46m
gitlab-webservice-default-54bd47f4b5-646zk    0/2     Init:Error              15 (5m16s ago)   52m
gitlab-webservice-default-6586849648-gnjkb    2/2     Running                 0                9h
gitlab-webservice-default-6586849648-hxtv5    2/2     Running                 0                9h

Toolkit reports:

$ oc logs gitlab-toolbox-58c845696f-ftzln
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
/var/opt/gitlab/templates/gitlab.yml.erb:193:in `read': No such file or directory @ rb_sysopen - /etc/gitlab/ldap/main/password (Errno::ENOENT)
        from /var/opt/gitlab/templates/gitlab.yml.erb:193:in `<main>'
        from /usr/lib/ruby/2.7.0/erb.rb:905:in `eval'
        from /usr/lib/ruby/2.7.0/erb.rb:905:in `result'
        from /usr/lib/ruby/2.7.0/erb.rb:890:in `run'
        from /usr/bin/erb:154:in `run'
        from /usr/bin/erb:175:in `<main>'

Steps to reproduce

Create stable gitlab instance without LDAP integration
Update gitlab crd with ldap configuration
gitlab-toolkit-* crashes

Configuration used

Gitlab CRD AFTER adding ldap snippet into spec.chart.values.global: Note: I'm NOT saying this ldap cfg works ... that's what I'm trying to work out how to do and get it working. BUT, documentation says I can use a secret for the bind id password. [Side note: I suspect that I need to replace the spaces in the LDAP search strings with %20]

apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
spec:
  chart:
    version: "5.10.0" 
    values:
      global:
        appConfig:
          ldap:
            enabled: true
            preventSignin: false
            servers:
              # 'main' is the GitLab 'provider ID' of this LDAP server
              main:
                label: 'Active-Directory'
                host: 'adldapsvc.somewhere.com'
                port: 389
                uid: 'sAMAccountName'
                encryption: 'plain'
                verify_certificates: false
                bind_dn: >-
                  CN=sa_gitlab,OU=Service Accounts,OU=Local Objects,DC=somewhere,DC=com
                password:
                  secret: gitlab-ldap-bind-secret
                  key: bind_password
                active_directory: true
                base: >-
                  OU=Users,OU=Local Objects,DC=somewhere,DC=com
                user_filter: >-
                  (&(objectCategory=person)(memberof:1.2.840.113556.1.4.1941:=CN=GitLab-OAuth-Access,OU=GitLab,OU=Security,OU=Groups,OU=Local Objects,DC=somewhere,DC=com))
        edition: ce
        hosts:
          domain: apps.ocp01.somewhere.com
          hostSuffix: live
        ingress:
          class: none
          annotations:
            route.openshift.io/termination: edge
          configureCertmanager: false
          tls:
            secretName: my-wildcard-tls-chain
      nginx-ingress:
        enabled: false
      certmanager:
        install: false

Current behavior

LDAP configuration fails to deploy. Deleting the change in gitlab crd does not revert the system back to a working state. Using oc debug gitlab-toolkit-* and digging about ...

I can see the line in /var/opt/gitlab/templates/gitlab.yml.erb that shows my ldap settings and the line password: <%= File.read('/etc/gitlab/ldap/main/password').strip.to_json %>

Looking into /var/gitlab

drwxrwsrwt. 9 root git  200 Apr 28 07:38 .
drwxr-xr-x. 1 root root  45 Apr 28 07:38 ..
-rw-r--r--. 1 git  git  664 Apr 28 07:38 .s3cfg
drwxr-sr-x. 2 git  git   60 Apr 28 07:38 gitaly
drwxr-sr-x. 2 git  git   80 Apr 28 07:38 minio
drwxr-sr-x. 2 git  git   60 Apr 28 07:38 postgres
drwxr-sr-x. 2 git  git   60 Apr 28 07:38 rails-secrets
drwxr-sr-x. 2 git  git   60 Apr 28 07:38 redis
drwxr-sr-x. 2 git  git   80 Apr 28 07:38 registry
drwxr-sr-x. 2 git  git   60 Apr 28 07:38 shell

we're missing the ldap subfolder

Checking I have a valid secret:

$ oc get secret gitlab-ldap-bind-secret -o yaml
apiVersion: v1
data:
  bind_password: {{base64 code holding password}}
  bind_userid: {{base64 code holding reference user id}}
kind: Secret
metadata:
  creationTimestamp: "2022-04-28T07:09:46Z"
  name: gitlab-ldap-bind-secret
  namespace: gitlab-system
  resourceVersion: "82346134"
  uid: ef331ecb-1f2a-4af8-a62d-e06c21420e3c
type: Opaque

The bind_userid value is there to provide a cross reference to to which account password they should be using.

Expected behavior

LDAP configuration is deployed successfully

Versions

Operator: 0.7.0
Platform:
- Self-hosted: OpenShift v4.9.15
Kubernetes: (as reported from my linux workstation)
- Client: version.Info{Major:"1", Minor:"21", GitVersion:"v0.21.0-beta.1", GitCommit:"96e95cef877ba04872b88e4e2597eabb0174d182", GitTreeState:"clean", BuildDate:"2021-10-01T00:19:03Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
- Server version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3+e790d7f", GitCommit:"3a0f2c90b43e6cffd07f57b5b78dd9f083e47ee2", GitTreeState:"clean", BuildDate:"2021-12-14T02:10:38Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

WARNING: version difference between client (0.21) and server (1.22) exceeds the supported minor version skew of +/-1 Opps! Better update the client! However, it's not really used in an OpenShift cluster that much.

Relevant logs

$ kubectl logs -l control-plane=controller-manager -c manager
2022-04-28T09:37:07.718Z        INFO    controllers.GitLab      self-signed certificates job skipped, not needed per configuration      {"gitlab": "gitlab.gitlab-system"}
2022-04-28T09:37:08.269Z        INFO    controllers.GitLab      running all migrations  {"gitlab": "gitlab-system/gitlab"}
2022-04-28T09:37:08.271Z        INFO    controllers.GitLab      reconciling Webservice Deployments      {"gitlab": "gitlab-system/gitlab", "pause": false}
2022-04-28T09:37:08.284Z        INFO    controllers.GitLab      reconciling Sidekiq Deployments {"gitlab": "gitlab-system/gitlab", "pause": false}
2022-04-28T09:37:23.369Z        INFO    controllers.GitLab      Reconciling GitLab      {"gitlab": "gitlab-system/gitlab"}
2022-04-28T09:37:23.371Z        DEBUG   controllers.GitLab      version information     {"gitlab": "gitlab-system/gitlab", "upgrade": false, "current version": "5.10.0", "desired version": "5.10.0"}
2022-04-28T09:37:23.446Z        INFO    controllers.GitLab      self-signed certificates job skipped, not needed per configuration      {"gitlab": "gitlab.gitlab-system"}
2022-04-28T09:37:23.666Z        INFO    controllers.GitLab      running all migrations  {"gitlab": "gitlab-system/gitlab"}
2022-04-28T09:37:23.671Z        INFO    controllers.GitLab      reconciling Webservice Deployments      {"gitlab": "gitlab-system/gitlab", "pause": false}
2022-04-28T09:37:23.758Z        INFO    controllers.GitLab      reconciling Sidekiq Deployments {"gitlab": "gitlab-system/gitlab", "pause": false}

This log file just keeps repeating the same set of messages.

Edited Apr 28, 2022 by Steve Power