Skip to content

Error upon config change: 'No such file or directory @ rb_sysopen - /etc/gitlab/objectstorage/object_store'

Summary

Enabling KAS breaks deployment of the chart by the GitLab Operator, apparently because the storage-config secret is no longer configured.

Steps to reproduce

(I am using the GitLab Operator, but I believe this is a bug in the Chart. I am not in a position to test the chart on its own at this time.)

  1. Set up GitLab Operator 0.4.1 with chart 5.7.1 and deploy successfully.
  2. Enable KAS (global.kas.enabled = true) and redeploy.
  3. See that gitlab-toolbox has an error, and that gitlab-sidekiq-all-in-1 and gitlab-webservice-default had errors in their init containers. (This does not take down the existing GitLab deployment, luckily.)
  4. Disable KAS (global.kas.enabled = false) and redeploy.
  5. See that GitLab has recovered and the newly-created pods are not in an error state.

Configuration used

apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
spec:
  chart:
    version: 5.7.1
    values:
      global:
        hosts:
          domain: [redacted]
          externalIP: [redacted]
          gitlab:
            name: [redacted]
        ingress:
          configureCertmanager: false
          annotations:
            kubernetes.io/tls-acme: true
            cert-manager.io/cluster-issuer: [redacted]
          tls:
            secretName: gitlab-tls
        kas:
          enabled: true
      gitlab:
        webservice:
          tls:
            secretName: webservice-tls
      gitaly:
        persistence:
          size: 1Ti
          storageClass: [redacted]
      minio:
        ingress:
          tls:
            secretName: minio-tls
        persistence:
          size: 10Ti
          storageClass: [redacted]
      registry:
        ingress:
          tls:
            secretName: registry-tls
      certmanager:
        install: false

Current behavior

The change to kas.enabled = true is never successfully deployed.

Expected behavior

KAS should be enabled and deployed successfully.

Versions

  • Operator 0.4.1
  • Chart: 5.7.1
  • Platform:
    • Self-hosted: Kubespray
  • Kubernetes: (kubectl version)
    • Client: v1.23.3
    • Server: v1.21.6
  • Helm: (helm version)
    • Client: ?
    • Server: ?

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

The gitlab-webservice pod's dependencies container fails with the following logs:

+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
/usr/lib/ruby/2.7.0/psych.rb:577:in `initialize': No such file or directory @ rb_sysopen - /etc/gitlab/objectstorage/object_store (Errno::ENOENT)
	from /usr/lib/ruby/2.7.0/psych.rb:577:in `open'
	from /usr/lib/ruby/2.7.0/psych.rb:577:in `load_file'
	from /var/opt/gitlab/templates/gitlab.yml.erb:45:in `<main>'
	from /usr/lib/ruby/2.7.0/erb.rb:905:in `eval'
	from /usr/lib/ruby/2.7.0/erb.rb:905:in `result'
	from /usr/lib/ruby/2.7.0/erb.rb:890:in `run'
	from /usr/bin/erb:154:in `run'
	from /usr/bin/erb:175:in `<main>'

The logs from a successful (kas.enabled = false) deployment's gitlab-webservice's configure container:

+ set -e
+ config_dir=/init-config
+ secret_dir=/init-secrets
+ mkdir -p /init-secrets/shell
+ cp -v -r -L /init-config/shell/. /init-secrets/shell/
+ mkdir -p /init-secrets/gitaly
'/init-config/shell/./.gitlab_shell_secret' -> '/init-secrets/shell/./.gitlab_shell_secret'
'/init-config/shell/.' -> '/init-secrets/shell/.'
+ cp -v -r -L /init-config/gitaly/. /init-secrets/gitaly/
'/init-config/gitaly/./gitaly_token' -> '/init-secrets/gitaly/./gitaly_token'
'/init-config/gitaly/.' -> '/init-secrets/gitaly/.'
+ mkdir -p /init-secrets/registry
+ cp -v -r -L /init-config/registry/. /init-secrets/registry/
'/init-config/registry/./notificationSecret' -> '/init-secrets/registry/./notificationSecret'
'/init-config/registry/./gitlab-registry.key' -> '/init-secrets/registry/./gitlab-registry.key'
'/init-config/registry/.' -> '/init-secrets/registry/.'
+ mkdir -p /init-secrets/rails-secrets
+ cp -v -r -L /init-config/rails-secrets/. /init-secrets/rails-secrets/
'/init-config/rails-secrets/./secrets.yml' -> '/init-secrets/rails-secrets/./secrets.yml'
'/init-config/rails-secrets/.' -> '/init-secrets/rails-secrets/.'
+ mkdir -p /init-secrets/gitlab-workhorse
+ cp -v -r -L /init-config/gitlab-workhorse/. /init-secrets/gitlab-workhorse/
+ '[' -e /init-config/redis ]
+ mkdir -p /init-secrets/redis
+ cp -v -r -L /init-config/redis/. /init-secrets/redis/
+ '[' -e /init-config/minio ]
+ '[' -e /init-config/objectstorage ]
+ mkdir -p /init-secrets/objectstorage
'/init-config/gitlab-workhorse/./secret' -> '/init-secrets/gitlab-workhorse/./secret'
'/init-config/gitlab-workhorse/.' -> '/init-secrets/gitlab-workhorse/.'
'/init-config/redis/./redis-password' -> '/init-secrets/redis/./redis-password'
'/init-config/redis/.' -> '/init-secrets/redis/.'
'/init-config/objectstorage/./object_store' -> '/init-secrets/objectstorage/./object_store'
'/init-config/objectstorage/.' -> '/init-secrets/objectstorage/.'
'/init-config/postgres/./psql-password-main' -> '/init-secrets/postgres/./psql-password-main'
'/init-config/postgres/.' -> '/init-secrets/postgres/.'
'/init-config/gitlab-workhorse/secret' -> '/init-secrets-workhorse/gitlab-workhorse/secret'
'/init-config/redis/redis-password' -> '/init-secrets-workhorse/redis/redis-password'
'/init-config/objectstorage/object_store' -> '/init-secrets-workhorse/objectstorage/object_store'
+ cp -v -r -L /init-config/objectstorage/. /init-secrets/objectstorage/
+ '[' -e /init-config/postgres ]
+ mkdir -p /init-secrets/postgres
+ cp -v -r -L /init-config/postgres/. /init-secrets/postgres/
+ '[' -e /init-config/ldap ]
+ '[' -e /init-config/omniauth ]
+ '[' -e /init-config/smtp ]
+ '[' -e /init-config/kas ]
+ '[' -e /init-config/pages ]
+ '[' -e /init-config/oauth-secrets ]
+ set -e
+ mkdir -p /init-secrets-workhorse/gitlab-workhorse
+ cp -v -r -L /init-config/gitlab-workhorse/secret /init-secrets-workhorse/gitlab-workhorse/secret
+ mkdir -p /init-secrets-workhorse/redis
+ cp -v -r -L /init-config/redis/redis-password /init-secrets-workhorse/redis/
+ '[' -f /init-config/objectstorage/object_store ]
+ mkdir -p /init-secrets-workhorse/objectstorage
+ cp -v -r -L /init-config/objectstorage/object_store /init-secrets-workhorse/objectstorage/

The logs from an unsuccessful (kas.enabled = true) configure container (note that it does not copy an objectstore folder or object_store file, as the successful deployment's container does:

+ set -e
+ config_dir=/init-config
+ secret_dir=/init-secrets
+ mkdir -p /init-secrets/shell
+ cp -v -r -L /init-config/shell/. /init-secrets/shell/
+ mkdir -p /init-secrets/gitaly
'/init-config/shell/./.gitlab_shell_secret' -> '/init-secrets/shell/./.gitlab_shell_secret'
'/init-config/shell/.' -> '/init-secrets/shell/.'
+ cp -v -r -L /init-config/gitaly/. /init-secrets/gitaly/
'/init-config/gitaly/./gitaly_token' -> '/init-secrets/gitaly/./gitaly_token'
'/init-config/gitaly/.' -> '/init-secrets/gitaly/.'
+ mkdir -p /init-secrets/registry
+ cp -v -r -L /init-config/registry/. /init-secrets/registry/
'/init-config/registry/./notificationSecret' -> '/init-secrets/registry/./notificationSecret'
'/init-config/registry/./gitlab-registry.key' -> '/init-secrets/registry/./gitlab-registry.key'
'/init-config/registry/.' -> '/init-secrets/registry/.'
+ mkdir -p /init-secrets/rails-secrets
+ cp -v -r -L /init-config/rails-secrets/. /init-secrets/rails-secrets/
'/init-config/rails-secrets/./secrets.yml' -> '/init-secrets/rails-secrets/./secrets.yml'
'/init-config/rails-secrets/.' -> '/init-secrets/rails-secrets/.'
'/init-config/gitlab-workhorse/./secret' -> '/init-secrets/gitlab-workhorse/./secret'
'/init-config/gitlab-workhorse/.' -> '/init-secrets/gitlab-workhorse/.'
'/init-config/redis/./redis-password' -> '/init-secrets/redis/./redis-password'
'/init-config/redis/.' -> '/init-secrets/redis/.'
'/init-config/postgres/./psql-password-main' -> '/init-secrets/postgres/./psql-password-main'
'/init-config/postgres/.' -> '/init-secrets/postgres/.'
'/init-config/kas/./.gitlab_kas_secret' -> '/init-secrets/kas/./.gitlab_kas_secret'
'/init-config/kas/.' -> '/init-secrets/kas/.'
+ mkdir -p /init-secrets/gitlab-workhorse
+ cp -v -r -L /init-config/gitlab-workhorse/. /init-secrets/gitlab-workhorse/
+ '[' -e /init-config/redis ]
+ mkdir -p /init-secrets/redis
+ cp -v -r -L /init-config/redis/. /init-secrets/redis/
+ '[' -e /init-config/minio ]
+ '[' -e /init-config/objectstorage ]
+ '[' -e /init-config/postgres ]
+ mkdir -p /init-secrets/postgres
+ cp -v -r -L /init-config/postgres/. /init-secrets/postgres/
+ '[' -e /init-config/ldap ]
+ '[' -e /init-config/omniauth ]
+ '[' -e /init-config/smtp ]
+ '[' -e /init-config/kas ]
+ mkdir -p /init-secrets/kas
+ cp -v -r -L /init-config/kas/. /init-secrets/kas/
+ '[' -e /init-config/pages ]
+ '[' -e /init-config/oauth-secrets ]
+ set -e
+ mkdir -p /init-secrets-workhorse/gitlab-workhorse
+ cp -v -r -L /init-config/gitlab-workhorse/secret /init-secrets-workhorse/gitlab-workhorse/secret
'/init-config/gitlab-workhorse/secret' -> '/init-secrets-workhorse/gitlab-workhorse/secret'
+ mkdir -p /init-secrets-workhorse/redis
+ cp -v -r -L /init-config/redis/redis-password /init-secrets-workhorse/redis/
'/init-config/redis/redis-password' -> '/init-secrets-workhorse/redis/redis-password'
+ '[' -f /init-config/objectstorage/object_store ]

The configure container receives the following relevant mount:

/init-config from init-webservice-secrets (ro)

kubectl describe gives the following information for that volume on a successful deployment:

  init-webservice-secrets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          gitlab-rails-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitlab-shell-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitaly-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-redis-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-postgresql-password
    SecretOptionalName:  <nil>
    SecretName:          gitlab-registry-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-registry-notification
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitlab-workhorse-secret
    SecretOptionalName:  <nil>
    SecretName:          storage-config
    SecretOptionalName:  <nil>

But the following information is provided for the unsuccessful deployment (note the gitlab-gitlab-kas-secret presence, and lack of storage-config secret):

  init-webservice-secrets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          gitlab-rails-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitlab-shell-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitaly-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-redis-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-postgresql-password
    SecretOptionalName:  <nil>
    SecretName:          gitlab-registry-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-registry-notification
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitlab-workhorse-secret
    SecretOptionalName:  <nil>
    SecretName:          gitlab-gitlab-kas-secret
    SecretOptionalName:  <nil>

It is likely relevant that the KAS secrets would be specified before the object_store secret in the projected volume, but I didn't see anything obvious about how the KAS secrets are specified that seemed likely to cause a problem (except possibly the comment at the end of line 12?).

I am not particularly experienced with charts, so hopefully that's enough to pin down what seems to be wrong. I'm also curious what part of my configuration might differ from more common configurations, as I assume this must work for other people. Thanks!

Edited by Mitchell Nielsen