installation fails in gitlab-sidekiq-all-in-1 dependencies initContainer stacktrace

Summary

I am trying to install gitlab via the gitlab/gitlab helm chart. It fails with a stack in the gitlab-sidekiq-all-in-1 dependencies initContainer.

Steps to reproduce

Run helm upgrade -i gitlab gitlab/gitlab -f values.yaml

Configuration used

global:
  edition: ce
  operator:
    enabled: false
    gitlabVersion: 11.6.2
  application:
    create: false
    links: []
  hosts:
    domain: ....
    https: true
    externalIP: <ip>
    ssh: ~
  ingress:
    configureCertmanager: false
    annotations: 
      certmanager.k8s.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: haproxy
      kubernetes.io/tls-acme: "true"
    enabled: true
    class: haproxy
  initialRootPassword: {}
  psql:
    password: {}
    # host: myexternalsql.hostedsomewhere.else
    # port: 123
    username: gitlab
    database: gitlab
  redis:
    password: {}
  gitaly:
    authToken: {}
    internal:
      names: ['default']
    external: []
  minio:
    enabled: false
    credentials: {}
  appConfig:
    enableUsagePing: true
    defaultCanCreateGroup: true
    usernameChangingEnabled: true
    issueClosingPattern:
    defaultTheme:
    defaultProjectsFeatures:
      issues: true
      mergeRequests: true
      wiki: true
      snippets: true
      builds: true
    webhookTimeout:
    gravatar:
      plainUrl:
      sslUrl:
    extra:
      googleAnalyticsId:
      piwikUrl:
      piwikSiteId:
    lfs:
      bucket: gitlab-lfs-storage
      connection: # https://gitlab.com/charts/gitlab/blob/master/doc/charts/globals.md#connection
        secret: objectstore-lfs
        key: connection
    artifacts:
      bucket: gitlab-artifacts-storage
      connection:
        secret: objectstore-artifacts
        key: connection
    uploads:
      bucket: gitlab-uploads-storage
      connection:
        secret: objectstore-uploads
        key: connection
    packages:
      bucket: gitlab-packages-storage
      connection:
        secret: objectstore-packages
        key: connection
    backups:
      bucket: gitlab-backup-storage
      tmpBucket: gitlab-tmp-storage
    incomingEmail:
      enabled: false
      address: ""
      host: "imap.gmail.com"
      port: 993
      ssl: true
      startTls: false
      user: ""
      password:
        secret: ""
        key: password
      mailbox: inbox
      idleTimeout: 60
  shell:
    authToken: {}
    hostKeys: {}
  railsSecrets: {}
  runner:
    registrationToken: {}
  # Outgoing email server settings
  smtp:
    enabled: false
    address: smtp.mailgun.org
    port: 2525
    user_name: ""
    password:
      secret: ""
      key: password
    # domain:
    authentication: "plain"
    starttls_auto: false
    openssl_verify_mode: "peer"
  # Email persona used in email sent by GitLab
  email:
    from: ''
    display_name: GitLab
    reply_to: ''
    subject_suffix: ''
  time_zone: UTC
  service:
    annotations: {}
  antiAffinity: soft
  workhorse: {}
  # configuration of certificates container & custom CA injection
  certificates:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
      tag: 20171114-r3
    customCAs: []
certmanager:
  # Install cert-manager chart. Set to false if you already have cert-manager
  # installed or if you are not using cert-manager.
  install: false
  # Other cert-manager configurations from upstream
  # See https://github.com/kubernetes/charts/tree/master/stable/cert-manager#configuration
  rbac:
    create: true

nginx-ingress:
  enabled: false
prometheus:
  install: true
  rbac:
    create: true
  alertmanager:
    enabled: false
  alertmanagerFiles:
    alertmanager.yml: {}
  kubeStateMetrics:
    enabled: false
  nodeExporter:
    enabled: false
  pushgateway:
    enabled: false

redis-ha:
  nameOverride: redis
  enabled: false

postgresql:
  install: true
  postgresUser: gitlab
  postgresDatabase: gitlab
  imageTag: 9.6.8
  usePasswordFile: true
  existingSecret: 'secret'
  metrics:
    enabled: true
    ## Optionally define additional custom metrics
    ## ref: https://github.com/wrouesnel/postgres_exporter#adding-new-metrics-via-a-config-file

registry:
  enabled: false

redis:
  enabled: true

shared-secrets:
  enabled: true
  rbac:
    create: true

gitlab-runner:
  install: false
  rbac:
    create: true
#  runners:
#    cache:
#      cacheType: s3
#      s3BucketName: runner-cache
#      cacheShared: true
#      s3BucketLocation: us-east-1
#      s3CachePath: gitlab-runner
#      s3CacheInsecure: false
gitlab:
  migrations:
    enabled: true
  unicorn:
    enabled: true
  sidekiq:
    enabled: true
  gitaly:
    enabled: true
  gitlab-shell:
    enabled: true
  task-runner:
    backups:
      objectStorage:
        config:
          secret: s3cmd-config
          key: config

Current behavior

gitlab-sidekiq-all-in-1 dependencies end in CrashBackLoop

Expected behavior

Gitlab to become available on the configured url.

Versions

Chart: 1.4.1
Platform:
- Self-hosted: kubeadm
Kubernetes: (kubectl version)
- Client: 1.13.1
- Server: 1.12.2
Helm: (helm version)
- Client: v2.11.0+g2e55dbe
- Server:v2.11.0+g2e55dbe

Relevant logs

k -n gitlab logs -f gitlab-sidekiq-all-in-1-7f8d94b9b-dw8hr -c dependencies
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/sidekiq_queues.yml
Writing /srv/gitlab/config/resque.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
+ exec /scripts/wait-for-deps
Checking database connection and schema version
rake aborted!
NoMethodError: undefined method `deep_stringify_keys!' for #<String:0x00005635cd74fe60>
/srv/gitlab/config/object_store_settings.rb:12:in `parse'
/srv/gitlab/config/initializers/1_settings.rb:184:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:287:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:287:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:259:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:287:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:648:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/notifications.rb:166:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:647:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:611:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:611:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:30:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:30:in `run'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:55:in `block in run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:44:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:44:in `tsort_each_child'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:54:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/application.rb:352:in `initialize!'
/srv/gitlab/config/environment.rb:11:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:293:in `require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:293:in `block in require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:259:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:293:in `require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/application.rb:328:in `require_environment!'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/application.rb:448:in `block in run_tasks_blocks'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Tasks: TOP => db:version => environment
(See full trace by running task with --trace)
Checking database connection and schema version
rake aborted!
NoMethodError: undefined method `deep_stringify_keys!' for #<String:0x000056080d665b50>
/srv/gitlab/config/object_store_settings.rb:12:in `parse'
/srv/gitlab/config/initializers/1_settings.rb:184:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:287:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:287:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:259:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:287:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:648:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/notifications.rb:166:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:647:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:611:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/engine.rb:611:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:30:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:30:in `run'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:55:in `block in run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:44:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:44:in `tsort_each_child'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/initializable.rb:54:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/application.rb:352:in `initialize!'
/srv/gitlab/config/environment.rb:11:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:293:in `require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:293:in `block in require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:259:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7/lib/active_support/dependencies.rb:293:in `require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/application.rb:328:in `require_environment!'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-5.0.7/lib/rails/application.rb:448:in `block in run_tasks_blocks'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Tasks: TOP => db:version => environment
(See full trace by running task with --trace)
Checking database connection and schema version
rake aborted!

k -n gitlab logs -f gitlab-postgresql-777886f68b-8rks5 -c gitlab-postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start

waiting for server to start....LOG:  database system was shut down at 2019-01-03 20:11:08 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections
FATAL:  role "gitlab" does not exist
 done
server started
CREATE DATABASE

CREATE ROLE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

LOG:  received fast shutdown request
LOG:  aborting any active transactions
waiting for server to shut down....LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

LOG:  database system was shut down at 2019-01-03 20:11:12 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

Edited Jan 03, 2019 by Karsten Nielsen